Per-process, per-disk IO accounting.

[tlhonmey]

I keep getting asked for a way to monitor not only how much disk IO each process on the system is using, but how much of it is going to which specific disk in the system.

All of the basic IO monitoring tools everyone recommends just give total IO per process and total IO per disk, which doesn't let one figure out which of the high-throughput processes is hammering the one specific disk that is overloaded.

There are a number of advanced system audit and logging tools, but the feature description is always a little vague on details, and the search engines I've tried don't seem to be able to work with the nuance of my requirements. So I'm down to asking if there's anyone out there who has set up such a thing before and knows which tools will actually work.

For clarity, the system is pushing something like 240 megabytes/second to bulk storage drives via a data processing pipeline, and some part of that pipeline is writing a lot more data to the OS drive than it should be, but I'm not finding any way to split out how much of a process' IO is going to bulk storage and how much is going to the OS drive, so there's no way to tell which of the processes involved is the misbehaving one.

I don't care how complicated it is to set up, I just don't want to waste a lot of time going through one tool after another just to find out that they don't do what I need.

Obviously per-file, per-process IO accounting would work too since it's easy to figure out which disk a file is on.

[druggo]

You can consider using the cgroup IO controller to achieve this. By placing each process that needs to be monitored into its own cgroup, you can determine the IO usage of each process by monitoring the IO activity of the cgroups.

By the way, programs started by systemd naturally have their own independent cgroups.

and if the system is already containerized, there is usually only one process per container. In such cases, the cgroup IO activity of the container will be quite clear.
_________________
AMD FX-8300 / Loongson-2f (lemote-yeeloong)