Per-process, per-disk IO accounting.

[tlhonmey]

I keep getting asked for a way to monitor not only how much disk IO each process on the system is using, but how much of it is going to which specific disk in the system.

All of the basic IO monitoring tools everyone recommends just give total IO per process and total IO per disk, which doesn't let one figure out which of the high-throughput processes is hammering the one specific disk that is overloaded.

There are a number of advanced system audit and logging tools, but the feature description is always a little vague on details, and the search engines I've tried don't seem to be able to work with the nuance of my requirements. So I'm down to asking if there's anyone out there who has set up such a thing before and knows which tools will actually work.

For clarity, the system is pushing something like 240 megabytes/second to bulk storage drives via a data processing pipeline, and some part of that pipeline is writing a lot more data to the OS drive than it should be, but I'm not finding any way to split out how much of a process' IO is going to bulk storage and how much is going to the OS drive, so there's no way to tell which of the processes involved is the misbehaving one.

I don't care how complicated it is to set up, I just don't want to waste a lot of time going through one tool after another just to find out that they don't do what I need.

Obviously per-file, per-process IO accounting would work too since it's easy to figure out which disk a file is on.