how do I stop buffer caching?

[corck]

Hi all,
I am developing a system to stream very large uncompressed videofiles which are about 1,5TB.
I need a bandwith of 195MB/s.
my system is now:

kernel: 2.4.18
xeon 2,4GHz
1GB RAM
3ware escalade controller 12 s-ata harddisks, seagate barracuda 7200.7 - 120GB - SCSI

when I do different tests with bonnie++ I get very good results for a 2GB file, readperformance 249mb/s,
but when the files get larger the speed is going down significantly.
at 100GB - 187MB/s
at 1TB - 161MB/s
the write performance stays at 183MB/s any time, up to 1TB.

I am not sure if the performance slows down just because of physical reasons, like the disks are slower on the inside in contrast to the outside or if linux slows it down with some kind of caching.

I don't know a lot about linux's memory mapping / caching so I'll be thankful for every advice.
when I look at the memory during the tests with bonnie I can see that my memory runs full, also my swap memory.

I read something about raw disk transfer, but than I have to rewrite the driver as I understood?

/Christoph

[robdavies]

There can be a number of reasons for slow down :

1) Disks are slower on inside (larger cyl numbers) as you mentioned
2) The file system may be inefficient, finding blocks for large files, eg) triple-indirect blocks, ext2/ext3 (shouldn't matter with serial read ahead).
3) Paging going on due to cache, causing periodically active program pages being swapped out, then back in, with overall loss of throughput.

Problem 1) is averted by using striping, I guess you're doing that already to get those throughput numbers.

Problem 2) can be mitigated by moving to 64bit file system, designed for large files, have you tried xfs?

Problem 3)

Using raw devices might bring you 5 or 10%, but you loose convenience of working with files, so I should avoid that route. Your memory ought to be full, it's there to be used and Linux will cache pages, just maintaining a small amount of free pages in reserve.

To control whether files are buffered or not, look at open(2) and the O_DIRECT flag, and madvise(2) man pages. Perhaps that's all you need to do, if so you're lucky

If you have background writes going on, throughput can be reduced, due to reads being synchronous. You can reduce the effect, by doing large reads say 32, 64 KiB minimum or try much larger if you don't mind latency (though in 2.4 the I/O is really done in 4KiB pagesize blocks). The 2.6.0-test series kernel has various parts of I/O system reworked, and features like Anticipatory disk scheduler, to improve throughput. Though they're fairly solid already for most ppl (I'm using 2.6.0-test5-mm4 at moment), and would likely be OK for development, they won't be recommended for production deployment for 4 months or so. 2.6 also is featuring real kernel support for POSIX AIO, which has potential for zero-copy operation, though I don't expect that currently to improve throughput in this application, it might though in the future when kernel AIO has matured.

When I did benchmarking, I found using RAID (even Software) mattered more than the filesystem, though I used a maximum of 0.5GiB files, rather than the sizes you're reaching.

If you can upgrade your kernel, to a more recent version, try out XFS, and if you still don't get the required performance post to LKML, as it's the sort of issue that is likely to attract suggestions from Kernel Hackers. Perhaps even patches ...

[corck]

I already tried different filesystems, I got the best performance with ext2. XFS worked also well, JFS had problems with writing (speed went down to 18MB/s), I suppose the journaling file caused this.
I will try the O_DIRECT flag, and hope it will affect my system.

I also thinking of maybe the filesystem affects the system with large files.

I try to fill upp my disk to 1TB and then make a new test, to see the speed when its full.

anyway I will also see what the kernel 2.6 can do for me.