HDD full out of nowhere

[trojahn]

I'm not really sure anymore where to look to solve this puzzle... A search on the forums didn't return anything relevant but I can't believe I'm the only one having this problem...

At least 3 servers of mine that happen to be Gentoo boxes are starting to fill some of their partitions out of nowhere until it gets completely full and crashes the whole system... This "invisible" data is there until I reboot the server...

df output shows me something like:
Filesystem Size Used Avail Use% Mounted on
/dev/hdc3 19G 15G 4.0G 80% /

and after reboot it goes down to:
Filesystem Size Used Avail Use% Mounted on
/dev/hdc3 19G 13G 6.0G 69% /

du -s -h /*, on the other hand, doesn't show me any directories containing that data, hence why I called it "invisible"...

I can't find any logs growing or anything that could be responsible for that growing in size...

This also happened to 3 different servers this week, some with partitions much larger than the one I'm showing as example here... Coincidence or not, all these servers were updated 2 weeks ago and after that it started happening...

There is no errors in /var/log/messages or anything that can give me any clue... The HDDs simply fill up over time with nothingness until the whole system is rebooted...

This is some relevant data about the server showing as example here:
Linux backup 2.6.27-gentoo-r7 #1 Fri Jan 2 13:29:44 BRST 2009 i686 Intel(R) Celeron(R) CPU 2.66GHz GenuineIntel GNU/Linux

/dev/hdc1 /boot ext2 defaults 1 2
/dev/hdc3 / reiserfs defaults 0 1
/dev/hdc2 none swap sw 0 0
(happened on ext3 on other server aswell)

Any tips on something I should be looking for or clues about what could be happening are appreciated...

[eccerr0r]

One possibility is that the machine is deleting files it's still using (has open file handles on). Un*x allow this by keeping pseudo files around that you can't access through tile filesystem and keeps them around on disk until all handles are done with them, then it frees the disk space. This is why you rarely see "file is in use" errors when deleting files.

Something that clearly shows this behavior is upgrading software (emerge). When you upgrade a package, it only upgrades the package on disk, but not in memory if it's running. A reboot will load a fresh image from disk and this is something you should do anyway. However it's just as good to (manually) just stop and restart each program after upgrading each package as needed when convenient.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

[trojahn]

Thanks eccerr0r

That was exactly what was happening... Took a while for the problem to happen again and that's why I'm only replying now...

I think the culprit was logrotate... It was deleting a file that was being used by quagga (and quagga wasn't being restarted after it). Here's what lsof returned... This space was released, as you mentioned, after I restarted the bgpd process.

bgpd 3648 quagga 6w REG 3,3 5079686826 856422 /var/log/zebra.log-20091204 (deleted)