Weirdest problem - predictable semi random crashes

[nunogt]

Hi guys, I got the weirdest problem, and despite being extemely annoying, it's being very difficult to understand exactly what's wrong, let alone solve it.
The thing is my server crashes randomly. But not always. This machine is supposed to be on 24/7. It's a proxy server, webserver, and ftp server. I use it mainly for development purposes and to serve as a gateway for my small home network. Nearly every night, after some time idle (which I can know exactly since my screensaver is a clock) the machine dies. Everything stops responding - every daemon, and even KDE and the mouse. It just freezes and stays on wasting power. This behaviour is not constant, though. There are some times (every 4 or 5 reboots) that the machine does not crash and works flawlessly for several days in a row, until I have to reboot it for some reason. Then, in the next boot, the cicle restarts.
One thing I notice when in one of these likely-to-freeze boots is that an 'emerge --sync' is very fast (it takes about 2 minutes syncinc and updating portage cache), while on a stable-boot it always takes 15 or more minutes. There's also the small detail that a vanilla-sources kernel 2.6 won't be able to boot at all on this machine. No idea why, and I see nothing useful in the system messages. The last message displayed when booting is "Initializing CPU#0", followed by intense HD activity. The previous one is something to do with APIC (not ACPI). I find it unlikely to be able to provide you the boot log since the system hangs and I have to restart it. gentoo-sources-2.6 boot fine, though, and that's the one which I'm currently using. I've tested different versions of vanilla-sources and all of them suffer from the same problem.
I suspected this could be a HD problem so I run seatools (a segate maintenance tool) in both of my discs, and it found nothing unusual. I also run memtest which said everything was fine. I'm really lost here.

My specs:

Pentium3
Motherboard with VIA chipset
320 MB RAM
nVidia GeForce2 Ti (using nvidia-kernel and nvidia-glx as explained here)
/ , /boot , and swap space in hda (Seagete 80GB)
/var, /tmp and swap space in hdb (Seagate 20GB)
gentoo-sources-2.6.12-r10 (stable)
no ~arch packages and no weird gcc flags (in fact i'm even using O2)

Wtf should I do?

[PaulBredbury]

[wally.hall]

Try setting a Cron job running to FTP all your logs to another computer say ever 5 minutes, then set the machine going and see what comes out nearest the lockup (that way you don't have to know in advanced when it goes down). If it is the HD, you may well see a lot of I/O errors or something.

Also, is your Swap OK? If there's some bad sectors on the HD, and the Swap is getting b0rk3d, anything could happen!

Also, check there's nothing odd in your BIOS turned on, no BIOS shadowing or anything, and check your CPU is clocked correctly.

Otherwise, I can't think of anything obvious as to why it'd be acting the way it is! I take it the machine is running KDE and the clock screensaver is freezing at lockups? Maybe not starting X, if it's something related to your Video hardware or something.

Hope you figure it out, problems arn't such a bother when you know what's causing it and you're getting somewhere to fixing it, I know how your feeling!

Take care,

wally
_________________
I like Gentoo why?
Because it works how I want it to work.

[nunogt]

[wally.hall]

I'd just log everything you can find which might be relevant! Write a lil script to copy the output of "ps aux" and "top", also copy all of your /var/log, then you've got everything available. I'm a bit tired right now (nearly 00:01am!), so I'll do some thinking and research tomorrow and post back what logs I think will be of use for us to look at. Feel free to E-Mail me a copy of your logs (just tar them up and E-Mail them, don't worry about the filesize) if you get a crash before I post on here again, I'll happily have a look!

Take care,

wally
_________________
I like Gentoo why?
Because it works how I want it to work.

[nosatalian]

Are you using a radeon card with the ATI drivers? I've had issues like this with ATI cards using fglrx drivers, particularly when in screensavers (even non-3d ones)

[nunogt]

[jschellhaass]

Have you tried memtest86 and/or CPUburn?

jeff

[nunogt]

[jschellhaass]

you can just emerge cpuburn.

jeff

[darkdigger]

I've been having the exact same problem as the original poster. About two weeks ago, my server (apache, ssh, sql, folding@home, no GUI) started randomly freezing up. No ping responses, no monitor output, no ssh, no keyboard response, nada. At first, I thought it may have had to do with the update to openssh I emerged around that time, but I disabled the sshd and the system still froze up. I disabled single services at a time and then moved on to disabling multiple services to determine the culprit, but it kept freezing. Next, I updated my kernel form 2.6.11-gentoo to 2.6.12-gentoo-r10 (genkernel) and it still kept freezing.

As of this evening, the server won't even POST anymore, so I guess hardware is the culprit. Not exactly sure what piece, but I'm betting it's either the CPU or motherboard. Hopefully, my bad luck will help you diagnose ur problem. The server is only 9 months old too. *sigh*

Best,
Arash

P.S. Here are the server specs for whats its worth
Athlon 2400
1GB RAM
MB Video
MB Ethernet
All additional motherboard features disabled (extra ports, devices, etc)
200GB Western Digital
O2 and no ~arch packages

[nunogt]

Downgrading from 2.6.12 to 2.6.9 without recycling the old settings did the trick. Tonight it didn't freeze.
Let's hope this wasn't a mere coincidence.