Assistance tracing logs related to random reboots

[dageyra]

We have a gentoo server running Apache and mysql that is randomly rebooting itself. We have syslog setup such that almost everything is logged, but we have not been able to determine what the cause of the reboot is. The machine comes right back up, so it's often not apparent the reboot occurred for perhaps days afterwards. There does not seem to be any noticeable pattern, it can reboot after a couple days or even up to a month, but it is ongoing. We've looked in dmesg, apache logs, ssh logs and so forth, but nothing seems out of the ordinary. The server does not appear to experience any heavy load from what we can tell, it has 4 GB of RAM and PHP does run on almost all of the couple dozen sites.

What can we look into, or perhaps what service could be used, to trace there reboots and identify the source? Is there some way to have the server log information when an system error occurs that would cause such reboots? We're certain there is some kind of crash, it's not a good reboot. If any further info is needed, we can provide it, and thanks for any suggestions you have regarding this situation.

[Hu]

There are very few things that can cause a system to instantly restart like that. A doublefault event when the kernel is built without CONFIG_DOUBLEFAULT=y (only applicable to x86_32) can cause this. It might also happen if you have configured the kernel to panic&reboot on certain error conditions. There are various hardware failures that can cause a system to reboot. How is the cooling on the system? Has this machine always had problems or did it just start to fail one day after multiple months of good operation?

You might be able to isolate when this occurs if you arrange for the initscripts to notify some other server when the system boots successfully. This other system could then log how often the server is resetting.