24/7 - watchdog for server processes?

stillman

hello!

today when i checked the status of my server, i realized that the asterisk(telephony-server) process had been shut down by oom-killer yesterday in the evening - my telephone was not reachable for 12h

now i know why it was so quiet yesterday

unfortunately i could not yet find out what the reason for the high memory usage was, i experienced this once when i forgot to disable sip debugging, but this was not the case yesterday. however, i want to make sure that this doesn't happen again in the future - the best solution i guess would be a script that checks if all processes i want to be running 24/7 are still there and in case they have died, restarts them (of course it will be difficult to verify if the process is working as expected but in the case of asterisk i could create callfiles so i get a call from the server every few hours)- i'm still quite unexperienced with scripting and i'm pretty sure that i'm not the only one who had this problem/idea so i guess there's no reason to reinvent the wheel - anyone knows if there's a software that does what i need? i know there's the linux watchdog that reboots the machine if it hangs - but that would be a bit over the leap again and if something like yesterday happens, it won't help much.
another interesting feature would be a monitoring suite that summarizes the status of the system - writes emails, sends sms or feeds a webserver with wap-pages so i can watch the status on my cell-phone.
any recommendations are welcome!

moocha · Watchman Joined: 21 Oct 2003 Posts: 5722

Um, perhaps the most reliable watchdog, i.e. /sbin/init?

Look at the manpage for init and inittab (/etc/inittab).
_________________
Military Commissions Act of 2006: http://tinyurl.com/jrcto

"Those who would give up essential liberty to purchase a little temporary safety deserve neither liberty nor safety."
-- attributed to Benjamin Franklin

morodoch · Guru Joined: 22 Sep 2005 Posts: 523 Location: England

I agree with moocha - start your process from inittab.

If you *do* want to run checks on your systems, I can recommend using nagios - just emerge it.

Docs here: http://www.nagios.org/

There are a lot of built-in checks for the usual suspects, but writing your own plugins is easy - just a process to write a line of standard output, and exit with 0 1 2 or 3 for different states. It supports multiple notifications and escalations, so you can e-mail and then SMS or whatever.
_________________
Well, the Sister was right. You boys could use a little churching up. Slide on down to the Triple Rock, and catch Rev. Cleophus. You boys listen to what he's got to say.

-- Curtis

stillman · Posted: Sat Dec 10, 2005 4:34 pm Post subject:

thanks for the feedback - someday i will know my system i hope

so far i've been successfull by adding the line

morodoch · Guru Joined: 22 Sep 2005 Posts: 523 Location: England

Things started from inittab are restarted automatically, that's the whole point, I guess.

To stop your process once it's been started this way, comment out the line from inittab (with a #), then run this command as root:

magic919 · Posted: Sat Dec 10, 2005 5:28 pm Post subject:

Have a look at Monit, it's in Portage.