| View previous topic :: View next topic |
| Author |
Message |
Vieri
l33t
Joined: 18 Dec 2005
Posts: 887
|
 Posted: Mon Jan 13, 2020 9:13 am Post subject: how to automatically restart a crashed init.d service |
 |
|
Hi,
I'm wondering how others are doing this.
For some critical daemons, I use inittab with the respawn option.
I'd like to do something similar with my init scripts, and automatically restart them if their corresponding services crash (but not necessarily all of them).
What is the best approach?
method 1:
Use a custom cron job or inittab script to parse the output of:
Restart the list of services.
The inconvenience of this method is that the restart does not necessarily start right after a crash (depends on the cron periodicity setting).
Also, I am not sure every single service crash case is reflected as "crashed" in rc-status -c. Maybe some "crashes" are sometimes reported as "stopped" services? Finally, will "-c" report a crashed service in any runlevel, including the "manual runlevel"?
method 2:
Use a custom cron job or inittab script to parse the output of:
| Code: | | rc-status -m -C | grep -v '\[ started \]' | cut -f 2 -d ' ' |
Restart the list of services in the manual runlevel.
The inconvenience of this method is that I don't know how to tell rc-status to list only the services in the default runlevel. I might also auto-restart a service that was purposely stopped manually. As in method 1, the restart does not necessarily take place right after a crash.
method 3:
Use supervise-daemon.
The huge inconvenience of this method is that Gentoo wiki states that it is still unstable software, requires manually hacking the init.d files of the services you want to supervise, and make sure the service doesn't use its own PID file or forks.
Any ideas?
Despite its disadvantages, method 1 seems to be the easiest to implement and maintain for now.
The supervise-daemon looks interesting, but it's too tedious to set up and maintain. |
|
| Back to top |
|
 |
mike155
Advocate
Joined: 17 Sep 2010
Posts: 4438
Location: Frankfurt, Germany
|
| Posted: Mon Jan 13, 2020 12:47 pm Post subject: |
|
|
Switch to Systemd. Use one of the restart options.
| Code: | [Service]
Restart=always
RestartSec=3 |
|
|
| Back to top |
|
|
toralf
Developer
Joined: 01 Feb 2004
Posts: 3925
Location: Hamburg
|
|
| Back to top |
|
|
Jaglover
Watchman
Joined: 29 May 2005
Posts: 8291
Location: Saint Amant, Acadiana
|
|
| Back to top |
|
|
Vieri
l33t
Joined: 18 Dec 2005
Posts: 887
|
| Posted: Mon Jan 13, 2020 1:36 pm Post subject: |
|
|
Not quite. OpenRC does not seem to restart crashed services in "Dynamic Runlevel: manual". It will restart crashed services in "Runlevel: default".
Thanks anyway.
Moving to systemd is another option, but that would require a lot of work and testing. |
|
| Back to top |
|
|
Vieri
l33t
Joined: 18 Dec 2005
Posts: 887
|
| Posted: Mon Jan 13, 2020 1:40 pm Post subject: |
|
|
| Jaglover wrote: | method 0:
Investigate why they crash and fix it.
|
Good one, and of course the preferred method. However, sometimes there's simply no time or resources to investigate. For instance, I had 1 crash of the tomcat service every 6 months, nothing worthwhile in the logs. That's the kind of service crash I don't mind simply restarting, but it has to be done right away. |
|
| Back to top |
|
|
Vieri
l33t
Joined: 18 Dec 2005
Posts: 887
|
| Posted: Mon Jan 13, 2020 1:55 pm Post subject: |
|
|
| Vieri wrote: |
Not quite. OpenRC does not seem to restart crashed services in "Dynamic Runlevel: manual". It will restart crashed services in "Runlevel: default".
|
Let me rephrase that.
will start a crashed service even if it's in the manual dynamic runlevel.
That's fine, but I need to run "openrc -n" if I want to avoid openrc stopping the services that are not in the default runlevel but in "Dynamic Runlevel: manual". That's a manual run of openrc -n.
I haven't found a config option in rc.conf equivalent to "-n". I guess that's why they came up with supervise-daemon. |
|
| Back to top |
|
|
GDH-gentoo
Veteran
Joined: 20 Jul 2019
Posts: 1549
Location: South America
|
| Posted: Tue Jan 14, 2020 1:41 am Post subject: Re: how to automatically restart a crashed init.d service |
|
|
The easiest alternatives, compared to changing the init system or installing additional software, are inittab respawn lines and supervise-daemon, indeed.
inittab respawn lines are the reliable alternative, at the cost of having almost no service management (services start all at the same time when a runlevel is entered, are killed when the runlevel is left, and pretty much that's it).
If you are using OpenRC, supervise-daemon comes for free, at the cost of having imperfect supervision (nobody supervises the supervisors).
| Vieri wrote: | method 1:
Use a custom cron job or inittab script to parse the output of:
[...]
The inconvenience of this method is that the restart does not necessarily start right after a crash (depends on the cron periodicity setting)
[...]
method 2:
Use a custom cron job or inittab script to parse the output of:
| Code: | | rc-status -m -C | grep -v '\[ started \]' | cut -f 2 -d ' ' |
[...]
As in method 1, the restart does not necessarily take place right after a crash. |
Exactly. That is polling. See "notification vs polling"
| Vieri wrote: | method 3:
Use supervise-daemon.
The huge inconvenience of this method is that Gentoo wiki states that it is still unstable software, |
New OpenRC features are usually labeled "experimental", and seemingly stay experimental forever. It's like the package never reaching version 1.0 
| Vieri wrote: | | requires manually hacking the init.d files of the services you want to supervise, |
If the service script does not have a start() and a stop() function, i.e. it relies implicitly on start-stop-daemon, the supervisor=supervise-daemon assignment to turn on supervision for the service, and the assignment to command_args_foreground, if needed, can be added to the corresponding /etc/conf.d file, which is meant to be modified (or created) by the administrator. And if the supervisor=supervise-daemon assignment is present, command_background and command_args_background are ignored in the service script, pidfile is used as the PID file of the supervise-daemon process, and start_stop_daemon_args is interpreted as command line options for supservise-daemon instead (or ignored if supervise_daemon_args is set). So directly modifying the service script might not be needed. But if you have to, what would be the problem?
| Vieri wrote: | | and make sure the service doesn't use its own PID file or forks. |
If this can be configured with command line arguments, command_args_foreground in the /etc/conf.d file can be used to do that. |
|
| Back to top |
|
|
Vieri
l33t
Joined: 18 Dec 2005
Posts: 887
|
| Posted: Tue Jan 14, 2020 7:19 am Post subject: Re: how to automatically restart a crashed init.d service |
|
|
| GDH-gentoo wrote: | | So directly modifying the service script might not be needed. But if you have to, what would be the problem? |
Maintainability.
I'd have to keep track of file changes between upgrades. I grant you it seldom happens with init scripts, but I'd need to check for that anyway.
Thanks for all the feedback. |
|
| Back to top |
|
|
Ionen
Developer
Joined: 06 Dec 2018
Posts: 2727
|
| Posted: Tue Jan 14, 2020 7:57 am Post subject: Re: how to automatically restart a crashed init.d service |
|
|
| Vieri wrote: | | I'd have to keep track of file changes between upgrades. I grant you it seldom happens with init scripts, but I'd need to check for that anyway. |
What I like to do in case like this is to use a user patch instead, so packages will install the pre-modified files (I notably have a few for openrc). Also nice to keep track of changes, if the patch breaks it likely mean I should check rather than use my old file as-is which may be incompatible. |
|
| Back to top |
|
|
Vieri
l33t
Joined: 18 Dec 2005
Posts: 887
|
| Posted: Tue Jan 14, 2020 5:30 pm Post subject: Re: how to automatically restart a crashed init.d service |
|
|
| Ionen wrote: | | if the patch breaks it likely mean I should check |
Thanks, Ionen.
I already do that for other parts of the system, so I was hoping not to have to do this for init scripts too...
Oh, well, openrc -n every couple of minutes isn't too bad if I don't get any major crashes during working hours. |
|
| Back to top |
|
|
GDH-gentoo
Veteran
Joined: 20 Jul 2019
Posts: 1549
Location: South America
|
| Posted: Tue Jan 14, 2020 11:50 pm Post subject: Re: how to automatically restart a crashed init.d service |
|
|
| Vieri wrote: | | Maintainability. |
I'd still have a look at the service script of the daemon you want to restart. Depending on what it does, maybe all you need to have it supervised is:
/etc/conf.d/$insert-service-name-here
| Code: | supervisor=supervise-daemon
command_args_foreground="some command-line options" |
|
|
| Back to top |
|
|
apurkrt
Tux's lil' helper
Joined: 26 Feb 2011
Posts: 116
Location: Czechia, Europe
|
|
| Back to top |
|
|
|
Other Things Gentoo
