| View previous topic :: View next topic |
| Author |
Message |
baak6
Tux's lil' helper
Joined: 30 Aug 2003
Posts: 141
|
 Posted: Mon Feb 21, 2005 6:00 pm Post subject: Apache crashes every now and then |
 |
|
Hello friends.
I'm running apache 2.0.52-r2 with SSL and PHP 4.3.10. A few days ago it started crashing for some reason, this is what the logs say:
| Code: |
[...]
[Mon Feb 21 17:09:21 2005] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 81
[Mon Feb 21 17:09:21 2005] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 81
[Mon Feb 21 17:09:21 2005] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 81
[Mon Feb 21 17:09:21 2005] [alert] Child 11310 returned a Fatal error...\nApache is exiting!
[Mon Feb 21 17:09:21 2005] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Mon Feb 21 17:09:21 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Mon Feb 21 17:09:33 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Mon Feb 21 17:09:34 2005] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Mon Feb 21 17:11:03 2005] [emerg] (22)Invalid argument: couldn't release the accept mutex
[Mon Feb 21 17:55:18 2005] [notice] Digest: generating secret for digest authentication ...
[Mon Feb 21 17:55:18 2005] [notice] Digest: done
[Mon Feb 21 17:55:19 2005] [warn] pid file /var/run/apache2.pid overwritten -- Unclean shutdown of previous Apache run?
[Mon Feb 21 17:55:19 2005] [notice] Apache/2.0.52 (Gentoo/Linux) mod_ssl/2.0.52 OpenSSL/0.9.7e PHP/4.3.10 configured -- resuming normal operations
|
After re-emerging apache once it complained about not being able to use some file in /var/cache/apache2/, I checked and the apache2 dir didn't exist, so I made it and it stopped complaining and started, that's possibly related, it was something regarding SSL and that mutex thing...
Does anyone know what's going on and how to fix this? I'm going crazy over complaints about the server being down. 
Thanks in advance. |
|
| Back to top |
|
 |
j-m
Retired Dev
Joined: 31 Oct 2004
Posts: 975
|
| Posted: Mon Feb 21, 2005 6:08 pm Post subject: |
|
|
You need /var/cache/apache2/ for mod_ssl. There should be no problems once this missing directory exists. Check the owner and rights.
| Code: |
drwxr-xr-x 2 apache apache 48 May 27 2004 apache2
|
|
|
| Back to top |
|
|
baak6
Tux's lil' helper
Joined: 30 Aug 2003
Posts: 141
|
| Posted: Mon Feb 21, 2005 6:10 pm Post subject: |
|
|
It was previously
| Code: |
drwxr-xr-x 2 apache root 72 feb 21 17:55 apache2
|
I changed it to apache:apache...I hope it works, thanks!  |
|
| Back to top |
|
|
midknight_gentoo
n00b
Joined: 13 Jul 2003
Posts: 22
|
| Posted: Tue Mar 01, 2005 12:58 am Post subject: |
|
|
my apache is doing the same thing now...
did this resolve the issue??? |
|
| Back to top |
|
|
baak6
Tux's lil' helper
Joined: 30 Aug 2003
Posts: 141
|
| Posted: Thu Mar 03, 2005 4:25 am Post subject: |
|
|
| It stopped crashing but now it started again...this i s driving me nuts. |
|
| Back to top |
|
|
tlarco
n00b
Joined: 26 Feb 2004
Posts: 7
|
| Posted: Mon Mar 07, 2005 4:24 pm Post subject: |
|
|
I also have the same thing happening. The error sounds like a limit on the number of logins or procs a UID is allowed to run and the resource becomes unavailable. I had some values defined in
/etc/security/limits.conf
that were suggested in the Gentoo Linux Security Guide (Code Listing 6.1). I am not sure if Apache uses these vaules, but the threshold being crossed could be set somewhere in apache itself and not in Linux.
Did you also apply hardening steps from the Gentoo Security Guide? This is the first time I have had this problem with the 10 Gentoo servers I use every day. The server in question is the only one with an entry in /etc/security/limits.conf. I removed the entries and will wait to see if it happens again. Hope this helps!
Tony |
|
| Back to top |
|
|
Herring42
Guru
Joined: 10 Mar 2004
Posts: 373
Location: Buckinghamshire
|
| Posted: Fri Mar 11, 2005 9:44 pm Post subject: |
|
|
I've been having these problems as well.
It became most noticable after I installed egroupware, especially when I connected kontact's contacts to it. It would crash every five minutes or so. Disconnecting kontact has improved things, but it still dies every so often.
_________________
"The problem with quotes on the internet is that it is difficult
to determine whether or not they are genuine." -- Abraham Lincoln |
|
| Back to top |
|
|
midknight_gentoo
n00b
Joined: 13 Jul 2003
Posts: 22
|
| Posted: Sat Mar 19, 2005 9:39 pm Post subject: |
|
|
i believe now i can trace mine to after i edited the limits.conf file for teh security howto... and it must be starving apache.. so it crashes.
is there a way in that file to allow apache more processes and not the whole system. |
|
| Back to top |
|
|
midknight_gentoo
n00b
Joined: 13 Jul 2003
Posts: 22
|
| Posted: Sun Mar 20, 2005 1:36 am Post subject: |
|
|
mine is now crashing all the damn time...
i had to disable ssl cause as soon as i go to horde it crashes right away
i removed all the stuff in the limits.conf as well....
this bothers me |
|
| Back to top |
|
|
geekmug
n00b
Joined: 02 Apr 2004
Posts: 39
|
| Posted: Mon Mar 21, 2005 10:24 pm Post subject: |
|
|
| I am replying because I am in search of the same answer.. my apache is hard limited to 15 processes and nothing I do with any of the limits stuff seems to change that. |
|
| Back to top |
|
|
geekmug
n00b
Joined: 02 Apr 2004
Posts: 39
|
|
| Back to top |
|
|
Ubnormal
n00b
Joined: 20 Aug 2004
Posts: 17
|
| Posted: Thu May 05, 2005 10:07 pm Post subject: Resource temporarily unavailable: setuid: unable to change t |
|
|
Hi all!
I have the same problem, my Apache 2.0.54 craches a few moments after startup with the following lines in the error_log:
| Code: |
[Fri May 06 00:56:46 2005] [notice] Apache configured -- resuming normal operations
[Fri May 06 00:56:46 2005] [info] Server built: May 5 2005 23:51:35
[Fri May 06 00:56:46 2005] [debug] worker.c(1632): AcceptMutex: sysvsem (default: sysvsem)
[Fri May 06 00:56:46 2005] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Fri May 06 00:56:46 2005] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Fri May 06 00:56:47 2005] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 81
[Fri May 06 00:56:47 2005] [alert] Child 3542 returned a Fatal error...\nApache is exiting!
[Fri May 06 00:56:47 2005] [warn] child process 3521 still did not exit, sending a SIGTERM
[Fri May 06 00:56:47 2005] [warn] child process 3523 still did not exit, sending a SIGTERM
[Fri May 06 00:56:47 2005] [warn] child process 3521 still did not exit, sending a SIGTERM
[Fri May 06 00:56:47 2005] [warn] child process 3523 still did not exit, sending a SIGTERM
[Fri May 06 00:56:48 2005] [warn] child process 3521 still did not exit, sending a SIGTERM
[Fri May 06 00:56:48 2005] [warn] child process 3523 still did not exit, sending a SIGTERM
[Fri May 06 00:56:53 2005] [error] child process 3521 still did not exit, sending a SIGKILL
[Fri May 06 00:56:53 2005] [error] child process 3523 still did not exit, sending a SIGKILL |
When i compile apache with mpm-prefork i have error_log with multiple "(22)Invalid argument: couldn't grab the accept mutex" lines.
I have 2Gb ram, 99% idle system, clear /etc/security/limits.conf, ulimit shows "unlimited". So "Resource temporarily unavailable" message in error_log looks very strange...
It seems /etc/init.d/apache doesn't use start-stop-daemon as described in the upper post but the problem very similar. Is there any step by step instruction to solve the problem? |
|
| Back to top |
|
|
krimhum
n00b
Joined: 18 Nov 2005
Posts: 1
|
| Posted: Fri Nov 18, 2005 7:05 pm Post subject: |
|
|
I'm having the same problems and am seeking a solution.
I read on one of the apache sites that changing your AcceptMutex value will fix the issue.
I changed mine to "AcceptMutex fcntl" and am going to wait a week and see if it still crashes.
Regards, |
|
| Back to top |
|
|
jensb
n00b
Joined: 20 Dec 2005
Posts: 2
|
| Posted: Tue Dec 20, 2005 12:53 am Post subject: apache not using pam to fork |
|
|
Hi,
I had the same problem with mysql. It also used to many processes. Even though I changed the values for
| Code: | | mysql hard nproc XX |
in /etc/security/limits.conf didn't result in a change of behaviour, which I could verify in the logfiles. I use hardened sources with logging of resource violations enabled. The log still showed the same limit of YY processes that was violated To narrow down the problem I changed the wildcard entry for all users
to a different value (ZZ) than before. Now after crashing mysql again the log showed ZZ processes as violated limit. That means that this file was the reason for the behaviour. Still I didn't understand why the system didn't see XX as limit since mysql ran as user mysql. The reason is - as mentioned earlier -
https://bugs.gentoo.org/show_bug.cgi?id=64700
When mysql is started the user is switched from root to mysql. This switch does not query the pam. I downloaded the diffs provided by the devolopers that took care of this matter and recompiled the start-stop-daemom with those changes. The start-stop-daemon starts mysql and with the new version querys the pam for the userchange. That way all the limits are set correctly.
A similar problem applies to apache, I think. The root process spawns child processes that run under the uid of the configured user (i.e. apache) but does not query the pam while doing that and therefore no limits can be set. Maybe that is also here the reason for this error.
Jens |
|
| Back to top |
|
|
jensb
n00b
Joined: 20 Dec 2005
Posts: 2
|
| Posted: Tue Dec 20, 2005 9:55 pm Post subject: solution |
|
|
For me the a little change in /etc/init.d/apache2 did the trick:
| Code: |
start() {
checkconfig || return 1
ebegin "Starting apache2"
[ -f /var/log/apache2/ssl_scache ] && rm /var/log/apache2/ssl_scache
[ -f /usr/lib/apache2/build/envvars ] && . /usr/lib/apache2/build/envvars
ulimit -u XX
${APACHE2} ${APACHE2_OPTS} -k start
eend $?
}
|
I added the line: ulimit -u XX which limits the number of processes per user.
Limits don't apply for root; they are shown though when using
but they have no effect. I guess these limits are now set for the session, so that all the child processes even though they're not root inherit these settings. Using this method work so far for me and now its possible to apply all the other limits like cpu-time, file-size etc. too.
Jens |
|
| Back to top |
|
|
Rainbow goblin
Tux's lil' helper
Joined: 29 Feb 2004
Posts: 132
|
| Posted: Tue Apr 25, 2006 8:09 am Post subject: |
|
|
You me have rescueed, thanks!  |
|
| Back to top |
|
|
padde
n00b
Joined: 19 Jan 2005
Posts: 54
Location: Germany
|
| Posted: Wed Dec 20, 2006 9:22 am Post subject: |
|
|
Then - what's the solution? Will there be an update? I'm experiencing this error since today (running subversion on my apache since yesterday, so I think this has something to do with it).
http://rafb.net/paste/results/N94yxi51.html
I'm feeling _quite_ uncomfortable with a crashing webserver... |
|
| Back to top |
|
|
padde
n00b
Joined: 19 Jan 2005
Posts: 54
Location: Germany
|
| Posted: Wed Jan 30, 2008 1:40 pm Post subject: again! |
|
|
It happened again today. After a subversion update (Tue Jan 29 02:20:13 2008 >>> dev-util/subversion-1.4.6). I didn't restart apache after the update (which is probably the cause), and a few hours later during a (harmless) attack consisting of around 10-20 accesses/second to some non-existing php scripts the server went down with the following log entries:
| Code: |
...
[Wed Jan 30 12:05:27 2008] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 81
[Wed Jan 30 12:05:27 2008] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 81
[Wed Jan 30 12:05:27 2008] [alert] Child 4646 returned a Fatal error... Apache is exiting!
[Wed Jan 30 12:05:27 2008] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 81
[Wed Jan 30 12:05:28 2008] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Wed Jan 30 12:05:28 2008] [emerg] (22)Invalid argument: couldn't grab the accept mutex
...
|
Funny thing was that I immediately found this thread (and my own >1 year old post)  |
|
| Back to top |
|
|
|
Networking & Security
