View previous topic :: View next topic |
Author |
Message |
Crisis l33t
Joined: 10 Feb 2003 Posts: 613 Location: Portland, OR
|
Posted: Mon Jul 12, 2004 2:27 pm Post subject: cron spawning many instances, machine locking |
|
|
Well I am having a problem on one of my gentoo boxes as follows.
It seems that CRON keeps running and leaving a process open to the point where there could be hundres of cron processes running. ONce it does this the machine will not respond to just about any command, including even shutdown -r now. Only way to fix it is to physically reboot the machine.
Now I am not sure if the cron instances are causing this, or if they are a symptom of a different problem. They are the only thing I can see that looks odd.
Here is a snippit of ps aux:
Code: | root 11082 0.0 0.0 1520 644 ? S 08:10 0:00 /USR/SBIN/CRON
root 11083 0.0 0.1 1856 824 ? S 08:10 0:00 /bin/bash -c test -x /usr/sbin/run-crons && /usr/sbin/run-crons
root 11084 0.0 0.1 1856 848 ? D 08:10 0:00 /bin/sh /usr/sbin/run-crons
root 11087 0.0 0.0 1520 644 ? S 08:20 0:00 /USR/SBIN/CRON
root 11088 0.0 0.1 1856 824 ? S 08:20 0:00 /bin/bash -c test -x /usr/sbin/run-crons && /usr/sbin/run-crons
root 11089 0.0 0.1 1856 848 ? D 08:20 0:00 /bin/sh /usr/sbin/run-crons
root 11094 0.0 0.0 1520 644 ? S 08:30 0:00 /USR/SBIN/CRON
root 11095 0.0 0.1 1856 824 ? S 08:30 0:00 /bin/bash -c test -x /usr/sbin/run-crons && /usr/sbin/run-crons
root 11096 0.0 0.1 1856 848 ? D 08:30 0:00 /bin/sh /usr/sbin/run-crons
root 11100 0.0 0.0 1520 644 ? S 08:40 0:00 /USR/SBIN/CRON
root 11101 0.0 0.1 1856 824 ? S 08:40 0:00 /bin/bash -c test -x /usr/sbin/run-crons && /usr/sbin/run-crons
root 11102 0.0 0.1 1856 848 ? D 08:40 0:00 /bin/sh /usr/sbin/run-crons
root 11105 0.0 0.0 1520 644 ? S 08:50 0:00 /USR/SBIN/CRON
root 11106 0.0 0.1 1856 824 ? S 08:50 0:00 /bin/bash -c test -x /usr/sbin/run-crons && /usr/sbin/run-crons
root 11107 0.0 0.1 1856 848 ? D 08:50 0:00 /bin/sh /usr/sbin/run-crons
root 11113 0.0 0.0 1520 644 ? S 09:00 0:00 /USR/SBIN/CRON
root 11115 0.0 0.0 1520 644 ? S 09:00 0:00 /USR/SBIN/CRON
root 11116 0.0 0.0 1328 356 ? D 09:00 0:00 rm -f /var/spool/cron/lastrun/cron.hourly
root 11114 0.0 0.1 1856 824 ? S 09:00 0:00 /bin/bash -c test -x /usr/sbin/run-crons && /usr/sbin/run-crons
root 11117 0.0 0.1 1856 848 ? D 09:00 0:00 /bin/sh /usr/sbin/run-crons
root 11122 0.0 0.0 1520 644 ? S 09:10 0:00 /USR/SBIN/CRON
root 11123 0.0 0.1 1856 824 ? S 09:10 0:00 /bin/bash -c test -x /usr/sbin/run-crons && /usr/sbin/run-crons
root 11124 0.0 0.1 1856 848 ? D 09:10 0:00 /bin/sh /usr/sbin/run-crons
root 11129 0.0 0.0 1520 644 ? S 09:20 0:00 /USR/SBIN/CRON
root 11130 0.0 0.1 1856 824 ? S 09:20 0:00 /bin/bash -c test -x /usr/sbin/run-crons && /usr/sbin/run-crons
root 11131 0.0 0.1 1856 848 ? D 09:20 0:00 /bin/sh /usr/sbin/run-crons
root 11135 0.0 0.0 1520 644 ? S 09:30 0:00 /USR/SBIN/CRON
root 11136 0.0 0.1 1856 824 ? S 09:30 0:00 /bin/bash -c test -x /usr/sbin/run-crons && /usr/sbin/run-crons
root 11137 0.0 0.1 1856 848 ? D 09:30 0:00 /bin/sh /usr/sbin/run-crons
root 11141 0.0 0.0 1520 644 ? S 09:40 0:00 /USR/SBIN/CRON
root 11142 0.0 0.1 1856 824 ? S 09:40 0:00 /bin/bash -c test -x /usr/sbin/run-crons && /usr/sbin/run-crons
root 11143 0.0 0.1 1856 848 ? D 09:40 0:00 /bin/sh /usr/sbin/run-crons
root 11148 0.0 0.0 1520 644 ? S 09:50 0:00 /USR/SBIN/CRON
root 11149 0.0 0.1 1856 824 ? S 09:50 0:00 /bin/bash -c test -x /usr/sbin/run-crons && /usr/sbin/run-crons
root 11150 0.0 0.1 1856 848 ? D 09:50 0:00 /bin/sh /usr/sbin/run-crons
root 11153 0.0 0.0 1520 644 ? S 10:00 0:00 /USR/SBIN/CRON
root 11155 0.0 0.0 1520 644 ? S 10:00 0:00 /USR/SBIN/CRON
root 11156 0.0 0.0 1328 356 ? D 10:00 0:00 rm -f /var/spool/cron/lastrun/cron.hourly
root 11154 0.0 0.1 1856 824 ? S 10:00 0:00 /bin/bash -c test -x /usr/sbin/run-crons && /usr/sbin/run-crons
root 11157 0.0 0.1 1856 848 ? D 10:00 0:00 /bin/sh /usr/sbin/run-crons
root 11162 0.0 0.0 1520 644 ? S 10:10 0:00 /USR/SBIN/CRON
root 11163 0.0 0.1 1856 824 ? S 10:10 0:00 /bin/bash -c test -x /usr/sbin/run-crons && /usr/sbin/run-crons
root 11164 0.0 0.1 1856 848 ? D 10:10 0:00 /bin/sh /usr/sbin/run-crons |
Any ideas? |
|
Back to top |
|
|
adaptr Watchman
Joined: 06 Oct 2002 Posts: 6730 Location: Rotterdam, Netherlands
|
Posted: Mon Jul 12, 2004 2:37 pm Post subject: |
|
|
Well, run-crons runs the appropriate jobs in /etc/cron.hourly|daily|weekly|monthly, so maybe one of those jobs is faulty ?
Or a job takes too long to complete, but probably the former... _________________ >>> emerge (3 of 7) mcse/70-293 to /
Essential tools: gentoolkit eix profuse screen |
|
Back to top |
|
|
Crisis l33t
Joined: 10 Feb 2003 Posts: 613 Location: Portland, OR
|
Posted: Mon Jul 12, 2004 2:50 pm Post subject: |
|
|
The cron jobs on this box are really standard, haven't edited them:
Code: | root@valhalla etc # ls cron.*
cron.d:
cron.daily:
makewhatis.cron slocate
cron.hourly:
cron.monthly:
cron.weekly:
root@valhalla etc # cd cron.daily/
root@valhalla cron.daily # ls -lahtr
total 11K
-rwxr-xr-x 1 root root 126 May 27 21:39 makewhatis.cron
-rwxr-xr-x 1 root root 152 May 27 22:41 slocate
drwxr-xr-x 2 root root 128 May 27 22:41 .
-rw-r--r-- 1 root root 0 Jun 10 08:49 .keep
drwxr-xr-x 51 root root 3.5K Jul 12 04:26 ..
root@valhalla cron.daily # cat makewhatis.cron
#!/bin/sh
# this is part of the man package
# it updates the searchly database
# for manpages
/bin/nice /usr/sbin/makewhatis
root@valhalla cron.daily # cat slocate
#! /bin/sh
if [ -x /usr/bin/updatedb ]
then
if [ -f /etc/updatedb.conf ]
then
nice /usr/bin/updatedb
else
nice /usr/bin/updatedb -f proc
fi
fi
|
|
|
Back to top |
|
|
adaptr Watchman
Joined: 06 Oct 2002 Posts: 6730 Location: Rotterdam, Netherlands
|
Posted: Tue Jul 13, 2004 12:21 pm Post subject: |
|
|
Hmmm.
This kind of behaviour is usually related to processes respawning too fast - either from a super-server (like apache or mysql) or from init.
Have you tried replacing it with another cron ? _________________ >>> emerge (3 of 7) mcse/70-293 to /
Essential tools: gentoolkit eix profuse screen |
|
Back to top |
|
|
Crisis l33t
Joined: 10 Feb 2003 Posts: 613 Location: Portland, OR
|
Posted: Fri Jul 16, 2004 1:58 pm Post subject: |
|
|
I took off vixie-cron and replaced it with dcron, we shall see if this happens again. |
|
Back to top |
|
|
Crisis l33t
Joined: 10 Feb 2003 Posts: 613 Location: Portland, OR
|
Posted: Thu Jul 29, 2004 6:10 pm Post subject: |
|
|
I'm writing this one off to a bad motherboard as the problems are increasing and I'm seeing random coruption etc. |
|
Back to top |
|
|
adaptr Watchman
Joined: 06 Oct 2002 Posts: 6730 Location: Rotterdam, Netherlands
|
Posted: Thu Jul 29, 2004 7:09 pm Post subject: |
|
|
Before you throw it away in disgust
Try running stresslinux on it overnight.
Random failures are also frequently caused by bad power supplies - which are far more common than faulty motherboards. _________________ >>> emerge (3 of 7) mcse/70-293 to /
Essential tools: gentoolkit eix profuse screen |
|
Back to top |
|
|
Crisis l33t
Joined: 10 Feb 2003 Posts: 613 Location: Portland, OR
|
Posted: Thu Jul 29, 2004 8:44 pm Post subject: |
|
|
True, but there are a lot more details not mentioned here regarding the hardware in this machine that had not been mentioned.
Namely that this is the third hardrive this machine has had, all of which have eventually started to do random corupt writes after time which leads me to believe it is an IDE controller problem and not bad hard drives in the past.
Also the 3v and 5v rails ar ereading fine on this PSU so I am being led to believe it is a bad IDE controller.
Hopefully in the next week or so I can try a different mobo and tell for sure. |
|
Back to top |
|
|
Tyrian n00b
Joined: 16 Sep 2003 Posts: 5
|
Posted: Sun Aug 01, 2004 8:20 pm Post subject: |
|
|
Hm. I'm having these same problems on a Dell 4600. Load was over 600 when I logged in yesterday.
Killing all the rogue procs and restarting cron seemed to fix it for a while, but at some point this morning, my box locked up. None of the logs indicate anything super odd, so I'm still trying to figure out where this is coming from. |
|
Back to top |
|
|
|