Apache2 is eating all my ram&swap [SOLVED]

[zark]

I'm having this recurrent issue with Apache2 on my gentoo server...

It's acting fine, then all of a sudden it starts taking ram and ram, then page memory increases .. until it can't run cgi anymore ...
if i do a /etc/init.d/apache2 stop and then i ps .. i see still LOTS of apache2 processes ... (and if i do a start it won't because port is still binded)

so i have to do a killall apache2 (4-5 times in a row) then i can start the service again ... and the whole thing is fine for another 4 days ..

this happens almost twice a week (mrtg graphs show a pattern)


apache2 config was the default ( talking about that MPM stuff), i changed the settings (turned down) but the problem persists ... seems like it opens threads but don't close them :s

[SilverOne]

Perhaps you're the victim of a (D)DOS attack?

Do your MRTG graphs show number of connections to Apache? Is there a significant increase in connections just before the "crash"?

[zark]

I thought about a DOS attack too ... but i kind of doubt it considering the 'regularity' in which it occurs ...

also the last time it happened (i installed mod_watch last thursday), on saturday, i saw it started happening before our daily traffic increased (our traffic comes 90% from europe, so its always 16h-20h). But why didn't it do that the day before ... the amount of hits are very similar....

i'm really without a clue .... and since this is production server, and we have around 10websites hosted on it ... i'm not really wanting to switch back to 1.3 (though i think about it more and more everyday)

Also.. it's not a SUDDEN thing ... it occurs over 2-3 hours ... then sometimes stabilizes a bit, then drops again ...

we get around : (copypaste from mrtg-modwatch)
Max Hits: 14.3 kH/h
Average Hits: 3809.0 H/h
Current Hits: 8844.0 H/h


so it's not a heavily used server .. ( + it's a 2.8E HT p4 with 1gb ddr )

[ai]

hmm looking for the source of this, try to monitor : cgi scripts, php scripts, and your net traffic (not only the average www - this high load can be coused by many errorous connections or smth like that). I believe that studying the logs (/var/log/apache2) might be helpfull. Also i strongly advice u to install & properly configure mod_security as it's a great and handy tool and can provide additional logs.

Looking forward to seeing some suspicious logs, ai [;
_________________
just nothing

[zark]

[zark]

i check with glsa -l


and found these

200403-04 [U] Multiple security vulnerabilities in Apache 2 ( net-www/apache )
200411-18 [U] Apache 2.0: Denial of Service by memory consumption ( net-www/apache )



if it says U .. means i should apply them ?

[ai]

[zark]

mmh, i just realised that apache2 is actually using WORKER MPM ... i thought the default was prefork and that worker was experimental ....

i reduced the 256 MaxClients to 25 and memory usage seems under control.



[ebuild R ] net-www/apache-2.0.52-r1 +berkdb -debug -doc +gdbm +ipv6 -ldap +ssl -static +threads 0 kB

[zark]

okay .. that didn't solve the problem ...

it ate thru 200mb of swap in 10minutes.



problem is that all my logs are vhost dependant ... so finding anything suspicious around the time of attack is kinda hard ....

i'll check out that mod_security thing

[Quinten]

I have exactly the same problem. System runs fine for a day or two and then grinds to a halt. Top shows me that apache2 uses all available swapspace and memory, and a stop and start frees it all up... until the next time.

[zark]

okay ...


i found the solution ... i recompiled apache,


USE "-threads" emerge apache


and that compiled it without the threads options, and it's been running smoothly for over a week now !