Mysql dies after some time ( too many connection )

[dhil]

Hey Guys,

I have a webserver running with apache and mysql .. it has been working fine for over a year now but now suddenly the mysql server just dies.. i have to restart the server and then start up the service .. after a 10 - 15 mins it dies again .. can anyone help me locate the problem ... where should i start looking ?

I ran a top to see if my server is comprimised .. does any of these applications look odd ?

top - 09:03:19 up 22 min, 1 user, load average: 1.19, 1.87, 2.06
Tasks: 191 total, 1 running, 188 sleeping, 0 stopped, 2 zombie
Cpu(s): 8.6% us, 1.0% sy, 0.0% ni, 83.7% id, 6.6% wa, 0.2% hi, 0.0% si
Mem: 1938144k total, 507260k used, 1430884k free, 43396k buffers
Swap: 503800k total, 0k used, 503800k free, 223768k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4540 nobody 20 0 7324 2620 1384 S 1 0.1 0:00.06 httpd
8762 root 20 0 2256 1148 832 R 1 0.1 0:00.02 top
1 root 20 0 1480 500 448 S 0 0.0 0:01.14 init
2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:00.06 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/1
6 root 15 -5 0 0 0 S 0 0.0 0:00.20 ksoftirqd/1
7 root 15 -5 0 0 0 S 0 0.0 0:00.04 events/0
8 root 15 -5 0 0 0 S 0 0.0 0:00.04 events/1
9 root 15 -5 0 0 0 S 0 0.0 0:00.02 khelper
114 root 15 -5 0 0 0 S 0 0.0 0:00.00 kintegrityd/0
115 root 15 -5 0 0 0 S 0 0.0 0:00.00 kintegrityd/1
116 root 15 -5 0 0 0 S 0 0.0 0:00.00 kblockd/0
117 root 15 -5 0 0 0 S 0 0.0 0:00.00 kblockd/1
118 root 15 -5 0 0 0 S 0 0.0 0:00.00 kacpid
119 root 15 -5 0 0 0 S 0 0.0 0:00.00 kacpi_notify
207 root 15 -5 0 0 0 S 0 0.0 0:00.00 ata/0
208 root 15 -5 0 0 0 S 0 0.0 0:00.00 ata/1
209 root 15 -5 0 0 0 S 0 0.0 0:00.00 ata_aux
210 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksuspend_usbd
215 root 15 -5 0 0 0 S 0 0.0 0:00.00 khubd
218 root 15 -5 0 0 0 S 0 0.0 0:00.00 kseriod
270 root 20 0 0 0 0 S 0 0.0 0:00.00 pdflush
271 root 20 0 0 0 0 S 0 0.0 0:00.02 pdflush
272 root 15 -5 0 0 0 S 0 0.0 0:00.00 kswapd0
273 root 15 -5 0 0 0 S 0 0.0 0:00.00 aio/0
274 root 15 -5 0 0 0 S 0 0.0 0:00.00 aio/1
277 root 15 -5 0 0 0 S 0 0.0 0:00.00 nfsiod
278 root 15 -5 0 0 0 S 0 0.0 0:00.00 cifsoplockd


UPDATE
--------------------

after just 5 mins it says too many connections .. thats the problem i guess.. i am unable to understand how it can have too many connections after just 5 mins of opening the site ?? Normally i used to have 200 - 250 concurrent users .. without anyproblem...


update

---------------

i think i found the error ...someone is attacking my server and i cant do anything

mysqladmin -i2 processlist status

shows that soemthign is scanning my sql database with queries like this

SELECT * FROM ibf_topics WHERE tid=6508
select f.name, f.id as forum_id, t.tid, t.title, t.description,

and it continues to scan my whole site ... any suggestions !


I will keep on updating this topic so if any experts would like to give some suggestions

[dhil]

i think i found the error ...someone is attacking my server and i cant do anything

mysqladmin -i2 processlist status

shows that soemthign is scanning my sql database with queries like this

SELECT * FROM ibf_topics WHERE tid=6508
select f.name, f.id as forum_id, t.tid, t.title, t.description,

and it continues to scan my whole site ... any suggestions !

[Earthwings]

Is it a single IP address where the requests come from? Then block that for a start (using iptables for example).
_________________
KDE

[dhil]

how can i find out which ip does this query ?

Now i somehow suspect it can be a broken sql string as well .. so i am running a

mysqlcheck -A -r -p


The last query was

Uptime: 771 Threads: 1 Questions: 2943 Slow queries: 2 Opens: 842 Flush tables: 1 Open tables: 64 Queries per second avg: 3.817

and it continues :s

[Earthwings]

It may also be a search engine crawler or similar "service" trying to index your forum. Try to block search engines using robots.txt (search for howtos on that).

The IP addresses can be found in the apache log files (e.g. /var/log/apache2/access.log).
_________________
KDE

[dhil]

i used to get crawlers at my site, both google, askalex and some other crawlers often come to my site for indexing .. never been a problem with that .. i will look into the apache logs .. once the table repair is done

[Earthwings]

Why do you need to wait for MySQL to repair things? The apache log files are simple files independent of MySQL.
_________________
KDE

[dhil]

because the putty is busy thats why .. ok now i cancelled the repair thing ... i have only turned on error logging

the path u specified doesnt exsist on my server .. its a OVH gentoo :S

[Earthwings]