View previous topic :: View next topic |
Author |
Message |
danny_b85 n00b


Joined: 16 Jun 2008 Posts: 10
|
Posted: Thu Jul 17, 2008 5:50 pm Post subject: Recompile iptables to block a MAC address[SOLVED] |
|
|
Hi all,
I've been having some problems these last couple of days with some "benevolent" people who have been attacking a couple of servers I am working on. The attacks are similar to what I've found here, without the ssh part, which is pretty much secured with public/private authentication, login is permitted from a single local IP, with a ten count on the number of tries before getting banned and the passphrase being a 25 character license key of a program.
But let's not digress, the problem is as follows, the attacks consist of either SYN flooding, for which I had previously activated tcp_syn_cookies, either by attacking different ports for the usual things, MySQL, VNC, Samba, etc. Now, with the SYN flood, let's say the syn_cookies help, but I don't know how much ... As for the rest of the ports they attack, the server is one of the servers whose primary purpose is to keep alive a webhosting company, so that implies having to open a number of ports for different needs, and my concern is the HTTP/HTTPS 80/443 ports that have to remain open for inbound traffic. These ports get pounded with packets, and blocking the originating IPs has done little good, as the attackers generate random IP addresses, and by blocking the ones I get the hits from won't help for long either because they try from new ones. The "newest" thing they've done was to send packets with the source IP of the neighbouring servers, which need to communicate to one another.
Even at this point, let's say it wouldn't bother me as much that they're knocking away at the heavens door, but the Apache webserver has crashed on multiple occasions ever since the attacks began (I'm using Apache 2.0). What I've seen is that they manage to create multiple Apache child processes, so it jumps from a normal 30-40 processes to over 100-150 processes, at which point the websites stop loading and Apache wouldn't come back to life no matter what I've tried. Looking through it's logs, I got this:
Code: | [Wed Jul 16 22:02:04 2008] [emerg] (28)No space left on device: Couldn't create accept lock
[Thu Jul 17 10:31:03 2008] [crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock
|
Which led me to a website that suggested this:
Code: | ipcs -s | grep nobody | perl -e 'while (<STDIN>) { @a=split(/\s+/); print `ipcrm sem $a[1]`}' |
Which restored functionality of Apache for a brief period of time. Now curing the effect won't cure the "disease", so for a more permanent solution I wanted to block all packets that originated from (get this) the same MAC address. Yes, the same MAC address on all packets, but there's a catch, I couldn't do it with iptables because the MAC address of the incoming packets has a length larger than 48 bits. Sample blocked packets:
Code: | Jul 17 17:18:13 server* kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:00:00 SRC=IP DST=MYSERVER LEN=40 TOS=0x00 PREC=0x00 TTL=95 ID=256 PROTO=TCP SPT=6000 DPT=1433 WINDOW=16384 RES=0x00 SYN URGP=0 |
So my first idea was to recompile iptables from source, find the regex which is looking for a match in the MAC address and replace it, should be something like this:
Code: | /^[A-z0-9][A-z0-9]\:\{5\}[A-z0-9][A-z0-9]$/ |
However I couldn't find in the source code what I've been searching for so I thought I'd come to ask if some knows how this can be done.
Also, if anybody has any good ideas about my problem, or even a different approach on how to handle this issue altogether, I'd really appreciate it.
Thanks in advance.
Last edited by danny_b85 on Tue Aug 12, 2008 7:36 am; edited 1 time in total |
|
Back to top |
|
 |
SnakeByte Apprentice


Joined: 04 Oct 2002 Posts: 177 Location: Europe - Germany
|
Posted: Thu Jul 17, 2008 10:35 pm Post subject: |
|
|
Hi,
some random things.
Can you post the physical connection setup of your machines,
for example, is there a single gateway to the internet,
and the ports which have to be open on which machine.
Maybe your iptales setup ( with real ips / hostnames removed )
could be some start for suggestions.
Maybe it is be possible to block MAC address zero?
best regards |
|
Back to top |
|
 |
Hu Administrator

Joined: 06 Mar 2007 Posts: 23131
|
Posted: Fri Jul 18, 2008 1:30 am Post subject: |
|
|
Is the MAC address actually all zero in the log or is it that way because you blanked out the real MAC address?
The MAC address showed up as 14 octets because the kernel prints the entire layer 2 frame and labels it MAC. The layer 2 frame consists of the destination MAC address, source MAC address, and layer 3 type.
Be aware that the MAC address available to you is the address of the next hop, not the ultimate peer. As an example, take a packet capture while you ping two distinct Internet destinations, such as Google and Yahoo. Look at the MAC addresses in the returned ICMP echo responses. Thus, filtering by MAC address is really only useful if you are using it to whitelist traffic that comes from a directly attached system which is known not to route malicious traffic.
SYN floods should not be visible to Apache, since the connection must complete before the server is notified. In a SYN flood, the attacker uses spoofed source addresses that never ACK the SYN|ACK. The attackers must also be opening full connections to cause problems for Apache. |
|
Back to top |
|
 |
danny_b85 n00b


Joined: 16 Jun 2008 Posts: 10
|
Posted: Mon Jul 28, 2008 3:43 pm Post subject: |
|
|
Quote: | Maybe your iptales setup ( with real ips / hostnames removed )
could be some start for suggestions. |
Code: | iptables -L -n
Chain INPUT (policy DROP)
target prot opt source destination
acctboth all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT udp -- ns-address 0.0.0.0/0 udp spts:1024:65535 dpt:53
ACCEPT tcp -- ns-address 0.0.0.0/0 tcp spts:1024:65535 dpt:53
ACCEPT udp -- ns-address 0.0.0.0/0 udp spt:53 dpts:1024:65535
ACCEPT tcp -- ns-address 0.0.0.0/0 tcp spt:53 dpts:1024:65535
ACCEPT udp -- ns-address 0.0.0.0/0 udp spt:53 dpt:53
ACCEPT udp -- ns2-address 0.0.0.0/0 udp spts:1024:65535 dpt:53
ACCEPT tcp -- ns2-address 0.0.0.0/0 tcp spts:1024:65535 dpt:53
ACCEPT udp -- ns2-address 0.0.0.0/0 udp spt:53 dpts:1024:65535
ACCEPT tcp -- ns2-address 0.0.0.0/0 tcp spt:53 dpts:1024:65535
ACCEPT udp -- ns2-address 0.0.0.0/0 udp spt:53 dpt:53
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
INVALID tcp -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:20
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:21
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:25
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:53
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:80
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:110
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:143
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:443
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:465
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:953
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:993
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:995
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:2077
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:2078
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:2082
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:2083
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:2086
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:2087
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:2095
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:2096
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:3306
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:28448
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:20
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:21
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:53
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:953
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 limit: avg 1/sec burst 5
LOGDROPIN all -- 0.0.0.0/0 0.0.0.0/0
Chain FORWARD (policy DROP)
target prot opt source destination
Chain OUTPUT (policy DROP)
target prot opt source destination
acctboth all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT udp -- 0.0.0.0/0 ns-address udp spt:53 dpts:1024:65535
ACCEPT tcp -- 0.0.0.0/0 ns-address tcp spt:53 dpts:1024:65535
ACCEPT udp -- 0.0.0.0/0 ns-address udp spts:1024:65535 dpt:53
ACCEPT tcp -- 0.0.0.0/0 ns-address tcp spts:1024:65535 dpt:53
ACCEPT tcp -- 0.0.0.0/0 ns-address tcp spt:53 dpt:53
ACCEPT udp -- 0.0.0.0/0 ns2-address udp spt:53 dpts:1024:65535
ACCEPT tcp -- 0.0.0.0/0 ns2-address tcp spt:53 dpts:1024:65535
ACCEPT udp -- 0.0.0.0/0 ns2-address udp spts:1024:65535 dpt:53
ACCEPT tcp -- 0.0.0.0/0 ns2-address tcp spts:1024:65535 dpt:53
ACCEPT tcp -- 0.0.0.0/0 ns2-address tcp spt:53 dpt:53
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
INVALID tcp -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:20
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:21
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:25
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:37
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:43
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:53
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:80
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:110
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:113
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:443
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:587
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:873
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:953
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:2087
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:2089
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:2703
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:28448
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:3306
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:20
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:21
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:53
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:113
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:123
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:873
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:953
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:6277
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 limit: avg 1/sec burst 5
LOGDROPOUT all -- 0.0.0.0/0 0.0.0.0/0
Chain INVALID (2 references)
target prot opt source destination
INVDROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID
INVDROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x00
INVDROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x3F
INVDROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x03/0x03
INVDROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x06/0x06
INVDROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x05/0x05
INVDROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x11/0x01
INVDROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x18/0x08
INVDROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x30/0x20
INVDROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:!0x17/0x02 state NEW
Chain INVDROP (10 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0
Chain LOGDROPIN (1 references)
target prot opt source destination
DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:67
DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:67
DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:68
DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:68
DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:111
DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:111
DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:113
DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:113
DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpts:135:139
DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp dpts:135:139
DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:445
DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:445
DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:513
DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:513
DROP tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:520
DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:520
LOG tcp -- 0.0.0.0/0 0.0.0.0/0 limit: avg 30/min burst 5 LOG flags 0 level 4 prefix `Firewall: *TCP_IN Blocked* '
LOG udp -- 0.0.0.0/0 0.0.0.0/0 limit: avg 30/min burst 5 LOG flags 0 level 4 prefix `Firewall: *UDP_IN Blocked* '
LOG icmp -- 0.0.0.0/0 0.0.0.0/0 limit: avg 30/min burst 5 LOG flags 0 level 4 prefix `Firewall: *ICMP_IN Blocked* '
DROP all -- 0.0.0.0/0 0.0.0.0/0
Chain LOGDROPOUT (1 references)
target prot opt source destination
LOG tcp -- 0.0.0.0/0 0.0.0.0/0 limit: avg 30/min burst 5 LOG flags 0 level 4 prefix `Firewall: *TCP_OUT Blocked* '
LOG udp -- 0.0.0.0/0 0.0.0.0/0 limit: avg 30/min burst 5 LOG flags 0 level 4 prefix `Firewall: *UDP_OUT Blocked* '
LOG icmp -- 0.0.0.0/0 0.0.0.0/0 limit: avg 30/min burst 5 LOG flags 0 level 4 prefix `Firewall: *ICMP_OUT Blocked* '
DROP all -- 0.0.0.0/0 0.0.0.0/0
Chain acctboth (2 references)
target prot opt source destination
tcp -- server-ip 0.0.0.0/0 tcp dpt:80
tcp -- 0.0.0.0/0 server-ip tcp spt:80
tcp -- server-ip 0.0.0.0/0 tcp dpt:25
tcp -- 0.0.0.0/0 server-ip tcp spt:25
tcp -- server-ip 0.0.0.0/0 tcp dpt:110
tcp -- 0.0.0.0/0 server-ip tcp spt:110
icmp -- server-ip 0.0.0.0/0
icmp -- 0.0.0.0/0 server-ip
tcp -- server-ip 0.0.0.0/0
tcp -- 0.0.0.0/0 server-ip
udp -- server-ip 0.0.0.0/0
udp -- 0.0.0.0/0 server-ip
all -- server-ip 0.0.0.0/0
all -- 0.0.0.0/0 server-ip |
Quote: | SYN floods should not be visible to Apache, since the connection must complete before the server is notified. |
Yes, the three-way handshake must be completed, otherwise the connection is in a half-open state.
Quote: | The attackers must also be opening full connections to cause problems for Apache. |
OK, but how do I differentiate between the normal connections that originate from people wanting to visit the websites we host, and the attackers opening up multiple connections to Apache, I mean looking at the Apache logs ... hard to tell these apart.
The problem remains as I've explained it the first time, Apache spawns multiple children until it cannot process any more and looking at the logs won't help much either. My solution (although it's really not a viable solution) was to run the command mentioned in my first post, with the ipcs, deleting resources from Apache once every hour from a cron job.
Putting aside my crappy solution, any good suggestions?
Thank you in advance.
p.s.: Quote: | The MAC address showed up as 14 octets because the kernel prints the entire layer 2 frame and labels it MAC. The layer 2 frame consists of the destination MAC address, source MAC address, and layer 3 type.
Be aware that the MAC address available to you is the address of the next hop, not the ultimate peer. |
10x for clearing up that for me, I didn't know that. |
|
Back to top |
|
 |
Hu Administrator

Joined: 06 Mar 2007 Posts: 23131
|
Posted: Mon Jul 28, 2008 10:48 pm Post subject: |
|
|
You might be able to use syncookies to combat the forged SYN packets that never complete a connection.
Since the attacks on Apache require a full three way handshake, the attackers must have control of the IP addresses that they use to initiate those connections. Theoretically, you could block them using Netfilter so that Apache does not receive new connections from those addresses. That said, it is possible that the attack is sufficiently heavy that you will not be able to identify and ban a significant portion of the attacking systems. If you cannot tell from the logs which connections are legitimate and which are malicious, this will be very hard. Without detailed knowledge of both legitimate behavior and malicious behavior, it is difficult to speculate on ways to distinguish them. |
|
Back to top |
|
 |
danny_b85 n00b


Joined: 16 Jun 2008 Posts: 10
|
Posted: Tue Aug 12, 2008 7:40 am Post subject: |
|
|
Hi, I found the problem, don't know why I didn't look into it earlier, I was using Apache 2.0.63, and it had a security vulnerability which was leading to a DoS attack (what a shock, that's what I was experiencing ... ).
I've upgraded to Apache 2.2.9, had some problems with the PHP not working correctly after the upgrade, solved that issue and now the server/s are working fine.
Hu, thanks a lot for your insight into the problem and for your help. |
|
Back to top |
|
 |
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|