| View previous topic :: View next topic |
| Author |
Message |
mimosinnet
l33t
Joined: 10 Aug 2006
Posts: 719
Location: Barcelona, Spain
|
 Posted: Mon Oct 24, 2011 2:28 pm Post subject: experience or alternatives to iplist and peerguardian? |
 |
|
I have found these two packages, iplist and PeerGuardian, that block connections to and from a specified range of hosts. It is possible to find some ebuilds for these packages (in this thread for PeerGuardian and in the sunrise and smithdanea overlays for iplist). Nevertheless, I wonder if anybody has had some experiencie with these packages, which one would you recommend and if there is some other alternative.
Thanks very much for your attention. |
|
| Back to top |
|
 |
Bones McCracker
Veteran
Joined: 14 Mar 2006
Posts: 1611
Location: U.S.A.
|
| Posted: Mon Oct 24, 2011 9:17 pm Post subject: |
|
|
iptables, or any firewall package
_________________
| patrix_neo wrote: | The human thought: I cannot win.
The ratbrain in me : I can only go forward and that's it. |
|
|
| Back to top |
|
|
mimosinnet
l33t
Joined: 10 Aug 2006
Posts: 719
Location: Barcelona, Spain
|
| Posted: Tue Oct 25, 2011 1:45 pm Post subject: |
|
|
| BoneKracker wrote: | | iptables, or any firewall package |
I am sorry I have not made the question clear enough. As I understand it, PeerGuardian and iplist get the list of hosts to ban from some sources (like, for example, http://www.bluetack.co.uk/config/ads-trackers-and-bad-pr0n.gz) and create the appropriate iptables rules from it. You could, for example, use iplist to ban certain countries.
I have received different attacks in the server, and I wonder about the suitability of having a pre-defined set of IPs to block. Also, it looks like a huge task to identify the offending IPs and manually adding iptables rules for it. I wonder if this could be done by just using iptables.
I have found fail2ban as an alternative, as it bans the IPs based on the actual behaviour of the IP (number of password failures) instead of a preset list.
Any hints would be appreciated. |
|
| Back to top |
|
|
Bones McCracker
Veteran
Joined: 14 Mar 2006
Posts: 1611
Location: U.S.A.
|
| Posted: Wed Oct 26, 2011 12:52 am Post subject: |
|
|
Oh, okay. I didn't understand that was what you were looking for.
You can still do it pretty easily with just iptables (using its ipset capabilities) and a simple script. No need for a separate product or some additional daemon running. The nice part is that you can use any source of information on the internet as input to what you want to block, and you aren't limited to some list you have to buy a subscription to.
For example, here is such a script that I've written (three versions of it, actually, each dynamically managing a different kind of block list, using information freely available on the Internet).
This first version of the script keeps my firewall blacklist up to date with Dshield's dynamic block list, which is updated every 15 minutes. It downloads the list, populates an ipset with the addresses, and replaces the previous version of itself in the running firewall without shutting it down. Using ipset is an extremely efficient way to manage blacklists or whitelists in a linux firewall, but it's not many people have caught on to it yet. So, while there are some packages that provide similar functionality, chances are that they're far less efficient. | Code: | #! /bin/bash
# /usr/local/sbin/block
# BoneKracker
# Rev. 14 August 2011
# Purpose: Load DShield.org Recommended Block List into an ipset in a running
# firewall. That list contains the networks from which the most malicious
# traffic is being reported by DShield participants.
# Notes: Call this from crontab. Feed updated every 15 minutes.
# netmask=24: dshield's list is all class C networks
# hashsize=64: default is 1024 but 64 is more than needed here
target="http://feeds.dshield.org/block.txt"
ipset_params="hash:ip --netmask 24 --hashsize 64"
filename=$(basename ${target})
firewall_ipset=${filename%.*} # ipset will be filename minus ext
data_dir="/var/tmp/${firewall_ipset}" # data directory will be same
data_file="${data_dir}/${filename}"
# if data directory does not exist, create it
/bin/mkdir -m 0750 ${data_dir} 2>/dev/null
# function to get modification time of the file in log-friendly format
get_timestamp() {
timestamp=$(/bin/date -r ${data_file} +%m/%d' '%R 2>/dev/null)
}
# file modification time on server is preserved during wget download
get_timestamp
old_timestamp=${timestamp}
# fetch file only if newer than the version we already have
/usr/bin/wget -qNP ${data_dir} ${target}
if [ "$?" -ne "0" ]; then
/usr/bin/logger -p cron.err "IPSet: ${firewall_ipset} wget failed."
exit 1
fi
get_timestamp
# compare timestamps because wget returns success even if no newer file
if [ "${timestamp}" != "${old_timestamp}" ]; then
temp_ipset="${firewall_ipset}_temp"
/usr/sbin/ipset create ${temp_ipset} ${ipset_params}
networks=( $(/bin/sed -rn 's/(^([0-9]{1,3}.){3}[0-9]{1,3}).*$/\1/p' ${data_file}) )
i=${#networks[*]}
while [ $((--i)) -ge 0 ]; do
/usr/sbin/ipset add ${temp_ipset} ${networks[i]}
done
# if ipset does not exist, create it
/usr/sbin/ipset create -exist ${firewall_ipset} ${ipset_params}
# swap the temp ipset for the live one
/usr/sbin/ipset swap ${temp_ipset} ${firewall_ipset}
/usr/sbin/ipset destroy ${temp_ipset}
# log the file modification time for use in minimizing lag in cron schedule
/usr/bin/logger -p cron.notice "IPSet: ${firewall_ipset} updated (as of: ${timestamp})."
fi |
I call it hourly from cron like so (I check for an update every 15 minutes, because the time of publication has varied). Fine tune this as you see fit: | Code: | # /etc/cron.d/update_block
# BoneKracker
# 31 March 2011
# Every hour, poll for update to dshield
# block list, and update firewall blacklist.
# Last I checked it was being published
# hourly between H +16 min and H +20 min.
#
# To check timing: grep block /var/log/crond
09 * * * * root /usr/local/sbin/block
24 * * * * root /usr/local/sbin/block
39 * * * * root /usr/local/sbin/block
54 * * * * root /usr/local/sbin/block |
This is another version of the same script, modified to block all the registered IP networks in any countries you specify. It uses an updated copy of the IANA regional registraar's list of networks for each country, and can efficiently block multiple lists including tens of thousands of networks of various size with virtually no performance impact (because netfilter is only matching against a single ipset, not tens of thousands of addresses). I use it to block three or four countries (whether that's useful or not is arguable, but I do it anyway): | Code: | #! /bin/bash
# /usr/local/sbin/ipdeny
# BoneKracker
# Rev. 17 May 2011
# Purpose: Load ip networks registered in a country into an ipset and load that
# ipset into a setlist containing several such ipsets, while this setlist is
# being used in a running firewall.
#
# Notes: Call this from crontab. Feed updated about 05:07 and 15:07 daily.
#
# Usage: 'ipdeny <TLD>' (where TLD is top-level national domain, such as "us")
[ -n "$1" ] && firewall_ipset="$1" || exit 1
ipset_params="hash:net"
filename="${firewall_ipset}.zone" # on server, files are "us.zone" etc.
target="http://www.ipdeny.com/ipblocks/data/countries/${filename}"
data_dir="/var/tmp/ipdeny"
data_file="${data_dir}/${filename}"
# if data directory does not exist, create it
/bin/mkdir -m 0750 ${data_dir} 2>/dev/null
# function to get modification time of the file in log-friendly format
get_timestamp() {
timestamp=$(/bin/date -r ${data_file} +%m/%d' '%R 2>/dev/null)
}
# file modification time on server is preserved during wget download
get_timestamp
old_timestamp=${timestamp}
# fetch file only if newer than the version we already have
/usr/bin/wget -qNP ${data_dir} ${target}
if [ "$?" -ne "0" ]; then
/usr/bin/logger -p cron.err "IPSet: ${firewall_ipset} wget failed."
exit 1
fi
get_timestamp
# compare timestamps because wget returns success even if no newer file
if [ "${timestamp}" != "${old_timestamp}" ]; then
temp_ipset="${firewall_ipset}_temp"
/usr/sbin/ipset create ${temp_ipset} ${ipset_params}
while read network; do
/usr/sbin/ipset add ${temp_ipset} ${network}
done < ${data_file}
# if ipset does not exist, create it
/usr/sbin/ipset create ${firewall_ipset} ${ipset_params} 2>/dev/null
# swap the temp ipset for the live one
/usr/sbin/ipset swap ${temp_ipset} ${firewall_ipset}
/usr/sbin/ipset destroy ${temp_ipset}
# if the setlist does not exit, create it
/usr/sbin/ipset create -exist ipdeny list:set
# if the ipset is not already in the setlist, add it
/usr/sbin/ipset add -exist ipdeny ${firewall_ipset}
# log the file modification time for use in minimizing lag in cron schedule
/usr/bin/logger -p cron.notice "IPSet: ${firewall_ipset} updated (as of: ${timestamp})."
fi |
I call this second one from cron twice a day, like so: | Code: | # /etc/cron.d/update_block
# BoneKracker
# 16 May 2011
# Twice per day, poll for update to ipdeny
# block lists, and update firewall blacklist.
# Last I checked by running hourly, they were being published as follows:
# 05:05 - 05:07
# 15:05 - 15:07
# To check timing: grep block /var/log/crond
countries="cn ru ir ng"
09 5 * * * root for c in $countries; do /usr/local/sbin/ipdeny $c; done
09 15 * * * root for c in $countries; do /usr/local/sbin/ipdeny $c; done |
And here's a third version which blocks all the addresses which nobody should be using. Such addresses are called "bogons". There are a few simple bogons (basically the private addresses and some reserved ones), but the actual list is much larger, including currently unassigned addresses. Unassigned addresses are frequently used for malicious activity.
It fetches the list every 4 hours from an organization called Cymru, who are kind enough to provide it and should not be abused.
Really, this is something end users shouldn't have to do, because it should be done by ISPs and backbone routers. But I get hits on it and it provides a redundant layer of security; whether you need that redundancy is up to you. | Code: | #! /bin/bash
# /usr/local/sbin/fullbogons-ipv4
# BoneKracker
# Rev. 14 August 2011
# Purpose: Periodically update an ipset used in a running firewall to block
# bogons. Bogons are addresses that nobody should be using on the public
# Internet because they are either private, not to be assigned, or have
# not yet been assigned.
#
# Notes: Call this from crontab. Feed updated every 4 hours.
target="http://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt"
ipset_params="hash:net"
filename=$(basename ${target})
firewall_ipset=${filename%.*} # ipset will be filename minus ext
data_dir="/var/tmp/${firewall_ipset}" # data directory will be same
data_file="${data_dir}/${filename}"
# if data directory does not exist, create it
/bin/mkdir -m 0750 ${data_dir} 2>/dev/null
# function to get modification time of the file in log-friendly format
get_timestamp() {
timestamp=$(/bin/date -r ${data_file} +%m/%d' '%R 2>/dev/null)
}
# file modification time on server is preserved during wget download
get_timestamp
old_timestamp=${timestamp}
# fetch file only if newer than the version we already have
/usr/bin/wget -qNP ${data_dir} ${target}
if [ "$?" -ne "0" ]; then
/usr/bin/logger -p cron.err "IPSet: ${firewall_ipset} wget failed."
exit 1
fi
get_timestamp
# compare timestamps because wget returns success even if no newer file
if [ "${timestamp}" != "${old_timestamp}" ]; then
temp_ipset="${firewall_ipset}_temp"
/usr/sbin/ipset create ${temp_ipset} ${ipset_params}
#/bin/sed -i '/^#/d' ${data_file} # strip comments
/bin/sed -ri '/^[#< \t]|^$/d' ${data_file} # occasionally the file has been xhtml
while read network; do
/usr/sbin/ipset add ${temp_ipset} ${network}
done < ${data_file}
# if ipset does not exist, create it
/usr/sbin/ipset create -exist ${firewall_ipset} ${ipset_params}
# swap the temp ipset for the live one
/usr/sbin/ipset swap ${temp_ipset} ${firewall_ipset}
/usr/sbin/ipset destroy ${temp_ipset}
# log the file modification time for use in minimizing lag in cron schedule
/usr/bin/logger -p cron.notice "IPSet: ${firewall_ipset} updated (as of: ${timestamp})."
fi |
I call this third one from crontab every 4 hours like so: | Code: | # /etc/cron.d/update_bogons
# BoneKracker
# Rev. 16 May 2011
# Every four hours, poll for update to ipv4-fullbogons
# block list, and update firewall blacklist.
# Last I checked by running hourly, it was being published as follows:
# 00:48 - 00:50
# 04:48 - 04:50
# 08:48 - 08:50
# 12:48 - 12:50
# 16:48 - 16:50
# 20:48 - 20:50
#
# To check timing: zgrep bogons /var/log/old_logs/cron*
52 0 * * * root /usr/local/sbin/fullbogons-ipv4
52 4 * * * root /usr/local/sbin/fullbogons-ipv4
52 8 * * * root /usr/local/sbin/fullbogons-ipv4
52 12 * * * root /usr/local/sbin/fullbogons-ipv4
52 16 * * * root /usr/local/sbin/fullbogons-ipv4
52 20 * * * root /usr/local/sbin/fullbogons-ipv4 |
While it's not entirely necessary, I also have an init script which saves the current ipsets when the machine is shut down, and then restores them when it comes back up. Gentoo provides such a script, but it's defective in that it does not handle setlists (setlist is a type of ipset that contains other ipsets; I use on in the ipdeny script):
| Code: | #!/sbin/runscript
# /etc/init.d/ipset
# BoneKracker
# Rev. 14 August 2011
opts="save"
depend() {
before iptables ip6tables
use logger
}
checkconfig() {
if [ ! -f "${IPSET_SAVE}" ] ; then
eerror "Not starting ${SVCNAME}. First create ipset(s) then run:"
eerror "/etc/init.d/${SVCNAME} save"
return 1
fi
return 0
}
start() {
checkconfig || return 1
ebegin "Loading ipset session"
/usr/sbin/ipset restore < "${IPSET_SAVE}"
eend $?
}
stop() {
if [ "${SAVE_ON_STOP}" = "yes" ] ; then
save || return 1
fi
ebegin "Removing kernel IP sets"
ipset destroy ${SETLISTS}
ipset destroy
eend $?
}
save() {
ebegin "Saving ipset session"
touch "${IPSET_SAVE}"
chmod 0600 "${IPSET_SAVE}"
ipset save > "${IPSET_SAVE}"
eend $?
} |
And it's config file: | Code: | # /etc/conf.d/ipset
# BoneKracker
# 2 October 2011
# Purpose: configuration of /etc/init.d/ipset (a custom initscript)
# Location in which ipsets initscript will save ipsets on service shutdown
IPSET_SAVE="/var/lib/iptables/ipsets-save"
# Save state on stopping iptables
SAVE_ON_STOP="yes"
# If you are using an ipset of type "list:set", put their names here.
SETLISTS="ipdeny" |
_________________
| patrix_neo wrote: | The human thought: I cannot win.
The ratbrain in me : I can only go forward and that's it. |
|
|
| Back to top |
|
|
mimosinnet
l33t
Joined: 10 Aug 2006
Posts: 719
Location: Barcelona, Spain
|
| Posted: Wed Oct 26, 2011 11:30 am Post subject: |
|
|
| BoneKracker wrote: | | You can still do it pretty easily with just iptables (using its ipset capabilities) and a simple script. No need for a separate product or some additional daemon running. The nice part is that you can use any source of information on the internet as input to what you want to block, and you aren't limited to some list you have to buy a subscription to. |
Excellent!!!  You have given me a lot to play with! I totally agree in avoiding daemons running, and the only attraction of PeerGuardian and iplist was that they created iptables rules that I could see. I will play with your scripts!
Very greatful! |
|
| Back to top |
|
|
Bones McCracker
Veteran
Joined: 14 Mar 2006
Posts: 1611
Location: U.S.A.
|
| Posted: Wed Oct 26, 2011 7:34 pm Post subject: |
|
|
| mimosinnet wrote: | | BoneKracker wrote: | | You can still do it pretty easily with just iptables (using its ipset capabilities) and a simple script. No need for a separate product or some additional daemon running. The nice part is that you can use any source of information on the internet as input to what you want to block, and you aren't limited to some list you have to buy a subscription to. |
Excellent!!! You have given me a lot to play with! I totally agree in avoiding daemons running, and the only attraction of PeerGuardian and iplist was that they created iptables rules that I could see. I will play with your scripts!
Very greatful! |
Well, these create ipsets. You also have to create a iptables rule that matches against those ipsets and takes the appropriate action (e.g., drop and log, or whatever).
The difference is that you only create one iptables rule (per ipset), and then you leave it alone. And you can even group the ipsets into a single ipset of type 'setlist' (as I did in the "ipdeny" script). Even though an ipset may contain tens of thousands of networks (or addresses, or address-to-port mappings or whatever), it only takes the kernel milliseconds to execute the test. It's a single operation, from the perspective of netfilter. Compared to perfoming tens of thousands of matching operations, that's a huge advantage in speed and efficiency.
For example, I have an iptables rule that corresponds to the "block" script (and +block ipset), an iptables rule that corresponds to the "fullbogons_ipv4" script (and +fullbogons_ipv4 ipset), and an iptables rule that corresponds to the "ipdeny" script (and +ipdeny ipset, which, being an ipset of type 'list', happens to contain multiple other ipsets, one for each country I"m blocking).
In understanding the efficiency of this, it's useful to note that there is not a one-to-one correspondence between iptables rules and netfilter rules and netfilter matching operations. Iptables in an abstraction interface. When you create an iptables rule that says "block these four ports" or "block these four source addresses", it spawns four rules in netfilter, and when a packet is processed, it is tested four times for a match. So that's a false efficiency (an efficiency only in terms of administration). An iptables rule that says, "block this ipset" (which happens to contain tens of thousands of addresses) spawns only one netfilter rule; the kernel only has to perform one matching operation: against the ipset. Given that you're typically matching every incoming syn packet against these rules, that cane make a huge difference, making it possible to run complex and finely-grained blacklists even on routers handling lots of traffic.
You may still prefer to use a package to do this, but I've been using these scripts for a couple of years now and they work fine for me.
On the other hand, if you would like to play around with dynamically creating iptables rules. I have a different script (that serves an entirely different purpose: single-packet authentication), which does that. If you're interested in dynamically manipulating iptables rules, you might want to look at that:
https://forums.gentoo.org/viewtopic-t-687956.html
_________________
| patrix_neo wrote: | The human thought: I cannot win.
The ratbrain in me : I can only go forward and that's it. |
|
|
| Back to top |
|
|
mimosinnet
l33t
Joined: 10 Aug 2006
Posts: 719
Location: Barcelona, Spain
|
| Posted: Sat Nov 05, 2011 2:55 am Post subject: |
|
|
| BoneKracker wrote: | | You may still prefer to use a package to do this, but I've been using these scripts for a couple of years now and they work fine for me. |
This is really what I was looking for! I did not know about ipset, and I have spend some time learning about it and understanding your great scripts. I have also had a look at this post by Nilesh in itech7.com and this post by Pandu Poluan in posterous.com to get some examples of the iptables rules.
I am just a beginner in the magic of itpables, and I do not quite understand the following sentence from ipset: Tips and Examples:
| Quote: | | In order to drop traffic to-from banned networks or IP addresses, use IP sets in the raw table of netfilter. |
From what I understand, this could mean something like...
| Code: | | iptables -t RAW -A PREROUTING -m set --match-set block src -j DROP |
... but I am getting the following error despite I have iptable_raw module loaded:
| Code: | | iptables v1.4.12.1: can't initialize iptables table `RAW': Table does not exist (do you need to insmod?) |
I have been reading the section Untracked connections and the raw table in Iptables Tutorial:Chapter 7. The state machine, and I would need some more late reading to fully understand it.
This is the rule I have ended up with the dshield block list:
| Code: | | iptables -A INPUT -m set --match-set block src -j DROP |
Thanks a lot! |
|
| Back to top |
|
|
Bones McCracker
Veteran
Joined: 14 Mar 2006
Posts: 1611
Location: U.S.A.
|
| Posted: Sat Nov 05, 2011 4:24 am Post subject: |
|
|
The reason they suggest the raw table for blacklisting is that it gets processed first. However, I think there is more than one school of thought on that. I have read elsewhere that this is not the purpose of the raw table and blacklisting should not be done there. 
Also, one must keep in mind that the prerouting chain only touches packets arriving by external interfaces (it does not touch packets generated locally on the host).
You should be fine doing it in the filter table, in the input (and, if appropriate, the output) chain. However, if this box will be functioning as a firewall for more than just itself, then you'd also need the same rule in the forward chain.
As to the "can't initialize iptables table 'foo'" error, that normally indicates you haven't compiled support for that table into the kernel or haven't loaded the module. But since you've got the module, and you say it's loaded, that can't be the case. Your rule looks good to me. I'm afraid I can't help you with that one. I'd start by verifying the relevant portions of my kernel configuration. Maybe there is some other prerequisite for using the raw table that you haven't met.
_________________
| patrix_neo wrote: | The human thought: I cannot win.
The ratbrain in me : I can only go forward and that's it. |
|
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Networking & Security
