Efficiently use a live blacklist feed in iptables (w/ ipset)

[Bones McCracker]

[Edit: Nov. 2011: Updated the 3 main scripts to ipset-6 syntax]
[Edit: Nov. 2012: Update to incorporate code improvements suggested by Truc]

Background on ipset (if you're familiar with it, skip this):

I'm not affiliated with the netfilter team. I'm writing this simply because there's not much instructive documentation (e.g. tutorials, how-tos) available pertaining to the use of ipsets, so I'm sharing what I've learned by playing with it, hopefully it will save people some time.

ipset is an extremely useful plugin to iptables, particularly if you want to have a firewall rule that matches against a large set of addresses and/or ports, or if you want to dynamically change the addresses and/or ports that a rule matches against.

It lets you create huge lists of ip addresses and/or ports (with tens of thousands of entries or more) which are stored in a tiny piece of ram with extreme efficiency. In your iptables rules, you can then simply refer to the lists by name, and the entire list is checked with remarkable speed and in a single netfilter rule. Also, you can change the contents of the list while the firewall is running.

iptables is actually just the user interface to netfilter. When you write an iptables rule that lists multiple addresses or ports to be blocked (other than "from-to" ranges or netmasks), each item in the list is actually translated into its own netfilter rule, each of which must be processed. Unnecessary processing of this type can really slow down your traffic, since every new connection (or in some cases, every packet) has to be processed through these rules.

Although it possible to add and remove iptables rules while the firewall is running, it is not very efficient to do so. This is how most firewalls (i.e. iptables "front-ends") add and remove entries to their "dynamic blacklist". Blacklist a new address? That's a new rule that must be processed for every connection. You can't "edit" an iptables rule in a running firewall to change the addresses or ports that it matches. Ipset is, in my opinion, by far the best way to run blacklists, whether static, periodically updated from the Internet, or dynamically managed by your firewall or intrusion detection rules.

Also, while iptables provides for "address ranges" and port ranges (in a from-to, netmask, or CIDR format), it cannot efficiently handle long lists of non-contiguous addresses. Try writing iptables rules to block 3,000 specific addresses that have nothing in common. With ipset, this is trivial. It's also efficient, because you can check a packet against the ipset in a single netfilter action (as opposed to 3,000) much, much faster.

Ipset is made by the same netfilter team that makes iptables. It is actually a collection of additional netfilter-related kernel modules (additional "matches" and "targets"). It is part of the Xtables add-on package. In Gentoo, you can get it simiply by emerging ipset (after your "make modules_install") and enabling the appropriate kernel modules.

There are different types of ipsets for storing different types of information (e.g., for random ip addresses, for addresses that are from the same netblock, for random blocks of addresses, for same-sized blocks of addresses, for random ports, etc.).

You can group multiple ipsets of different types into a single "setlist" (an ipset of type list:set) which is still treated as a single ipset by iptables (therefore, you only need one rule to match packets against multiple ipsets). You can bind ipsets together (e.g. a list of addresses and a list of ports). The man page explains it all.

Here is some basic information, and I will provide a couple of practical examples of managing a basic ipset for use in a firewall.
http://ipset.netfilter.org/ipset.man.html#index

An example of updating an ipset in an automated fashion, from an internet source:

There are many ways to use ipsets. This script is just an example. This script is intended to periodically (hourly) update an ipset used as a "blacklist" in a firewall (while the firewall is running and actively processing). The list in this case is a list of "Class C"-sized networks (i.e. CIDR /24 blocks of addresses), published hourly by DShield.org, listing the top blocks of networks from which port-scanning activity has been coming in the last 3 days. It's a tiny list as ipsets go, but it serves the purpose of this example (ipsets can contain many thousands of entries).

That's all incidental. What's important is that it's a list of /24 netblocks, and we want to blacklist them. There is a type of ipset called an "iphash" (hash:ip) that is very efficient and handling lists of same-sized networks (i.e., the ipset efficiently contains a list of networks that have the same netmask, in this case /24), so we'll use that hash:ip type of ipset. We'll use wget to retrieve the block list only if it's been updated. Then, if we've got a new list, we'll load it up to the firewall. Since we can instantaneously swap the contents of one ipset for another, that's how we'll update the live firewall -- we'll parse each address out of the downloaded block list and add it to a temporary ipset, then swap the contents of the temporary ipset into the live ipset in the running firewall, instantly updating the firewall's blacklist en masse.

In later posts, there are other examples that are variations on the theme, demonstrating the use of different types of ipsets and other related concepts. This, however, is good place to start.

This script runs hourly by cron on my system, about five minutes after DShield publishes its hourly update (presently, between HH+15m - HH:20m). For logging purposes it uses "logger", which you can install, or you can substitute what you want or modify the logging lines to simply echo to your log file.

[ssteinberg]

This is actually really interesting, well done. I'd be interested in more feeds.

[Bones McCracker]

Here's another. This one is used to accurately match (block or allow) traffic from or to entire countries.

This script takes a single command-line argument: the 2-letter IANA country code (the top-level domain, such as "us" or "cn"). Based on that, it downloads a list of all the networks of various sizes that are registered to that country code with the regional registrar. Last I checked, these lists are published twice daily by the ipdeny.com project. These are larger lists than the previous example.

Like the list in the previous example, this is a list of networks. However, unlike the previous example, they are networks which vary in size (i.e., have various netmasks). For that reason, we will use the nethash type of ipset (formally now called hash:net, although referring to it the old way still works).

Selecting the appropriate hashsize: In the example above, where the list is always a specific size and very small, I specifically set the hashsize to something smaller than the default. We will not do that here, because the default hashsize (1024) is appropriate for all but the smallest of the country codes, and the hashsize is just a starting point anyway (as an ipset gets larger, the hashsize is adjusted upward dynamically). Setting the hashsize manually only makes sense when you know the approximate size of the ipset in advance, and the point of doing so is to: (a) save a few KiB of ram in the rare case where you are using a list you know is smaller than the default; or (b) to save a bit of rehashing (which occurs as the ipset grows) in the cases where you know the list is larger than the default. It is also possible to optimize this dynamic resizing process in various ways using the "--probes" and "--resize" options, but the defaults are fine, and we don't need to go into that now.

This script also demonstrates the use of of the setlist type of ipset. All of the country-specific ipsets are added to a single, combined setlist, so we can refer to the whole group as a single ipset (and matching against it is a single efficient netfilter action). In the script you'll see where the required setlist is created (if the setlist does not already exist) and where the ipsets are added to the setlist (if they are not already a member of the setlist).

Other than taking a command-line paramater, using a different type of ipset that is of more of a normal size, and employing a hierarchy of ipsets (i.e., using a setlist), this script is pretty much the same as the one above. I actually run this script twice a day from cron, about an hour after ipdeny.com updates the lists. I run it multiple times, fetching the lists for several countries and creating a corresponding ipset for each (these are just example countries here; no offense intended to anyone):

[ssteinberg]

Thanks. I was aware of filtering based on country. Doesn't appeal to me personally as much as a dynamic block list, it is just too general, but people should find it useful. Well done with the scripts.

[Bones McCracker]

[Bones McCracker]

Here is another that's more practically useful. This script creates an ipset that can be used to block all bogons (not just rfc1918 private IP addresses, but every network block that is unassignable or has not yet been assigned by the regional authorities). These addresses are often used by botnets, spammers, and so on. It creates a fairly large ipset of a complex type, so it takes tens of seconds to initially load up the temporary ipset (the swapout operation is still virtually instantaneous, as are queries).

[Bones McCracker]

If anyone is wondering how I came up with the ${ipset_params}, the answer is trial-and-error. In the meantime, I'll share the limited understanding I have gained. I should also point out that I've been working exclusively with hash-type ipsets and the iptreemap, so my optimization insights do not extend to other types.

I would be appreciative of any tips regarding this by a more knowledgeable person, on how to identify optimal values for of these and similar ipset parameters.

When you build an ipset that is one of the "hash" types, do an 'ipset -L' and look at it's size. The default size is 1024 (which I assume to be bytes). If your list is small and you guess that a smaller hash could store it, or if the hashsize has been dynamically "grown" to be substantially larger (e.g. well into the multiple megabyte range), then you may want to try to optimize it (i.e., cause the process of loading the ipset to create a more efficient hash).

This is not necessary, since ipset will dynamically grow and rehash an ipset as entries are added (I haven't tested to see if it will dynamically shrink them as entries are removed, but I doubt it). When all is said and done, pursuing this optimization process might cut the size of an ipset in half, and since they typically stay resident in RAM, this can free up some tens of megabytes of RAM. Whether that's worth your time is up to you.

There may be a better way to do this, but I basically run the process of loading the ipset multiple times, with varying parameters, trying to get a smaller hash size. Living by the rule "default is good", and noting that there is typically a point of diminishing returns on varying from them, one can bracket one's way into a reasonable value without too much effort. Basically the process would be:

[mimosinnet]

I find these scripts really useful. I am learning some perl and I have made a perl version of them. I started with perl on Christmas, as a hobby, so do not expect too much of this version. In the the exercise, I have learned some perl, some bash, and really enjoyed deciphering the regex expressions, still a mystery to me!

Thanks for this great scripts and the introduction to ipsets!

[Bones McCracker]

Glad you found them useful, and thanks for the credit on your page and in your scripts. I'd be interested to know how they compare in terms of performance.
_________________

[mimosinnet]

[Bones McCracker]

Interesting.
_________________

[truc]

[Bones McCracker]

Truc, it took me a long time to get back to this, but thank you for the excellent corrections and suggestions. I am incorporating all of them (with one exception, and one in modified form, as below).

Exception: I got rid of the BASH array as you suggested (don't know what I was thinking there), but I left the $networks variable in place because I think it's more apparent what's going on that way.

Modified: The stderr redirection in the get_timestamp function was there to allow processing to continue if the file is not present (which is a normal condition on first run and when the user might purge the data file from /var/tmp). I took it out per your suggestion, so I made the assignment of "old_timestamp" (which calls the function) conditional on the presence of the data file.

Thank you.
_________________

[elmar283]

Thank you for the script. When I tried it I got the next error:

[khayyam]

[elmar283]

Thank you. It works fine now!

[Rain91]

Could you post the updated script?

[elmar283]

I changed the script so I could put it in my cron width 'crontab -e'.
This are my working scripts:

[Bones McCracker]

[upengan78]

hello,

Can I use this http://www.wizcrafts.net/exploited-servers-iptables-blocklist.html in addition to http://feeds.dshield.org/block.txt ? Thanks for the steps by the way, I got it working on my box but I just felt the list on block.txt isn't very comprehensive.

ipset list

[Bones McCracker]

Sure. That looks fine to me. That's what I intended, that these examples would help people create and maintain their own, and in general make use of ipsets.
_________________

[upengan78]

Sorry to post into this old thread. I noticed this morning target="http://feeds.dshield.org/block.txt" is not working for me since this morning. It's redirected to something. I am using target="http://www.dshield.org/block.txt" . Hope this helps some people.

[Bones McCracker]

Thank you. I suggest changing it to:
"https://www.dshield.org/block.txt"

(Note https.)

[OverrideZ]

hi,

i have made a python version, except that do not check timestamp, just update new entry and delete old one. i am not a expert dev. thanks for block list source

[bdpita]

Hi Bones, inspired by your script I made one a little more flexible and easy to upgrade.
I hope to be of your interest and help to others using Shorewall + ipset with a live blacklist feed.

Best regards, Bernardo.