| View previous topic :: View next topic |
| Author |
Message |
christoph_peter_s
Tux's lil' helper
Joined: 30 Nov 2015
Posts: 108
|
 Posted: Fri Feb 22, 2019 4:07 pm Post subject: Ad-Blocking by DNS via Bind RPZ and Pixelserv |
 |
|
Dear fellow Gentooers,
I was sick of these movies running on the side of some news websites while reading, so I decided to explore on how I could block this crap by using my local Bind9 name server. The idea was to redirect requests to these offensive websites to a local pixelserv-server, i.e. a webserver, which returns a single transparent pixel on all received requests.
Thus the problem is a twofold one:
- get pixelserv to run on Gentoo and
- modify my Bind9 installation.
Part 1: pixelserv-tls
There is a nice variant of pixelserv, which is able to serve https with a self produced certificate. It is available from here: https://github.com/kvic-z/pixelserv-tls. In the following I will explain how I did install this in a local overlay and how I made it working.
The very first step is to set up a local overlay as described here: https://wiki.gentoo.org/index.php?title=Handbook:AMD64/Portage/CustomTree&oldid=181586. I did chose /usr/local/overlay to hold the overlay, so the first step is to create the necessary directory.
| Code: | | mkdir -p /usr/local/overlay/www-servers/pixelserv-tls/files |
That is, I selected "pixelserv-tls" as the package name, which will appear in the "www-servers" category. I also created the files subfolder, which later holds the init script.
After that first step I wrote the ebuild pixelserv-tls-2.2.1.ebuild:
| Code: | # Copyright 1999-2015 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
EAPI="6"
inherit autotools
DESCRIPTION="Serve one transparent pixel for ad blocking"
HOMEPAGE="https://github.com/kvic-z/pixelserv-tls/wiki"
SRC_URI="https://github.com/kvic-z/pixelserv-tls/archive/${PV}.tar.gz"
LICENSE="LGPL-3"
SLOT="0"
KEYWORDS="~amd64 ~arm"
IUSE=" "
DEPEND="dev-libs/openssl"
RDEPEND="${DEPEND}"
src_prepare() {
default
eautoreconf
}
src_install() {
default
# copy init script
newinitd "${FILESDIR}"/pixelserv.init pixelserv
dodoc ChangeLog || die
doman ${PN}.1 || die
}
pkg_postinst() {
ewarn "Make sure, that You supply pixelserv-tls with the appropriate "
ewarn "CA certificate. Also note, that You have to ensure that Your"
ewarn "clients do accept the certificate"
elog "see: https://github.com/kvic-z/pixelserv-tls/wiki"
} |
This is the first ebuild I wrote by myself... so a review from a more experienced fellow won't do any harm. The ebuild needs to be placed in the www-servers/pixelserv-tls directory. In order to craft the ebuild, I did look through the installation readme file of pixelserv-tls. This was a mere standard procedure, except, that is required an 'autoreconf -i' before the ubiquitious './configure'. After a short search it became clear, that the corresponding ebuild routine is eautoreconf, which gets into the ebuild by inheriting the autotools. Note also, that the portage system cares for all the default installation actions. This is pretty nice. 
The second file required, is the startup script. As inidicated in the newinitd command in the ebuild this is named pixelserv.init and needs to sit in the files subfolder. I did direct the newinitd command to rename it to pixelserv during install, which means, that the service will be called pixelserv later, rather than pixelserv-tls (or even worse: pixelserv.init).
Amyway, this is my startup script pixelserv.ini
| Code: | #!/sbin/openrc-run
# Copyright 1999-2011 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
#
# set the options according to the man-file
#
# -p HTTP_PORT - http port served by pixelserv
px_p_port="80"
# -k HTTPS_PORT - https port served by pixelserv
px_s_port="443"
# -A PORT - port for administrative URI ('/servstats' etc)
px_a_port="8080"
# -l LEVEL - debug level: 0 (silent) .. 5 (very verbose)
l_level="1"
# -z CERT_PATH - directory for CD certificate and private key
# caveat: rw-access for user nobody(?)
z_ca_path="/var/lib/pixelserv"
# -T THREADS - max. no of threads/users
t_threads="1200"
command="/usr/bin/pixelserv-tls"
command_args="-f -p ${px_p_port} -k ${px_s_port} -A ${px_a_port} -l ${l_level} -z ${z_ca_path} -T ${t_threads}"
pidfile="/var/run/pixelserv-tls.pid"
start_stop_daemon_args="--background --make-pidfile --pidfile ${pidfile}"
description="pixelserv bogus file server"
depend(){
need net
} |
This may look ugly, since it was better to set the preferences in /etc/conf.d rather than editing the init script. But I don't have found an easy explanation, on what I need to do, to get the entries from conf.d to the init script. Maybe I change this later - it wasn't wasn't my top prio at that stage of the project...
The next step is the generation of the manifest...
| Code: | | ebuild pixelserv-tls-2.2.1.ebuild digest |
The ebuild uses the ~arm/~amd64 keyword, so we need to add the pixelserv-tls in package.accept_keywords. Then we can fix the ownership and install the new package:
| Code: | chown -R portage:portage /usr/local/overlay
emerge --ask www-servers/pixelserv-tls |
Now we can start the pixelserv service for the first time and add it to the default bootlevel, i.e. let it start automatically after boot.
| Code: | rc-service pixelserv start
rc-update add pixelserv default |
pixelserv now works for normal http on port 80 - but some additional steps are necessary to make it working for https, too. First we need to create a writeable directory for holding the certificates...
| Code: | mkdir /var/lib/pixelserv
chmod 777 /var/lib/pixelserv |
... and then we generate first the private key and the certificate.
| Code: | cd /var/lib/pixelserv
openssl genrsa -out ca.key 1024
openssl req -key ca.key -new -x509 -days 3650 -sha256 -extensions v3_ca -out ca.crt -subj "/CN=Pixelserv CA"
chmod 644 ca.* |
After a restart pixelserv should now be able to serve its pixel on https, too. Tip: pointing Your browser to http://your.pixelserv.server/ca.txt is an easy way to install the certificate in Your browser.
Btw: if someone would explain me, on how to prepare the directories for the certificates from within the ebuild, I would gladly add this. It is not a big deal for a single machine. But it certainly would be inacceptable for an official ebuild.
Part 2: Modifying Bind using a Response Policy Zone (RPZ)
RPZ is a means of Bind9 for the manipulation of the DNS. The idea to use this for ad-blocking is not new (see the notes below). There are however only a few descriptions out there, which really seem to work. I have read, that RPZ results in a way higher workload on the DNS server. So You have been warned..
I presume, that a working bind installation is present on Your machine. You should also make sure, that bind had been compiled with the rpz use flag.
The first things to add are the rpz related entries in /etc/bind/named.conf:
| Code: | options {
...
response-policy {
zone "rpz.adblock";
};
};
logging {
...
channel rpz_log {
file "/var/log/named/rpz.log" versions 0 size 50M;
print-time yes;
print-severity yes;
print-category yes;
severity info;
};
...
category rpz { rpz_log; };
};
...
zone "rpz.adblock" {
type master;
file "pri/db.rpz";
}; |
Logging might be useful, I'd rather switch it off once everything works smoth - or at least reduce the amount of logged incidents. So basically we had first activated RPZ in the options, and then declared the corresponding zone.
Now we need a suited rpz-zone file. Its syntax is pretty straight forward, more or less the same as for a standard zone file. My zone file looks like this:
| Code: | $TTL 86400
@ IN SOA dns.example.com. chef.example.com. (
2019922201 ; serial
3600 ; refresh
1800 ; retry
86400 ; expiry
3600) ; minimum
IN NS LOCALHOST.
adservice.google.nl CNAME pixelserv.example.com.
*.adservice.google.nl CNAME pixelserv.example.com.
... |
This means, that any DNS request for adservice.google.nl is answered with the address of our pixerserv machine...
The idea is to get one of these lists out of the net and format them for the usage as our rpz zone file.
To achive this I have slightly modified a small python script found on one of the websites referred to in the notes section below.
| Code: | #!/usr/bin/env python
# script found here: https://medium.com/@d.robertson/dns-level-ad-blocking-on-lan-with-bind-rpz-32dbfdf2e4fe
# original author: Daniel Robertson, 12.08.2018
# fixed processing of comments: Peter Serbe, 22.02.2019
#
import urllib.request
import re
boguswww = "pixelserv.example.com" # local server running pixelserv
specialnets = ("127.0.0.1", "255.255.255.255", "::1", "f")
defaultRoute = "0.0.0.0"
blocklist = "https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts"
zoneHeader = """$TTL 2w
@ IN SOA localhost. root.localhost. (
2 ; serial
2w ; refresh
2w ; retry
2w ; expiry
2w) ; minimum
IN NS localhost."""
print(zoneHeader)
with urllib.request.urlopen(blocklist) as f:
for bytes in f:
line = bytes.decode("utf-8").strip()
# get rid of all comments from # to end of line
line = re.sub(r'#.*$', "", line)
if (not line or line.startswith(specialnets)):
continue
# ignore the ip address; extract the domain
domain = line[8:]
if domain == defaultRoute:
continue
print(domain, " CNAME ", boguswww, ".", sep="")
print("*.", domain, " CNAME ", boguswww, ".", sep="") |
I've copied this to /etc/bind and generated the rpz-zone (one might want to adjust the pixelserv address) and adjust the file attributes:
| Code: | ./write_blacklist.py > /var/bind/pri/db.rpz
chown root:named db.rpz
chmod 640 db.rpz |
Now it is time to restart named and test it:
| Code: | brutus /var/bind/pri # rc-service named restart
* Caching service dependencies ... [ ok ]
* Stopping named ...
* Checking named configuration ... [ ok ]
* Starting named ...
* Checking named configuration ... [ ok ]
brutus /var/bind/pri # nslookup 30-day-change.com
Server: 192.168.1.25
Address: 192.168.1.25#53
Non-authoritative answer:
30-day-change.com canonical name = brutus.example.com.
Name: brutus.example.com.
Address: 192.168.11.205 |
Everything looks fine. The DNS request for the address of the offending URL got the IP address of our local pixelserv server.
Have fun! & All the best
Peter
Notes:
- For the work on pixelserv-tls I mostly used the information from https://github.com/kvic-z/pixelserv-tls - plus extensive resouces on the Gentoo wiki.
- For the overall approach I first tried to follow https://fattylewis.com/2015/08/08/blocking-ads-by-dns-using-bind/ - this did fail however, as the more modern versions of bind no longer allow using one zone file for multiple zones.
- An overview on RPZ can be found here: https://www.securityzones.net/images/downloads/BIND_RPZ_Installation_Guide.pdf
- The connection of RPZ with the usage of an external list is described here: https://medium.com/@d.robertson/dns-level-ad-blocking-on-lan-with-bind-rpz-32dbfdf2e4fe - the small nit here is a nasty bug in the python script, which inhibits the loading of the rpz zone into bind.
PS: I have quite some doubts, whether it was OK to use Robertsons script here. I some finds this unacceptable, then I would write a perl script with the same functionality... I was not able to contact the author and ask for his permission.  But given the fact, that he did publish it, and given the second fact, that I did my very best to give him the credits he does deserve, I decided by myselft that is should be OK to give the scipt listing above. Moreover I fixed an error with did prevent the produced zone file to load...
Last edited by christoph_peter_s on Thu Feb 28, 2019 11:01 pm; edited 1 time in total |
|
| Back to top |
|
 |
Syl20
l33t
Joined: 04 Aug 2005
Posts: 621
Location: France
|
| Posted: Tue Feb 26, 2019 1:29 pm Post subject: |
|
|
Interesting. Thank you for the process.
That said, that seems a bit overkill. Most of the hosts block files (for example : https://someonewhocares.org/hosts/) simply redirect to 127.0.0.1. And that works well. Or perhaps I missed something ? |
|
| Back to top |
|
|
Maf
Guru
Joined: 15 May 2005
Posts: 310
|
|
| Back to top |
|
|
christoph_peter_s
Tux's lil' helper
Joined: 30 Nov 2015
Posts: 108
|
| Posted: Thu Feb 28, 2019 8:16 am Post subject: |
|
|
pi-hole does more or less the same, yes. But there are two issues:
- first I run Bind on the machine, as it is the domain controller of my home Samba4 domain. So adding a second DNS server on this machine is not really straightforward, although feasible (running it on a non-standard port, then using this as forward server...). But if I didn't run Bind already, ph-hole was the better approach, I agree.
- second I am reluctant to install S/W outside of the portage system. It is so much easier to maintain the system with everything inside portage. So in my eyes it was OK to spend the time in researching this. And in fact was most of the time spend learning more on Gentoo.
Regards
- Peter |
|
| Back to top |
|
|
Ralphred
l33t
Joined: 31 Dec 2013
Posts: 668
|
| Posted: Thu Feb 28, 2019 10:05 pm Post subject: |
|
|
| christoph_peter_s wrote: | - first I run Bind on the machine, as it is the domain controller of my home Samba4 domain. So adding a second DNS server on this machine is not really straightforward, although feasible (running it on a non-standard port, then using this as forward server...).
Regards
- Peter |
I'm in the same situation and have recently been toying with finding host lists and generating zone files.
Thanks for the OP, you seem to have saved me a lot of donkey work on the DNS side.
Still need to mess around with my existing apache and create a vhost for the DNS redirect, but that's not a huge deal.
Thanks again. |
|
| Back to top |
|
|
Hell-Razor
Guru
Joined: 10 Jun 2004
Posts: 458
|
| Posted: Fri Mar 08, 2019 5:30 pm Post subject: |
|
|
Great guide but I second / third a pi-hole. You can do it with bind, but I would highly recommend moving off of bind as its pretty vulnerable unless you spend a lot of time locking it down.
_________________
Don't ever tell anybody anything. If you do, you start missing everybody. |
|
| Back to top |
|
|
pietinger
Moderator
Joined: 17 Oct 2006
Posts: 5219
Location: Bavaria
|
| Posted: Sat Mar 09, 2019 12:23 am Post subject: |
|
|
| Maybe a silly question; but why you didnt use a proxy like "privoxy" ? |
|
| Back to top |
|
|
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
| Posted: Wed Dec 16, 2020 2:06 pm Post subject: |
|
|
Old thread, but I want to say thanks for writing it. I was oblivious to this RPZ stuff until I read an article the other day about using it in Unbound, and ended up here.
I've been doing it wrong for years: using a script to generate ~40k `zone "ads.foo.example" { file "empty.zone"; }` named.conf lines from a hosts file. That does get the job done… but it takes minutes just to start the server! |
|
| Back to top |
|
|
Banana
Moderator
Joined: 21 May 2004
Posts: 1788
Location: Germany
|
|
| Back to top |
|
|
pa4wdh
l33t
Joined: 16 Dec 2005
Posts: 892
|
| Posted: Thu Dec 17, 2020 6:22 pm Post subject: |
|
|
Interesting solution, thanks for sharing christoph_peter_s.
I have a similar setup, but dns wise i define zones for the domains i want to block, the zone itself is just a wildcard which points to my webserver. Right now this works ok for 562 blocked domains, but it doesn't scale well to the 10000+ line blocklists you'll find on the net. Maybe RPZ scales better since it's effectively one big zone, that's something i'll have to try.
_________________
The gentoo way of bringing peace to the world:
USE="-war" emerge --newuse @world
My shared code repository: https://code.pa4wdh.nl.eu.org
Music, Free as in Freedom: https://www.jamendo.com |
|
| Back to top |
|
|
Banana
Moderator
Joined: 21 May 2004
Posts: 1788
Location: Germany
|
|
| Back to top |
|
|
user
Apprentice
Joined: 08 Feb 2004
Posts: 212
|
| Posted: Thu Dec 17, 2020 7:14 pm Post subject: |
|
|
only using bind to block/filter ~1.100.000 records from https://dbl.oisd.nl/
zone conf
| Code: | response-policy {
zone "rpz-malicious-domain";
} break-dnssec yes qname-wait-recurse no nsip-wait-recurse no;
zone "rpz-malicious-domain" {
type master;
masterfile-format map;
file "/var/cache/bind/rpz-malicious-domain.zone.map";
allow-query { none; };
allow-transfer { none; };
allow-update { none; };
};
|
weekly update script running by named user
| Code: |
#!/bin/bash
set -euo pipefail
cd /var/cache/bind/
# 252(maxlen) - 20(zonenamelen) - 2(*. wildcard) = 230(maxentry without eol char)
wget -qO - --compression=auto https://dbl.oisd.nl/ |\
grep -v -E '^(#|$|.{231})' |\
sed -e 's/^\(.*\)$/\1 CNAME .\n*.\1 CNAME ./' -e '1i$TTL 1d\
@ IN SOA localhost. dnsmaster.localhost. ('$(date +%Y%m%d)'01 28800 14400 604800 86400)\
IN NS localhost.' > rpz-malicious-domain.zone.new
/usr/sbin/named-checkzone -q rpz-malicious-domain rpz-malicious-domain.zone.new
/usr/sbin/named-compilezone -qF map -o rpz-malicious-domain.zone.map.new rpz-malicious-domain rpz-malicious-domain.zone.new
mv rpz-malicious-domain.zone{.new,}
mv rpz-malicious-domain.zone.map{.new,}
/usr/sbin/rndc -q reload rpz-malicious-domain
|
bind map file consume ~600MiB memory, reload of zone within a few seconds |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Documentation, Tips & Tricks
