Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Ad-Blocking by DNS via Bind RPZ and Pixelserv
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
christoph_peter_s
Tux's lil' helper
Tux's lil' helper


Joined: 30 Nov 2015
Posts: 108

PostPosted: Fri Feb 22, 2019 4:07 pm    Post subject: Ad-Blocking by DNS via Bind RPZ and Pixelserv Reply with quote

Dear fellow Gentooers,

I was sick of these movies running on the side of some news websites while reading, so I decided to explore on how I could block this crap by using my local Bind9 name server. The idea was to redirect requests to these offensive websites to a local pixelserv-server, i.e. a webserver, which returns a single transparent pixel on all received requests.

Thus the problem is a twofold one:
- get pixelserv to run on Gentoo and
- modify my Bind9 installation.


Part 1: pixelserv-tls

There is a nice variant of pixelserv, which is able to serve https with a self produced certificate. It is available from here: https://github.com/kvic-z/pixelserv-tls. In the following I will explain how I did install this in a local overlay and how I made it working.

The very first step is to set up a local overlay as described here: https://wiki.gentoo.org/index.php?title=Handbook:AMD64/Portage/CustomTree&oldid=181586. I did chose /usr/local/overlay to hold the overlay, so the first step is to create the necessary directory.
Code:
mkdir -p /usr/local/overlay/www-servers/pixelserv-tls/files

That is, I selected "pixelserv-tls" as the package name, which will appear in the "www-servers" category. I also created the files subfolder, which later holds the init script.
After that first step I wrote the ebuild pixelserv-tls-2.2.1.ebuild:
Code:
# Copyright 1999-2015 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

EAPI="6"

inherit autotools

DESCRIPTION="Serve one transparent pixel for ad blocking"
HOMEPAGE="https://github.com/kvic-z/pixelserv-tls/wiki"
SRC_URI="https://github.com/kvic-z/pixelserv-tls/archive/${PV}.tar.gz"
LICENSE="LGPL-3"
SLOT="0"
KEYWORDS="~amd64 ~arm"
IUSE=" "

DEPEND="dev-libs/openssl"
RDEPEND="${DEPEND}"

src_prepare() {
    default
    eautoreconf
}


src_install() {
    default

    # copy init script
    newinitd "${FILESDIR}"/pixelserv.init pixelserv

    dodoc ChangeLog || die
    doman ${PN}.1 || die
}

pkg_postinst() {
    ewarn "Make sure, that You supply pixelserv-tls with the appropriate "
    ewarn "CA certificate. Also note, that You have to ensure that Your"
    ewarn "clients do accept the certificate"

    elog  "see: https://github.com/kvic-z/pixelserv-tls/wiki"
}

This is the first ebuild I wrote by myself... so a review from a more experienced fellow won't do any harm. The ebuild needs to be placed in the www-servers/pixelserv-tls directory. In order to craft the ebuild, I did look through the installation readme file of pixelserv-tls. This was a mere standard procedure, except, that is required an 'autoreconf -i' before the ubiquitious './configure'. After a short search it became clear, that the corresponding ebuild routine is eautoreconf, which gets into the ebuild by inheriting the autotools. Note also, that the portage system cares for all the default installation actions. This is pretty nice. :-)

The second file required, is the startup script. As inidicated in the newinitd command in the ebuild this is named pixelserv.init and needs to sit in the files subfolder. I did direct the newinitd command to rename it to pixelserv during install, which means, that the service will be called pixelserv later, rather than pixelserv-tls (or even worse: pixelserv.init).
Amyway, this is my startup script pixelserv.ini
Code:
#!/sbin/openrc-run
# Copyright 1999-2011 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

#
# set the options according to the man-file
#

# -p HTTP_PORT  - http port served by pixelserv
px_p_port="80"
# -k HTTPS_PORT - https port served by pixelserv
px_s_port="443"
# -A PORT       - port for administrative URI ('/servstats' etc)
px_a_port="8080"
# -l LEVEL      - debug level: 0 (silent) .. 5 (very verbose)
l_level="1"
# -z CERT_PATH  - directory for CD certificate and private key
#                 caveat: rw-access for user nobody(?)
z_ca_path="/var/lib/pixelserv"
# -T THREADS    - max. no of threads/users
t_threads="1200"


command="/usr/bin/pixelserv-tls"
command_args="-f -p ${px_p_port} -k ${px_s_port} -A ${px_a_port} -l ${l_level} -z ${z_ca_path} -T ${t_threads}"
pidfile="/var/run/pixelserv-tls.pid"
start_stop_daemon_args="--background --make-pidfile --pidfile ${pidfile}"
description="pixelserv bogus file server"

depend(){
        need net
}

This may look ugly, since it was better to set the preferences in /etc/conf.d rather than editing the init script. But I don't have found an easy explanation, on what I need to do, to get the entries from conf.d to the init script. Maybe I change this later - it wasn't wasn't my top prio at that stage of the project...
The next step is the generation of the manifest...
Code:
ebuild pixelserv-tls-2.2.1.ebuild digest

The ebuild uses the ~arm/~amd64 keyword, so we need to add the pixelserv-tls in package.accept_keywords. Then we can fix the ownership and install the new package:
Code:
chown -R portage:portage /usr/local/overlay
emerge --ask www-servers/pixelserv-tls

Now we can start the pixelserv service for the first time and add it to the default bootlevel, i.e. let it start automatically after boot.
Code:
rc-service pixelserv start
rc-update add pixelserv default

pixelserv now works for normal http on port 80 - but some additional steps are necessary to make it working for https, too. First we need to create a writeable directory for holding the certificates...
Code:
mkdir /var/lib/pixelserv
chmod 777 /var/lib/pixelserv

... and then we generate first the private key and the certificate.
Code:
cd /var/lib/pixelserv
openssl genrsa -out ca.key 1024
openssl req -key ca.key -new -x509 -days 3650 -sha256 -extensions v3_ca -out ca.crt -subj "/CN=Pixelserv CA"
chmod 644 ca.*

After a restart pixelserv should now be able to serve its pixel on https, too. Tip: pointing Your browser to http://your.pixelserv.server/ca.txt is an easy way to install the certificate in Your browser.
Btw: if someone would explain me, on how to prepare the directories for the certificates from within the ebuild, I would gladly add this. It is not a big deal for a single machine. But it certainly would be inacceptable for an official ebuild.


Part 2: Modifying Bind using a Response Policy Zone (RPZ)
RPZ is a means of Bind9 for the manipulation of the DNS. The idea to use this for ad-blocking is not new (see the notes below). There are however only a few descriptions out there, which really seem to work. I have read, that RPZ results in a way higher workload on the DNS server. So You have been warned..

I presume, that a working bind installation is present on Your machine. You should also make sure, that bind had been compiled with the rpz use flag.
The first things to add are the rpz related entries in /etc/bind/named.conf:
Code:
options {
...
        response-policy {
                zone "rpz.adblock";
        };
};
logging {
...
        channel rpz_log {
                file "/var/log/named/rpz.log" versions 0 size 50M;
                print-time yes;
                print-severity yes;
                print-category yes;
                severity info;
        };
...
        category rpz { rpz_log; };
};
...
zone "rpz.adblock" {
        type master;
        file "pri/db.rpz";
};

Logging might be useful, I'd rather switch it off once everything works smoth - or at least reduce the amount of logged incidents. So basically we had first activated RPZ in the options, and then declared the corresponding zone.

Now we need a suited rpz-zone file. Its syntax is pretty straight forward, more or less the same as for a standard zone file. My zone file looks like this:
Code:
$TTL 86400
@ IN SOA dns.example.com. chef.example.com. (
        2019922201  ; serial
        3600        ; refresh
        1800        ; retry
        86400       ; expiry
        3600)       ; minimum
    IN NS LOCALHOST.
adservice.google.nl CNAME pixelserv.example.com.
*.adservice.google.nl CNAME pixelserv.example.com.
...

This means, that any DNS request for adservice.google.nl is answered with the address of our pixerserv machine...
The idea is to get one of these lists out of the net and format them for the usage as our rpz zone file.

To achive this I have slightly modified a small python script found on one of the websites referred to in the notes section below.
Code:
#!/usr/bin/env python
# script found here: https://medium.com/@d.robertson/dns-level-ad-blocking-on-lan-with-bind-rpz-32dbfdf2e4fe
# original author: Daniel Robertson, 12.08.2018
# fixed processing of comments: Peter Serbe, 22.02.2019
#

import urllib.request
import re

boguswww = "pixelserv.example.com"    # local server running pixelserv
specialnets = ("127.0.0.1", "255.255.255.255", "::1", "f")
defaultRoute = "0.0.0.0"
blocklist = "https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts"
zoneHeader = """$TTL 2w
@ IN SOA localhost. root.localhost. (
       2   ; serial
       2w  ; refresh
       2w  ; retry
       2w  ; expiry
       2w) ; minimum
    IN NS localhost."""

print(zoneHeader)

with urllib.request.urlopen(blocklist) as f:
 for bytes in f:
 
  line = bytes.decode("utf-8").strip()
 
  # get rid of all comments from # to end of line
  line = re.sub(r'#.*$', "", line)

  if (not line or line.startswith(specialnets)):
   continue
 
  # ignore the ip address; extract the domain
  domain = line[8:]
 
  if domain == defaultRoute:
   continue
 
  print(domain, " CNAME ", boguswww, ".", sep="")
  print("*.", domain, " CNAME ", boguswww, ".", sep="")

I've copied this to /etc/bind and generated the rpz-zone (one might want to adjust the pixelserv address) and adjust the file attributes:
Code:
./write_blacklist.py > /var/bind/pri/db.rpz
chown root:named db.rpz
chmod 640 db.rpz

Now it is time to restart named and test it:
Code:
brutus /var/bind/pri # rc-service named restart
 * Caching service dependencies ...                                                           [ ok ]
 * Stopping named ...
 * Checking named configuration ...                                                           [ ok ]
 * Starting named ...
 * Checking named configuration ...                                                           [ ok ]
brutus /var/bind/pri # nslookup 30-day-change.com
Server:         192.168.1.25
Address:        192.168.1.25#53

Non-authoritative answer:
30-day-change.com       canonical name = brutus.example.com.
Name:   brutus.example.com.
Address: 192.168.11.205

Everything looks fine. The DNS request for the address of the offending URL got the IP address of our local pixelserv server.

Have fun! & All the best
Peter


Notes:
- For the work on pixelserv-tls I mostly used the information from https://github.com/kvic-z/pixelserv-tls - plus extensive resouces on the Gentoo wiki.
- For the overall approach I first tried to follow https://fattylewis.com/2015/08/08/blocking-ads-by-dns-using-bind/ - this did fail however, as the more modern versions of bind no longer allow using one zone file for multiple zones.
- An overview on RPZ can be found here: https://www.securityzones.net/images/downloads/BIND_RPZ_Installation_Guide.pdf
- The connection of RPZ with the usage of an external list is described here: https://medium.com/@d.robertson/dns-level-ad-blocking-on-lan-with-bind-rpz-32dbfdf2e4fe - the small nit here is a nasty bug in the python script, which inhibits the loading of the rpz zone into bind.

PS: I have quite some doubts, whether it was OK to use Robertsons script here. I some finds this unacceptable, then I would write a perl script with the same functionality... I was not able to contact the author and ask for his permission. :-( But given the fact, that he did publish it, and given the second fact, that I did my very best to give him the credits he does deserve, I decided by myselft that is should be OK to give the scipt listing above. Moreover I fixed an error with did prevent the produced zone file to load...


Last edited by christoph_peter_s on Thu Feb 28, 2019 11:01 pm; edited 1 time in total
Back to top
View user's profile Send private message
Syl20
l33t
l33t


Joined: 04 Aug 2005
Posts: 621
Location: France

PostPosted: Tue Feb 26, 2019 1:29 pm    Post subject: Reply with quote

Interesting. Thank you for the process.

That said, that seems a bit overkill. Most of the hosts block files (for example : https://someonewhocares.org/hosts/) simply redirect to 127.0.0.1. And that works well. Or perhaps I missed something ?
Back to top
View user's profile Send private message
Maf
Guru
Guru


Joined: 15 May 2005
Posts: 310

PostPosted: Wed Feb 27, 2019 8:24 pm    Post subject: Reply with quote

Didn't RTF post, but did you check out https://github.com/pi-hole/pi-hole ?
Back to top
View user's profile Send private message
christoph_peter_s
Tux's lil' helper
Tux's lil' helper


Joined: 30 Nov 2015
Posts: 108

PostPosted: Thu Feb 28, 2019 8:16 am    Post subject: Reply with quote

Maf wrote:
... did you check out https://github.com/pi-hole/pi-hole ?


pi-hole does more or less the same, yes. But there are two issues:

- first I run Bind on the machine, as it is the domain controller of my home Samba4 domain. So adding a second DNS server on this machine is not really straightforward, although feasible (running it on a non-standard port, then using this as forward server...). But if I didn't run Bind already, ph-hole was the better approach, I agree.

- second I am reluctant to install S/W outside of the portage system. It is so much easier to maintain the system with everything inside portage. So in my eyes it was OK to spend the time in researching this. And in fact was most of the time spend learning more on Gentoo.

Regards
- Peter
Back to top
View user's profile Send private message
Ralphred
l33t
l33t


Joined: 31 Dec 2013
Posts: 665

PostPosted: Thu Feb 28, 2019 10:05 pm    Post subject: Reply with quote

christoph_peter_s wrote:
- first I run Bind on the machine, as it is the domain controller of my home Samba4 domain. So adding a second DNS server on this machine is not really straightforward, although feasible (running it on a non-standard port, then using this as forward server...).
Regards
- Peter

I'm in the same situation and have recently been toying with finding host lists and generating zone files.
Thanks for the OP, you seem to have saved me a lot of donkey work on the DNS side.
Still need to mess around with my existing apache and create a vhost for the DNS redirect, but that's not a huge deal.

Thanks again.
Back to top
View user's profile Send private message
Hell-Razor
Guru
Guru


Joined: 10 Jun 2004
Posts: 458

PostPosted: Fri Mar 08, 2019 5:30 pm    Post subject: Reply with quote

Great guide but I second / third a pi-hole. You can do it with bind, but I would highly recommend moving off of bind as its pretty vulnerable unless you spend a lot of time locking it down.
_________________
Don't ever tell anybody anything. If you do, you start missing everybody.
Back to top
View user's profile Send private message
pietinger
Moderator
Moderator


Joined: 17 Oct 2006
Posts: 5211
Location: Bavaria

PostPosted: Sat Mar 09, 2019 12:23 am    Post subject: Reply with quote

Maybe a silly question; but why you didnt use a proxy like "privoxy" ?
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 6920

PostPosted: Wed Dec 16, 2020 2:06 pm    Post subject: Reply with quote

Old thread, but I want to say thanks for writing it. I was oblivious to this RPZ stuff until I read an article the other day about using it in Unbound, and ended up here.

I've been doing it wrong for years: using a script to generate ~40k `zone "ads.foo.example" { file "empty.zone"; }` named.conf lines from a hosts file. That does get the job done… but it takes minutes just to start the server!
Back to top
View user's profile Send private message
Banana
Moderator
Moderator


Joined: 21 May 2004
Posts: 1787
Location: Germany

PostPosted: Wed Dec 16, 2020 7:16 pm    Post subject: Reply with quote

Interesting. Looking into to run it alongside pihole: https://github.com/kvic-z/pixelserv-tls/wiki/%5BPI%E2%80%91HOLE%5D-Setup-pixelserv%E2%80%91tls-for-Pi%E2%80%91Hole
_________________
Forum Guidelines

PFL - Portage file list - find which package a file or command belongs to.
My delta-labs.org snippets do expire
Back to top
View user's profile Send private message
pa4wdh
l33t
l33t


Joined: 16 Dec 2005
Posts: 892

PostPosted: Thu Dec 17, 2020 6:22 pm    Post subject: Reply with quote

Interesting solution, thanks for sharing christoph_peter_s.

I have a similar setup, but dns wise i define zones for the domains i want to block, the zone itself is just a wildcard which points to my webserver. Right now this works ok for 562 blocked domains, but it doesn't scale well to the 10000+ line blocklists you'll find on the net. Maybe RPZ scales better since it's effectively one big zone, that's something i'll have to try.
_________________
The gentoo way of bringing peace to the world:
USE="-war" emerge --newuse @world

My shared code repository: https://code.pa4wdh.nl.eu.org
Music, Free as in Freedom: https://www.jamendo.com
Back to top
View user's profile Send private message
Banana
Moderator
Moderator


Joined: 21 May 2004
Posts: 1787
Location: Germany

PostPosted: Thu Dec 17, 2020 7:07 pm    Post subject: Reply with quote

I'm using pihole with a raspberry for over half a year now. Until now I do not have any performance issues. Right now I'm not sure if this is just another solution or an addition to pihole
_________________
Forum Guidelines

PFL - Portage file list - find which package a file or command belongs to.
My delta-labs.org snippets do expire
Back to top
View user's profile Send private message
user
Apprentice
Apprentice


Joined: 08 Feb 2004
Posts: 212

PostPosted: Thu Dec 17, 2020 7:14 pm    Post subject: Reply with quote

only using bind to block/filter ~1.100.000 records from https://dbl.oisd.nl/

zone conf
Code:
response-policy {
     zone "rpz-malicious-domain";
} break-dnssec yes qname-wait-recurse no nsip-wait-recurse no;

zone "rpz-malicious-domain" {
    type master;
    masterfile-format map;
    file "/var/cache/bind/rpz-malicious-domain.zone.map";
    allow-query { none; };
    allow-transfer { none; };
    allow-update { none; };
};


weekly update script running by named user
Code:

#!/bin/bash
set -euo pipefail

cd /var/cache/bind/

# 252(maxlen) - 20(zonenamelen) - 2(*. wildcard) = 230(maxentry without eol char)
wget -qO - --compression=auto https://dbl.oisd.nl/ |\
grep -v -E '^(#|$|.{231})' |\
sed -e 's/^\(.*\)$/\1 CNAME .\n*.\1 CNAME ./' -e '1i$TTL 1d\
@ IN SOA localhost. dnsmaster.localhost. ('$(date +%Y%m%d)'01 28800 14400 604800 86400)\
  IN NS  localhost.' > rpz-malicious-domain.zone.new

/usr/sbin/named-checkzone -q rpz-malicious-domain rpz-malicious-domain.zone.new
/usr/sbin/named-compilezone -qF map -o rpz-malicious-domain.zone.map.new rpz-malicious-domain rpz-malicious-domain.zone.new

mv rpz-malicious-domain.zone{.new,}
mv rpz-malicious-domain.zone.map{.new,}

/usr/sbin/rndc -q reload rpz-malicious-domain


bind map file consume ~600MiB memory, reload of zone within a few seconds
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum