Tracking internet activity

The Doctor · Moderator Joined: 27 Jul 2010 Posts: 2678

I want to monitor all sites my box visits. To be clear I'm not trying to monitor another (legitimate) user. My reason is to track (and block) ad/tracking sites or, in the worst case, detect malware calling home.

My ideal format would be a list of human readable urls which I can filter out trusted sites like forums.gentoo.org and eliminate duplicates. The list will be e-mailed daily in a cron job. The last part is fairly easy so once I have the log it shouldn't be a problem. I can't seem to find a good unix way to generate said log. Any tips and tricks? And did I miss something obvious? Google has not been helpful.

Thanks.
_________________
First things first, but not necessarily in that order.

Apologies if I take a while to respond. I'm currently working on the dematerialization circuit for my blue box.

saturnalia0 · Apprentice Joined: 13 Oct 2016 Posts: 165

khayyam · Posted: Tue May 02, 2017 6:08 am Post subject:

The Doctor ...

one possible solution is to use a caching proxy like net-proxy/polipo, you would then set logLevel=4 (for LOGGING_MAX). The advantage to this might be that as it's cached you would be able to do some diagnostics should you need to (though you would probably want to rotate the logs on a regular basis).

@saturnalia0 ... I don't recommend /etc/hosts for ad blocking and the like, something like this is better managed by a dns proxy such as net-dns/unbound (using an 'include' file generated via unbound-block-hosts).

best ... khay

cboldt · Veteran Joined: 24 Aug 2005 Posts: 1046

I use dnsmasq to return 127.0.0.1 for ad servers, and get (monthly update) a list of adserver names from https://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq

Edit to add, dnsmasq can also directly log inquiries made to it. The logging function is quite flexible.

pietinger · Posted: Tue May 02, 2017 11:55 am Post subject: Re: Tracking internet activity

Zucca · Posted: Tue May 02, 2017 1:51 pm Post subject:

I don't monitor the traffic (but I should).

Instead I use /etc/hosts mainly and update it from here. While it's not intended for that use case, it's perhaps the most simple way.

szatox · Advocate Joined: 27 Aug 2013 Posts: 3608

Iptables offers LOG target. You can use it to catch new connections and define information you want to keep in log (e.g. IP and port number).
Not a very clever solution, but it certainly does let you collect some stats.

The Doctor · Moderator Joined: 27 Jul 2010 Posts: 2678

Thanks for the replies.

I neglected to mention that I've already got ublock and some /etc/hosts blocking going already. My problem is simply figuring out what I might want to block in my case.

net-proxy/polipo looks close to what I want, but it is dead upstream. This led me to squid and I spent most of the day playing with it. Maybe I'll make some progress with it tomorrow.

Actually the problem seems much harder than I thought it would be. Redirecting system wide traffic to the proxy is straight forward, but it can't handle https traffic and leaves a lot of activity unmonitored and, with my uber impressive computer skills, more than half the internet inaccessible.

I considered the iptables solution but I'm hesitant because it might get spammy and it only gets ips. Perhaps there is an idea for a project here. Write a utility that takes an iptables log and converts it into something readily usable. I could use host to convert the ips back then a little shell magic to make the log. I think I'll play with that tomorrow as well.
_________________
First things first, but not necessarily in that order.

Apologies if I take a while to respond. I'm currently working on the dematerialization circuit for my blue box.

pietinger · Posted: Wed May 03, 2017 11:48 am Post subject:

Zucca · Posted: Wed May 03, 2017 2:45 pm Post subject: