Posted by & filed under Programming & Sysadmin.

Some might say it’s a bit over-the-top “NSA like” to monitor your own home network. But honestly, if you are raising kids these days, you know that creepy people may try to get into your home, through a fiber of glass, or a twisted pair of wires, over the wifi, and into your kid’s heads. You can’t stop the creeps without turning off the Internet. And well, good luck with that.

That is the reason for monitoring, and for any types of controls you might place on access. So, with the justification out of the way, here’s the problem, and the solution.

The problem with commercial home routers is that they are designed to be as simple as possible, so that they generate very few support calls for the ISP. If you want “parental controls”, the ISP will provide that on their end, via some subscription $ervice, third party, or not at all. You can put the family PC in the living room. If you are lucky, your router may have some basic controls that allow you to block certain sites by name, or turn off the wifi at certain hours. But there’s nothing that will tell you what sites were visited, when, and how many times. You can install software on computers to monitor this stuff (bluecoat, etc), but it’s a little more difficult to monitor a couple phones, tablets, rokus, an xbox, ps3, and other devices that might attach to your wifi.

The only way to monitor those devices is with a proxy server, or using packet capture and analysis. A transparent proxy server will do this without having to configure the end user device, so that’s the approach I took, since it’s not practical to configure a proxy server on every device. Packet capture is better, but it requires more hardware than I was willing to throw at this problem.

My $dayjob is linux sysadmin, so I’m comfortable using open source router firmware and other open source tools and applications. Of course, there are a lot of different ways to put open source software together, and a lot of choices. The approach outlined here is just one way to accomplish what I wanted.

  1. Get a router that supports it, and install openwrt, an open source, and very customizable router firmware. Get an image with a web interface – Luci. Get it working first.
  2. Install some additional packages to openwrt: tinyproxy, luci-app-tinyproxy.
  3. If you don’t have one already, I’d recommend adding a PC to your network, configured as a linux server that can capture and analyze your proxy logs. You could probably do this right on your openwrt router, but depending on the size of the logs, it might be difficult. The PC can be any old desktop PC you have available.

Note that tinyproxy on openwrt only proxies port 80 traffic, so if you need to also proxy port 443 (ssl) traffic, then you’ll want to look at using squid instead.

To setup a transparent proxy using tinyproxy, open openwrt admin. Under tinyproxy’s configuration, set port 8123. Under Network -> Firewall -> Custom Rules, configure firewall rules to forward packets on port 80 to tinyproxy. Note that you probably do not want to proxy every device, so I’d recommend setting all devices up with static leases in the dhcp settings in openwrt. That way all devices always get the same IP and you can use that IP to select certain devices for proxying. Add the following line for each device you want to proxy. Modify as needed for your network:

iptables -t nat -A PREROUTING -i br-lan -s -p tcp –dport 80 -j DNAT –to

In the above example, is the device you want to proxy, is your openwrt router’s lan IP address, and 8123 is the port that tinyproxy is listening on. “br-lan” is the default name given to the lan interface on my router. Yours may be different. Test it!

If you are logging to an external linux box, you’ll need to configure openwrt. Under tinyproxy settings, check the box to “log via syslog”, then configure syslog settings under system settings so that logs are forwarded to your linux box. Set the IP and port. See “man syslog” on your linux server for how to configure it to receive logs, or google “your distro remote host logging with syslog”.

Analyzing Logs:

I couldn’t find anything free on the Internet to process the tinyproxy logs, so here’s a script I wrote that outputs a couple html tables daily via a daily cron job.

proxy_report.tar.gz (includes some required unmodified jquery js libs)

It requires perl. An example of what the tables look like is below:

proxy report




32 Responses to “Home networks – A Transparent Proxy to Monitor Kids”

      • André Gonzalez

        Fiz as alteração recomendadas no script e instalei o perl, mas na hora de executar o script dá um erro de compilação nas linhas 10 e 11. Quando eu comento as linhas 10 e 11 o script funciona e até gera a página report.html, mas não aparece nenhuma informação. Verifiquei se meu arquivo de log esta sendo gerado com tail -f /var/log/tiniproxy.log e o mesmo é alterado quando faço acessos http. Por favor, poderia me ajudar?

        • admin

          It sounds like you have a lighter version of perl installed (no or warnings).
          I have tinyproxy configured to log at level INFO, and the box checked for “log to syslog”.
          I’ve only tested it on my install of perl, with tinyproxy. It seems like it may not be parsing the log correctly. If you send a few lines of your log, I’ll take a look and see if it can be fixed.

          • André Gonzalez

            Boa tarde! Consegui resolver o problema do script! Estou usando o OpenWrt Attitude Adjustment 12.09. Quando eu executava o script era necessário ter os arquivos e em /usr/lib/perl5/5.10 resolvi o problema instalando perlbase-essential via opkg.
            Agora não sei porque as informações não são importadas do meu arquivo de log para o report,html após a execução do scritp.

            Se marca a opção “Use syslog” como você indica o caminho do log no script?

  1. André Gonzalez

    Fiz as alterações no recomendadas no arquivo, mas quando eu vou executar o scrirpt aparece um erro nas linhas 10 e 11. Tenho o perl instalado, e meu arquivo de log está sendo gerado normalmente. O report.html só é gerado quando eu comento as linhas 10 e 11 do script, mas não faz importação do meu arquivo de log.
    Por favor, poderia me ajudar?

  2. Cartero

    Hi, great job.
    I did exactly what you did but don’t get any url access log from tinyproxy. It seems to start correctly but does not broadcast any messages from the ip I set in the firewall’s custom rule. Is there any other config for tinyproxy or setting the port is enough?

    • admin

      If the firewall rule is working correctly, the status tab for the tinyproxy service should show the number of requests increasing.

    • André Gonzalez


      If you are using OpenWrt enable tinyporxy. Use netstat-tl and see if the port you configured in tinyproxy is as listen.

      You should also do port forwarding with “iptables-t nat-A PREROUTING-i -p tcp – dport 80-j REDIRECT – to-port “, where eth* is the interface that the tiniproxy will listen for requests.

      Important: The log file tinyproxy must have read permission, written by user nobody and group nogroup.

      After use “tail-f / var / log / My.Log” to see if logs are being written.

  3. James

    Can’t seem to see any sites visited on the report. I’m sure I have everything setup correct. Report runs correctly just no data included in the report. Any help is appreciated

    • admin

      Is tinyproxy writing log entries?
      If YES, please share a line or two of the log. Maybe it’s not being parsed properly.

  4. Michael Tarbox

    I thought I had followed your instructions, except for ports to a T, however it doesn’t appear to be working and there are no log files generated. Using a modified openwrt version. Ideas?

    • admin

      Try running the perl script manually. Does it write a file?
      Does the user running the script have permissions to write the file where you configured it to?

      • Michael Tarbox

        Attempted that, still not generating log files.
        Running as nobody and nogroup, so permissions I think are good.
        Running on, with allowed clients on so maybe my port needs to be switched? BTW, complete ‘nix noob. However I do like to think I can read and comprehend directions.

  5. Squidblacklist

    We are the worlds leading publisher of Squid ‘Native ACL’ formatted blacklists, that allow for web filtering directly with Squid proxy. Of course we also offer alternative formats for the most widely used third party plugins, such as DansGuardian and Squidguard. And while our blacklists are subscription based, they are as a result of our efforts, of a much higher degree of quality than the free alternatives.

    We hope to serve you,


    Benjamin E. Nichols

  6. Brian

    Thank you for this helpful post. I’m actually in the midst of trying to set this configuration up on my own and have run into trouble. Doing a web search has lead me to this post and I hope maybe some simple guidance from you can help me.

    I’ve installed the latest appropriate version of OpenWRT on my WRT610n. After doing that the router works fine without making any changes. Then I try to follow the steps outlined here:

    The install succeeds and I go through all the steps and then the router doesn’t work anymore. I can connect with a terminal to it, but it doesn’t route web traffic anymore.

    I know nothing about Linux but I have figured out how to navigate around and open files. But I don’t exactly get the bigger picture, like what files are for or where they are… So your instructions like this “Under tinyproxy’s configuration, set port 8123. Under Network -> Firewall -> Custom Rules, configure firewall rules to forward packets on port 80 to tinyproxy.” I’m not able to exactly follow. I figure “/etc/config/tinyproxy” is the tinyproxy config file… I can edit the line in the file that says “option Port ‘8888’ to ‘8123’” but I have no idea how to find “Network -> Firewall -> Custom Rules”. You refer to the “status” tab in one of your comments, but as far as I can tell there’s no GUI in OpenWRT in which to find a tab, unless I’m REALLY missing something. I’d really appreciate it if you could spell things out to the point of simplistic detail. :)

  7. śmieszne esy


    Would you please confirm the build / version of the openwrt you intended to use?
    Was the image compiled by yourself or have you got one prebuilt downloaded?

  8. Thomas

    If you’re gonna setup a whole box just to do logging, why not just use prixovy instead? Then you have the added bonus of getting rid of all kinds of crappy spam as well.

    • admin

      Good point. It depends on your use case, I guess. I use the logging box for a lot of other things besides logging though, so it was not an issue. And, I like having the proxy acting in a minimally invasive way, as directly connected as possible. The goal is just to observe, not modify the traffic. Tinyproxy does a good job with that.

  9. Sebastian

    Hi! Thank’s a lot for the tutorial. I find tinyproxy a great way to monitor my network traffic, so I went ahead and did everything to get your report tool working on my openwrt router.
    Traffic goes through the proxy, logs are being written and everything seems to be working properly, except I get empty reports. I re-checked twice and I’m sure everything’s fine. I also tried to switch tinyproxy from its own log to syslog and it didn’t work either.
    Any ideas?

    • admin

      You say logs are being written.
      Is the perl script configured to find them where you are writing them?
      Can you run the perl script manually and see what it outputs?
      If still nothing, can you send me a few lines from your log? It may be something that isn’t being parse properly by the perl script that I can fix.

  10. Sebastian

    So… I just did the following:
    1. Put the script, js and css files in one folder and copied part of the log to the same folder.
    2. Set up $log to ‘tinyproxy.log’ (same folder) and $output_file to ‘report.html’ (same folder).
    3. Run the script using: perl
    4. And finally I got a report.html file that was empty.
    I compressed the script, log and resulting html file to a zip that can be found here:
    Thanks a lot for your help.

    • admin

      The problem is the logs are different.
      The perl script needs to be altered to find the relevant parts in the right places in the log lines.
      In perl these are array indexes, and my original script wrongly assumes that logs will be the same.
      In my log (see below) The word “Request” is found at index 5 (the 5th item in the log, beginning at index 0).
      In your log it is at 7.

      This can be fixed this by adding 2 to every index in the file.
      There is a fixed version here:

      Your log:
      Sun Feb 1 00:52:54 2015 tinyproxy[4049]: Request (file descriptor 1): GET /whatever HTTP/1.1
      Sun Feb 1 00:52:54 2015 tinyproxy[4049]: process_request: trans Host GET http://whatever for 6
      Sun Feb 1 00:52:54 2015 tinyproxy[4049]: No upstream proxy for

      My log:
      Feb 2 05:54:47 tinyproxy[4411]: Request (file descriptor 1): GET /whatever HTTP/1.1
      Feb 2 05:54:47 tinyproxy[4411]: process_request: trans Host GET http://whatever for 1
      Feb 2 05:54:47 tinyproxy[4411]: No upstream proxy for

      • Sebastian

        Awesome. Thank you so much. Couldn’t realise that on my own, since I know nothing about perl scripting. Now it worked perfectly well.

        • Sebastian

          One last thing I just realised…
          I switched my tinyproxy setup from “log to syslog” to its own log and the report tool stopped working. Checked the log and found that it would now write something as follows:

          CONNECT Feb 02 15:04:29 [2781]: Request (file descriptor 6): GET /whatever HTTP/1.1
          INFO Feb 02 15:04:29 [2785]: process_request: trans Host GET http://whatever

          So, the indexes match the original script, but it wouldn’t work because the PID is in “[####]” format instead of “tinyproxy[####]”.
          I removed the line where it looks for “tinyproxy” in the PID and left only the one that removes the ” [ ] ” from the PID and that was it. Working like a charm again.

      • Tech crazy

        Can this technique of monitoring be implemented using Tcpdump package using perl

        • admin

          Sure. Tcpdump could be used on the lan interface to write a log that could be parsed. That might impact traffic less than using tinyproxy. Something like
          tcpdump -i $brlan -A port 80 and host $HostIP >> $logfile
          might work. Of course tcpdump will make a lot more log entries than tinyproxy, so you’ll need a strategy to deal with the size of the log.

  11. Tan

    Hi.. Thanks a lot to all and the owner of the post for help… And to help to understand the logics nd all .. Please can any person can provide me a bug free help and code for getting the output display like in ” proxy report ” snapshot ..I am not able to complete and fix the error
    – Thanks in advance for help

  12. Ken Kin

    I ported to lua to make it serve as a cgi of my router ..
    — code begin —

    # proxy report from tinyproxy
    # IMPORTANT: change the config below as needed
    # Tinyproxy logs are written by multiple children at the same time.
    # So this script first sorts by child pid because each child processes requests in order.
    # Then it looks for the normal Connect, Established, Request, Closed sequence that make up a proxy request.
    # Ken A (2014)
    # ported to lua by Ken Kin (2015)

    use strict;
    use warnings;

    ### CHANGE THESE SETTINGS !! #################

    — # where syslog is configured to write tinyproxy logs (hint: set logrotate to daily)
    log = ‘/var/log/tinyproxy.log’;

    — ##############################################;

    — HTTP header
    print(“Content-Type: text/html”)
    print(“”) — An empty line

    function isNotBlank(x)
    if x==nil then
    return false
    return not tostring(x):find(“^%s*$”)

    function split(x, delim)
    if delim == nil then
    delim = “%s”

    local t={}
    for str in string.gmatch(x, “([^”..delim..”]+)”) do
    table.insert(t, str)
    return t

    function string.starts(String,Start)
    return string.sub(String,1,string.len(Start))==Start

    — # need an array of arrays to sort

    for line in io.lines(log, “r”) do
    if isNotBlank(line) then
    local columns=split(line)
    columns[5]=string.match(columns[5], “%d+”)
    table.insert(tplog_array, columns)

    — # sort by pid
    table.sort(tplog_array, function(a, b) return tonumber(a[5])<tonumber(b[5]) end)
    sorted_tplog = tplog_array;


    table.insert(html, "Proxy Report”)
    table.insert(html, “Generated: “””)
    table.insert(html, “”);
    table.insert(html, “TimeHostMethodVisited Site”)
    table.insert(html, “”);

    — # counting visited hosts

    for key, logline in pairs(sorted_tplog) do
    if string.starts(logline[6], “Closed”) then
    — # if we reached “Closed” then reset collecting to 0
    — # so that next Connect will start new.
    if not SEEN then

    if showAnyway or (isNotBlank(time) and isNotBlank(host) and isNotBlank(method) and isNotBlank(visithost)) then
    entry=table.concat({“”, “”},
    table.concat({time, host, method, visithost}, “”));

    if isNotBlank(entry) then
    table.insert(html, entry)
    — # reset variables
    time=””; host=””; visithost=””; query=””; method=””;
    elseif string.starts(logline[6], “Connect”) and not COLLECTING then
    time = table.concat({logline[2], logline[3], logline[4]}, “\32”)
    host = table.concat({logline[10], logline[11]}, “\32”)
    COLLECTING = true;
    — #print “Connect: $host at $time \n”;
    elseif string.starts(logline[6], “Established”) and COLLECTING then
    visithost=string.gsub(logline[10] , “\””, “”)

    if hosts_seen[visithost]==nil then

    — # print “host: logline[10] : $hosts_seen{logline[10]}\n”;
    elseif string.starts(logline[6], “Request”) and COLLECTING then
    if isNotBlank(logline[10]) then

    if isNotBlank(logline[11]) then
    — # elseif ((logline[6] =~ /^process_request/) && ($COLLECTING)) then
    — # $query = “logline[9] logline[10]”;

    table.insert(html, “”)

    table.insert(html, “\&nbsp\;”)
    table.insert(html, “Visited Host Count”)
    table.insert(html, “”)
    table.insert(html, “HostTimes”)

    for key, value in pairs(hosts_seen) do
    table.insert(html, “”..key..””..value..””);

    table.insert(html, “”);

    head = [[

    widgets : [‘zebra’, ‘columns’],
    usNumberFormat : false,
    sortReset : true,
    sortRestart : true


    body=table.concat(html, “\n”)
    content=table.concat({“”, “”, head, “”, body, “”, “”}, “\n”)
    — code end —
    However, I can’t grab the full url the users visited. Any Idea?

    • admin

      Yes, although the data is in the logs, getting the full primary url visited is tricky. URLS these days sometimes cause 10-20 additional requests to ad & tracking services.
      It’s a lot of data to try to pull meaningful info from.



  1.  Content filtering firewall based on openwrt and raspberry pi (Intro) | linklocal

Leave a Reply

  • (will not be published)