1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
Dismiss Notice

Welcome To SNBForums

SNBForums is a community for anyone who wants to learn about or discuss the latest in wireless routers, network storage and the ins and outs of building and maintaining a small network.

If you'd like to post a question, simply register and have at it!

While you're at it, please check out SmallNetBuilder for product reviews and our famous Router Charts, Ranker and plenty more!

Unbound GUI Stats including Top Blocked, Top Replies, Today's Replies

Discussion in 'Asuswrt-Merlin' started by juched, Apr 8, 2020.

  1. juched

    juched Senior Member

    Joined:
    Jan 14, 2018
    Messages:
    301
    Location:
    Canada
    Hello Everyone,

    I have worked on extending the Unbound stats, from currently the stats around lookup times, cache hit, etc, to also allow you to see the details on the replies happening on your network. Since I have diversion disabled, I was missing some of the reports it would generate, so this allows me to see some of the same data/

    I have not made this part of the main branch as it requires logging of replies and local actions, which may have an impact on performance, so I wasn't ready to suggest that to everyone yet. However, to allow others to try, I thought I would post here what I have done, and how to set up your self for this as well.

    High-level review for what I have done, yes there may be other ways to do the same, so up to you:
    1. Disabled dnsmasq DNS portion. Since it all just passes to unbound anyways with no diversion blocking for me, I wanted to skip the extra step. Also, this allows unbound to get the client IP, so you can see in the reports who is making the request.
    2. Configured unbound to allow for anyone on the network to speak to it, (not just localhost) since it needs to accept all requests from clients on the network. Also changed the port to 53 since dnsmasq is no longer the proxy for requests.
    3. Configured unbound to enable local-actions and replies logs. I also enabled the use of syslog-ng. This way I can get data on what was requested, and what was blocked explicitly. By using syslog-ng, it benefits from the in-memory handling of logs to not slow down unbound (as I understand syslog-ng).
    4. Created an hourly job to process the unbound logs, and insert them into an SQLite DB, so we can generate stats and output lists in the UI of the router. This script also cleans the unbound logs as well as removes any stats from the DB older than 7 days.
    5. Updated the stats script to output the new graphs and tables.

    To do this, the following steps are what I took.

    1. Create or update /jffs/configs/dnsmasq.conf.add file and insert line:
    Code:
    port=0
    dhcp-option=lan,6,192.168.0.1
    
    Make sure you enter your router IP. This tells DHCP to send your your router as the DNS server.


    You can either reboot at the end, or run the command to restart dnsmasq:
    Code:
    service restart_dnsmasq
    2. Create or update /opt/share/unbound/configs/unbound.conf.add and insert lines:
    Code:
    port: 53
    interface: 0.0.0.0
    access-control: 0.0.0.0/0 allow
    log-replies: yes
    log-local-actions: yes
    log-tag-queryreply: yes
    use-syslog: yes
    extended-statistics: yes
    
    The first 3 lines change the port to be the main DNS handler and open permissions for all clients on your network. The log lines enable output in a format for my script to handle and the last one just ensures extended-stats is enabled for some of the existing UI stats.

    use-syslog means you need to have Scribe installed. If you don't want to use scribe, it "should" work with normal logs, just leave out that line.

    3. Install the develop version of stats.
    This can be done via unbound_manager. Make sure you are running the latest build of 2.18. If it shows an update is available, run the "u" command then run:
    Code:
    sgui dev
    Restart unbound services using the "rs" command if it hasn't restarted yet.

    At this point you should see changes to the Unbound tab under Addons in Asus-Merlin. the logs are processed at 57 minutes on the hour, and the stats are updated at 59 minutes on the hour. You may need to wait for them to run to get stats, or you can force them by running:
    Code:
    /jffs/addons/unbound/unbound_log.sh
    /jffs/addons/unbound/unbound_stats.sh generate
    Top 10 blocked domains (over last 7 days):
    [​IMG]

    Top 10 DNS replies with return code (over last 7 days):
    [​IMG]

    Today's DNS replies - limited to 250:
    upload_2020-4-8_16-4-9.png
     
    Last edited: Apr 9, 2020
    Chris0815, eclp, joe scian and 2 others like this.
  2. rgnldo

    rgnldo Very Senior Member

    Joined:
    Nov 12, 2018
    Messages:
    908
    Location:
    Brazil
    I think there will still be a problem, dnsmasq needs a listening port. I'm using the method that FW Merlin uses with Stubby. That is, Unbound with interface 127.0.1.1 and port 53.
    on /etc/dnsmasq.conf
    Code:
    server:127.0.1.1
    Try suppressing access-control: 0.0.0.0/0 allow
    and let dnsmasq take care of this service.
    With this method, you may experience connection problems if the unbound process dies.
     
    L&LD likes this.
  3. dave14305

    dave14305 Part of the Furniture

    Joined:
    May 19, 2018
    Messages:
    2,959
    Location:
    USA
    A bold move, but might be a deal-breaker for people who rely on dnsmasq to resolve local hostnames. Perhaps better than disabling DNS with port=0 might be to change dnsmasq to a non-53 port and configure Unbound to forward your local domain (e.g. home.lan) to 127.0.0.1:5353 (pick your own port).

    Also, how does it behave on a reboot to disable dnsmasq? I went down this road once but it just wasn't important enough for me to take the risk.
     
  4. juched

    juched Senior Member

    Joined:
    Jan 14, 2018
    Messages:
    301
    Location:
    Canada
    Rebooting works fine. I was a bit concerned about the NTP on startup, so I switched to an IP for the NTP settings used on boot (I use ntpdMerlin after that). I used one from time.google.com pool (216.239.35.0) and one from time.cloudflare.com pool (162.159.200.1).

    Also, for local hostnames you can add other entires like:
    Code:
    local-zone: "myname.lan." static
    local-data: "service.myname.lan. IN A 192.168.0.50"
    
    
    ---- edit ----

    Not a bad idea to forward back to dnsmasq for local entires. I don't use local network resolution myself, so how does that work? Does every DHCP machine name get auto added to the dnsmasq domain lookup? Or do clients register themselves to be added to the local dns name?

    Something like this?
    Code:
    forward-zone:
      name: "myname.lan."
      forward-addr: [email protected]
    
     
    Last edited: Apr 8, 2020
  5. Jack Yaz

    Jack Yaz Part of the Furniture

    Joined:
    Apr 20, 2017
    Messages:
    3,396
    Very nice! It's nice to see my amateur code and dev work used as a base in other projects :)
    Now do realtime logging ;-) at least, that's on the roadmap for uiDivStats to be able to tail the dns queries in the WebUI a la Pi-Hole
     
    Last edited: Apr 8, 2020
    fourty6and2, QuikSilver and juched like this.
  6. juched

    juched Senior Member

    Joined:
    Jan 14, 2018
    Messages:
    301
    Location:
    Canada
    Thanks for the info. I will see about trying it, but running this way for over a week now, everything works fine. Yes, if unbound service dies, then it will impact the ability to browse and all DNS is down, but dnsmasq works just fine with port=0; I dont' see the need to introduce yet another service forwarding requests to another.
     
  7. Jack Yaz

    Jack Yaz Part of the Furniture

    Joined:
    Apr 20, 2017
    Messages:
    3,396
    How much of a CPU hit do you see with the hourly log processing?
     
  8. dave14305

    dave14305 Part of the Furniture

    Joined:
    May 19, 2018
    Messages:
    2,959
    Location:
    USA
    Yes, I had posted an example way back here: Unbound - Authoritative Recursive Caching DNS Server

    EDIT: I see I was testing differently, with dnsmasq listening only on the loopback and Unbound presumably listening on br0.
     
  9. juched

    juched Senior Member

    Joined:
    Jan 14, 2018
    Messages:
    301
    Location:
    Canada
    On my 88U, it processes in a few seconds... about 1 MB of logs. Roughly as I recall. I never measured the actual CPU time.
     
    Jack Yaz likes this.
  10. QuikSilver

    QuikSilver Very Senior Member

    Joined:
    Jan 30, 2019
    Messages:
    518
    Location:
    BFE
    Interested in this! Great work @juched
     
  11. rgnldo

    rgnldo Very Senior Member

    Joined:
    Nov 12, 2018
    Messages:
    908
    Location:
    Brazil
    Code:
    port=5353
    unbound.postconf
    Code:
    #!/bin/sh
    CONFIG=$1
    source /usr/sbin/helper.sh
    logger -t "(dnsmasq.postconf)" "Updating $CONFIG for unbound....."
    if [ -n "$(pidof unbound)" ];then
            pc_delete "servers-file" $CONFIG
            pc_delete "no-negcache" $CONFIG
            pc_delete "domain-needed" $CONFIG
            pc_delete "bogus-priv" $CONFIG
            pc_replace "cache-size=1500" "cache-size=0" $CONFIG
    fi
    I think that's what you want
    Log's:
    Code:
    Apr 08 18:20:49 unbound[4677:0] notice: init module 0: dns64
    Apr 08 18:20:49 unbound[4677:0] notice: init module 1: validator
    Apr 08 18:20:49 unbound[4677:0] notice: init module 2: iterator
    Apr 08 18:20:49 unbound[4677:0] info: start of service (unbound 1.9.6).
    Apr 08 18:20:51 unbound[4677:0] info: generate keytag query _ta-4f66. NULL IN
    Apr 08 18:20:52 unbound[4677:0] reply: 10.0.30.11 youtubei.googleapis.com. AAAA IN NOERROR 0.919234 0 69
    Apr 08 18:20:52 unbound[4677:0] reply: 10.0.30.11 clients3.google.com. AAAA IN NOERROR 1.466845 0 89
    Apr 08 18:20:53 unbound[4677:0] reply: 10.0.30.11 youtubei.googleapis.com. A IN NOERROR 0.694327 0 57
    Apr 08 18:20:53 unbound[4677:0] reply: 10.0.30.11 clients3.google.com. A IN NOERROR 0.480974 0 77
    Apr 08 18:20:55 unbound[4677:0] reply: 10.0.30.11 safebrowsing.googleapis.com. AAAA IN NOERROR 0.454100 0 73
    Apr 08 18:20:55 unbound[4677:0] reply: 10.0.30.11 www.gstatic.com. AAAA IN NOERROR 0.550986 0 61
    Apr 08 18:20:55 unbound[4677:0] reply: 10.0.30.11 safebrowsing.googleapis.com. A IN NOERROR 0.218787 0 61
    Apr 08 18:20:55 unbound[4677:0] reply: 10.0.30.11 www.gstatic.com. A IN NOERROR 0.270147 0 49
    Apr 08 18:21:40 unbound[4677:0] reply: 10.0.30.11 api-global.netflix.com. A IN NOERROR 0.782349 0 242
    Apr 08 18:21:40 unbound[4677:0] reply: 10.0.30.11 api-global.netflix.com. A IN NOERROR 0.782349 0 242
    Apr 08 18:21:40 unbound[4677:0] reply: 10.0.30.11 api-global.netflix.com. A IN NOERROR 1.820851 0 242
    Apr 08 18:21:40 unbound[4677:0] reply: 10.0.30.11 api-global.netflix.com. A IN NOERROR 1.820851 0 242
    Apr 08 18:21:51 unbound[4677:0] reply: 10.0.30.11 ichnaea.netflix.com. A IN NOERROR 0.817625 0 233
    Apr 08 18:21:51 unbound[4677:0] reply: 10.0.30.11 ichnaea.netflix.com. A IN NOERROR 0.817625 0 233
    Apr 08 18:21:59 unbound[4677:0] reply: 10.0.30.18 bag.itunes.apple.com. A IN NOERROR 1.873209 0 179
    Apr 08 18:21:59 unbound[4677:0] reply: 10.0.30.18 bag.itunes.apple.com. AAAA IN NOERROR 1.873209 0 303
     
    Last edited: Apr 8, 2020
  12. juched

    juched Senior Member

    Joined:
    Jan 14, 2018
    Messages:
    301
    Location:
    Canada
    Yes, those are the logs with client IP. I take it you are using unbound.conf.add (or unbound.postconf in the /opt folder) to change the port for unbound to 53, so clients talk to it directly?

    Who would talk to dnsmasq on port 5353? Why keep it around?
     
  13. rgnldo

    rgnldo Very Senior Member

    Joined:
    Nov 12, 2018
    Messages:
    908
    Location:
    Brazil
    I assume that by omitting port 53 for dnsmasq, you are assuming DHCP only, that is, IP distribution and leases. It is necessary to organize a service port for the firewall to be organized. It was improvised to indicate the door. You need to see the firewall.
    Well, I can be wrong. We are improving.
     
  14. juched

    juched Senior Member

    Joined:
    Jan 14, 2018
    Messages:
    301
    Location:
    Canada
    Interesting. Seems to me the firewall is working fine, I am seeing blocked traffic from skynet still. Also, I can browse and upnp works ( although I do limit it to 2 devices only).

    Yes, dnsmasq still operates but just for leases. Only the DNS side stops.

    Let’s see what we find with testing.
     
    Markster and QuikSilver like this.
  15. rgnldo

    rgnldo Very Senior Member

    Joined:
    Nov 12, 2018
    Messages:
    908
    Location:
    Brazil
    You're right.

    Actually, port=0 disables only the DNS function, leaving port 67 DHCP.
    Code:
    udp        0      0 0.0.0.0:67              0.0.0.0:*                           3398/dnsmasq
     
  16. rgnldo

    rgnldo Very Senior Member

    Joined:
    Nov 12, 2018
    Messages:
    908
    Location:
    Brazil
    Done.

    unbound.conf

    Code:
    server:
        # Interface and port answer
        port: 53
        interface: 0.0.0.0
        access-control: 0.0.0.0/0 allow
    on dnsmasq.conf.add
    Code:
    port=0
    on unbound.postconf insert dhcp-option=lan,6,IP ROUTER,0.0.0.0
    Code:
    #!/bin/sh
    CONFIG=$1
    source /usr/sbin/helper.sh
    logger -t "(dnsmasq.postconf)" "Updating $CONFIG for unbound....."
    if [ -n "$(pidof unbound)" ];then
            pc_delete "servers-file" $CONFIG
            pc_delete "no-negcache" $CONFIG
            pc_replace "cache-size=1500" "cache-size=0" $CONFIG
            pc_append "dhcp-option=lan,6,IP ROUTER,0.0.0.0" $CONFIG
    fi
    
    Reboot and enjoy!

    --------------------------------------------------------------------------------------+
    Log's
    Code:
    Apr 08 22:24:14 unbound[2377:0] notice: init module 0: dns64
    Apr 08 22:24:14 unbound[2377:0] notice: init module 1: validator
    Apr 08 22:24:14 unbound[2377:0] notice: init module 2: iterator
    Apr 08 22:24:14 unbound[2377:0] info: start of service (unbound 1.9.6).
    Apr 08 22:24:14 unbound[2377:0] info: incoming.telemetry.mozilla.org. always_nxdomain [email protected] incoming.telemetry.mozilla.org. A IN
    Apr 08 22:24:14 unbound[2377:0] reply: 10.0.30.44 incoming.telemetry.mozilla.org. A IN NXDOMAIN 0.000000 1 48
    Apr 08 22:24:14 unbound[2377:0] info: incoming.telemetry.mozilla.org. always_nxdomain [email protected] incoming.telemetry.mozilla.org. AAAA IN
    Apr 08 22:24:14 unbound[2377:0] reply: 10.0.30.44 incoming.telemetry.mozilla.org. AAAA IN NXDOMAIN 0.000000 1 48
    Apr 08 22:24:17 unbound[2377:0] info: generate keytag query _ta-4f66. NULL IN
    Apr 08 22:24:17 unbound[2377:0] reply: 10.0.30.44 e17437.dscb.akamaiedge.net. AAAA IN NOERROR 0.281109 0 100
    Apr 08 22:24:17 unbound[2377:0] reply: 10.0.30.44 e17437.dscb.akamaiedge.net. A IN NOERROR 0.699268 0 60
    Apr 08 22:24:17 unbound[2377:0] reply: 10.0.30.44 a1806.dscb.akamai.net. A IN NOERROR 0.476606 0 151
    Apr 08 22:24:17 unbound[2377:0] reply: 10.0.30.44 a1806.dscb.akamai.net. AAAA IN NOERROR 0.563315 0 179
    Apr 08 22:24:20 unbound[2377:0] reply: 10.0.30.44 api-glb-mia.smoot.apple.com. A IN NOERROR 0.424305 0 61
    Apr 08 22:24:20 unbound[2377:0] reply: 10.0.30.44 api-glb-mia.smoot.apple.com. AAAA IN NOERROR 0.533617 0 73
    Apr 08 22:24:22 unbound[2377:0] reply: 10.0.30.44 ocsp.apple.com. A IN NOERROR 0.095587 0 98
    Apr 08 22:24:22 unbound[2377:0] reply: 10.0.30.44 ocsp.apple.com. A IN NOERROR 1.108530 0 98
    Apr 08 22:24:22 unbound[2377:0] reply: 10.0.30.44 ocsp.apple.com. AAAA IN NOERROR 0.229998 0 122
    Apr 08 22:24:22 unbound[2377:0] reply: 10.0.30.44 ocsp.apple.com. AAAA IN NOERROR 1.242941 0 122
    Apr 08 22:24:23 unbound[2377:0] reply: 10.0.30.44 world-gen.g.aaplimg.com. AAAA IN NOERROR 0.939353 0 97
    Apr 08 22:24:25 unbound[2377:0] reply: 10.0.30.44 firefoxusercontent.com. A IN NOERROR 0.354188 0 104
    Apr 08 22:24:25 unbound[2377:0] reply: 10.0.30.44 firefoxusercontent.com. AAAA IN NOERROR 0.353039 0 264
    Apr 08 22:24:25 unbound[2377:0] reply: 10.0.30.44 content-signature-2.cdn.mozilla.net. A IN NOERROR 0.092695 0 157
    Apr 08 22:24:25 unbound[2377:0] reply: 10.0.30.44 content-signature-2.cdn.mozilla.net. A IN NOERROR 1.092429 0 157
    Apr 08 22:24:25 unbound[2377:0] reply: 10.0.30.44 firefox.settings.services.mozilla.com. AAAA IN NOERROR 0.995063 0 322
    Apr 08 22:24:25 unbound[2377:0] reply: 10.0.30.44 firefox.settings.services.mozilla.com. A IN NOERROR 0.995601 0 162
    Apr 08 22:24:25 unbound[2377:0] reply: 10.0.30.44 push.services.mozilla.com. A IN NOERROR 0.873711 0 97
    Apr 08 22:24:25 unbound[2377:0] reply: 10.0.30.44 d2nxq2uap88usk.cloudfront.net. AAAA IN NOERROR 0.074838 0 271
     
    L&LD likes this.
  17. juched

    juched Senior Member

    Joined:
    Jan 14, 2018
    Messages:
    301
    Location:
    Canada
    Yes! Thank you. I forgot that I had set option 6 on dhcp settings. Updated my original post. It can be put straight into the same conf.add as port=0.

    Thank you


    I set option 42 as well, for NTP.
     
    QuikSilver likes this.
  18. rgnldo

    rgnldo Very Senior Member

    Joined:
    Nov 12, 2018
    Messages:
    908
    Location:
    Brazil
    You have to think about how the user configures his NTP. On the FW Merlin NTP is:
    Code:
    dhcp-option=lan,42,0.0.0.0
    I am thinking of a failover situation, in which the unbound is dead.
    Overall, your proposal is correct.
     
  19. here1310

    here1310 Occasional Visitor

    Joined:
    Nov 1, 2017
    Messages:
    45
    To you, with the request for review:

    Code:
    logger -t "(dnsmasq.postconf)" "Updating $CONFIG for unbound....."                        # unbound_manager
    if [ -n "$(pidof unbound)" ];then
        pc_delete "servers-file" $CONFIG
        pc_delete "no-negcache" $CONFIG
        pc_append "dhcp-option=lan,6,192.168.44.4,0.0.0.0" $CONFIG
        pc_append "dhcp-option=lan,42,192.168.44.4,0.0.0.0" $CONFIG
        #pc_delete "domain-needed" $CONFIG
        #pc_delete "bogus-priv" $CONFIG
        # By design, if GUI DNSSEC ENABLED then attempt to modify 'cache-size=0' results in dnsmasq start-up fail loop
        #       dnsmasq[15203]: cannot reduce cache size from default when DNSSEC enabled
        #       dnsmasq[15203]: FAILED to start up
        if [ -n "$(grep "^dnssec" $CONFIG)" ];then
            pc_delete "dnssec" $CONFIG
            logger -t "(dnsmasq.postconf)" "**Warning: Removing 'dnssec' directive from 'dnsmasq' to allow DISABLE cache (set 'cache-size=0')"
        fi
        pc_replace "cache-size=1500" "cache-size=0" $CONFIG
        UNBOUNDLISTENADDR="127.0.0.1#53535"
    
    "UNBOUNDLISTENADDR" should also be changed, right?




    how can you solve the "rest" of dnsmasq.conf.add with unbound?

    local or vpn-client-connection

    Code:
    interface=wg*
    
    adresse=/checkip.synology.com/0.0.0.0
    
    adresse=/.ftl.ddnss.de/ftl.ddnss.de/svr.home/svr.local/192.168.44.2 (local)
    adresse=/.dd.ddnss.de/dd.ddnss.de/svr.home/svr.local/192.168.77.2 (vpn-connection)
    
    Adresse=/www.google.com/216.239.38.120
    Adresse=/www.google.de/216.239.38.120
    Adresse=/www.bing.com/204.79.197.220
    Adresse=/www.duckduckgo.com/176.34.155.20
     
    Last edited: Apr 24, 2020
  20. dave14305

    dave14305 Part of the Furniture

    Joined:
    May 19, 2018
    Messages:
    2,959
    Location:
    USA
    Add statements like these to unbound.conf in the server: section:
    Code:
    local-data: "checkip.synology.com. A 0.0.0.0"
    local-data: "www.google.com. A 216.239.38.120"