Tutorial How to monitor DNS traffic in real-time

[Chuckles67]

For my setup, all lines are green or yellow (DNS over TLS to Cloudflare), except one red line that looks like this, and has several duplicates:
udp src=<WAN-IP> dst=<WAN-DNSServer1-IP> dport=53 src=<WAN-DNSServer1-IP> dst=<WAN-IP>

Following up: if I run amtm and check for script updates, the red line duplicates rise up from low single digit to nearly twenty, so this confirms scripts and amtm do send dns in plain text. I think this is expected (DoT implemented in WAN settings is for LAN client traffic)

 

[eibgrad]

Following up: if I run amtm and check for script updates, the red line duplicates rise up from low single digit to nearly twenty, so this confirms scripts and amtm do send dns in plain text. I think this is expected (DoT implemented in WAN settings is for LAN client traffic)

Once again, this is why the script is so useful.

When you configure DoT over the WAN, it actually *does* include the router itself, NOT just the DHCP server used by the WLAN/LAN clients. Notice at the top of the header where it indicates the WAN's DNS servers, it shows 127.0.1.1, which is Stubby. However, it *also* includes the DNS servers defined on the WAN from the ISP (or the custom DNS servers you chose instead).

As a result, *any* of those DNS servers is just as likely as another to be used. What you need to do is configure "Connect to DNS server automatically" as No, but leave the custom servers blank! When you do, now you'll find that only Stubby (127.0.1.1) is available to the router.

That's why I include the header info. So you can see *why* things behave the way they do. Few ppl would even notice this is happening, let alone be able to understand why, unless they had access to the script.

 

[LimJK]

Given this is the Asuswrt-Merlin forum, it wasn't my intention to support the OEM/stock firmware. Not that I would mind if it did, of course. But I don't even have access to it. And reviewing the code right now, I don't see an obvious reason why if worked previously, it wouldn't work now.

One of the differences in the latest release is that it will actually report <no-data> after the column headings when there is literally no DNS activity. The initial release reported nothing (which was really a bug), which I thought might lead some users to think it wasn't working. Is that the No Data you're referring to, or something else?

Hi,

I just switched back to Merlin's 386.4 for my RT-AX88U, your DNS Monitoring is working properly now, which did not work on Stock Firmware after your updated version. Thanks.

 

[ringlord]

Gentle request to have a 'colorblind mode' where you swap the yellow or green for example by blue ?
I am having difficulty distinguishing yellow and green.

 

[Viktor Jaep]

As a result, *any* of those DNS servers is just as likely as another to be used. What you need to do is configure "Connect to DNS server automatically" as No, but leave the custom servers blank! When you do, now you'll find that only Stubby (127.0.1.1) is available to the router.

Question about this, @eibgrad ... if you leave the DNS Server fields blank under the "Connect to DNS Server Automatically", then where do you specify what DNS servers your router should be using?

Or do you mean, under the DNS Filtering section?

 

[SomeWhereOverTheRainBow]

Once again, this is why the script is so useful.

When you configure DoT over the WAN, it actually *does* include the router itself, NOT just the DHCP server used by the WLAN/LAN clients. Notice at the top of the header where it indicates the WAN's DNS servers, it shows 127.0.1.1, which is Stubby. However, it *also* includes the DNS servers defined on the WAN from the ISP (or the custom DNS servers you chose instead).

As a result, *any* of those DNS servers is just as likely as another to be used. What you need to do is configure "Connect to DNS server automatically" as No, but leave the custom servers blank! When you do, now you'll find that only Stubby (127.0.1.1) is available to the router.

That's why I include the header info. So you can see *why* things behave the way they do. Few ppl would even notice this is happening, let alone be able to understand why, unless they had access to the script.

Alternatively users can set, use dns cache for local resolving to yes which is set by no defautly on the router. If set to yes, dnsmasq will only query itself and not the isp dns. This is when use local cache for dns resolver is set to no , the isp dns is added to /etc/resolv.conf. if set to yes then /etc/resolv.conf will have 127.0.0.1 defined and isp resolvers will be ignored.

The router himself gets its servers for local traffic from /etc/resolv.conf. and not dnsmasq contrary to what some may think. If your isp is defined in resolv.conf, then it is an option for the router to use. Similar to if 127.0.1.1 is defined there. Then Stubby will be considered an upstream option. If the isp and Stubby are both listed, then both will be used.

 

[Chuckles67]

Great suggestion @eibgrad to leave the WAN DNS Setting > DNS Server1 and DNS Server2 blank. This worked for me - meaning no red line in the script output.

Question about this, @eibgrad ... if you leave the DNS Server fields blank under the "Connect to DNS Server Automatically", then where do you specify what DNS servers your router should be using?

View attachment 39344

The box below is "DNS-over-TLS Server List": try this. I'm seeing by entering servers from the pulldown options in "DNS-over-TLS Server List" then the script updates with these servers as Destination IP in yellow (which is what I expected).

 

[Viktor Jaep]

Great suggestion @eibgrad to leave the WAN DNS Setting > DNS Server1 and DNS Server2 blank. This worked for me - meaning no red line in the script output.



The box below is "DNS-over-TLS Server List": try this. I'm seeing by entering servers from the pulldown options in "DNS-over-TLS Server List" then the script updates with these servers as Destination IP in yellow (which is what I expected).

You mean like this? So the one thing I'm missing is making sure my WAN DNS server entries are blank?

 

[SomeWhereOverTheRainBow]

Great suggestion @eibgrad to leave the WAN DNS Setting > DNS Server1 and DNS Server2 blank. This worked for me - meaning no red line in the script output.



The box below is "DNS-over-TLS Server List": try this. I'm seeing by entering servers from the pulldown options in "DNS-over-TLS Server List" then the script updates with these servers as Destination IP in yellow (which is what I expected).

Does your router have correct time on reboots? If not your traffic may not be getting encrypted and falling back to plaintext if the Servers you are using verify time stamps.

 

[Chuckles67]

You mean like this? So the one thing I'm missing is making sure my WAN DNS server entries are blank?

View attachment 39347

Yes (these settings are mine at least for today, or until the next learning - whichever comes first

 

[eibgrad]

Gentle request to have a 'colorblind mode' where you swap the yellow or green for example by blue ?
I am having difficulty distinguishing yellow and green.

Next update (which is already in the works) will have a monochrome option, which differentiates connections based on font attributes (e.g., bold, italic, normal) rather than color. A little easier on the eyes too.

 

[Chuckles67]

Does your router have correct time on reboots? If not your traffic may not be getting encrypted and falling back to plaintext if the Servers you are using verify time stamps.

I'm using RT-AX86U with Merlin 386.4. Don't know the answer to your question about correct time.

I just tried rebooting the router, once router was back up then ssh and ran the script (took me less than a minute after the router had gone back on line) and there was a red line to one of the IP addresses from my DNS-over-TLS Server List: but it quickly turned yellow.

Such a short period of DNS plaintext after a router reboot is good enough for this household (spent many years living with DNS plaintext before!)

 

[eibgrad]

I just tried rebooting the router, once router was back up then ssh and ran the script (took me less than a minute after the router had gone back on line) and there was a red line to one of the IP addresses from my DNS-over-TLS Server List: but it quickly turned yellow.

During testing, I noticed this too. But it's easy to miss. You have to get into ssh as soon as possible after bootup and and start the script to catch it. IIRC, it's udp too. I think Stubby is checking for the latest server IPs rather than depending on the Stubby config file, but that's just a guess (a tcpdump would probably confirm one way or the other).

This is why I made the suggestion in the original post that you *might* want to consider making a permanent install into /jffs, enabling logging, and using the screen utility so you can get the script running ASAP after a reboot and catch theses kinds of anomalies. There may or may NOT be anything you can do about them, but at least you'll KNOW about them. As I said, few ppl even understand what is actually happening wrt DNS. Most are just relying on guesswork.

P.S. This is why it's next to impossible to have absolutely no DNS leaks. The various protections available like DoT or VPNs have to get started first. And inevitably you're going to find these kinds of chicken and egg problems.

 

[jsbeddow]

During testing, I noticed this too. But it's easy to miss. You have to get into ssh as soon as possible after bootup and and start the script to catch it. IIRC, it's udp too. I think Stubby is checking for the latest server IPs rather than depending on the Stubby config file, but that's just a guess (a tcpdump would probably confirm one way or the other).

This is why I made the suggestion in the original post that you *might* want to consider making a permanent install into /jffs, enabling logging, and using the screen utility so you can get the script running ASAP after a reboot and catch theses kinds of anomalies. There may or may NOT be anything you can do about them, but at least you'll KNOW about them. As I said, few ppl even understand what is actually happening wrt DNS. Most are just relying on guesswork.

P.S. This is why it's next to impossible to have absolutely no DNS leaks. The various protections available like DoT or VPNs have to get started first. And inevitably you're going to find these kinds of chicken and egg problems.

Doesn't the router need to do a basic plain insecure DNS lookup to get the proper time before any of the more secure methods (DOT, DOH, Unbound,....) can work? That might explain the brief connection(s) upon startup. It seems to me that this was always the reason to include the (simple) DNS servers in the GUI even if you were ultimately switching over after the initial startup.

Edit: This is presuming that you don't hard-code the NTP time servers with an IP address (possible, but also not recommended).

 

[SomeWhereOverTheRainBow]

I'm using RT-AX86U with Merlin 386.4. Don't know the answer to your question about correct time.

I just tried rebooting the router, once router was back up then ssh and ran the script (took me less than a minute after the router had gone back on line) and there was a red line to one of the IP addresses from my DNS-over-TLS Server List: but it quickly turned yellow.

Such a short period of DNS plaintext after a router reboot is good enough for this household (spent many years living with DNS plaintext before!)

Well this may have worked for a Server that allows fallback, but if it didn't you might have had some issues. So safe to say this may work in some situations, but could be troublesom in others.

 

[SomeWhereOverTheRainBow]

I'm using RT-AX86U with Merlin 386.4. Don't know the answer to your question about correct time.

I just tried rebooting the router, once router was back up then ssh and ran the script (took me less than a minute after the router had gone back on line) and there was a red line to one of the IP addresses from my DNS-over-TLS Server List: but it quickly turned yellow.

Such a short period of DNS plaintext after a router reboot is good enough for this household (spent many years living with DNS plaintext before!)

What is the difference between this, and leaving dns servers set to automatic, but specifying use dns cache as local dns resolver on the tools page under others?

 

[eibgrad]

Doesn't the router need to do a basic plain insecure DNS lookup to get the proper time before any of the more secure methods (DOT, DOH, Unbound,....) can work? That might explain the brief connection(s) upon startup. It seems to me that this was always the reason to include the (simple) DNS servers in the GUI even if you were ultimately switching over after the initial startup.

Edit: This is presuming that you don't hard-code the NTP time servers with an IP address (possible, but also not recommended).

As I said, there are potential chicken and egg problems when it comes to DNS. No doubt. And there may be no way to avoid it. But as you can see, if you DON'T exclude the WAN's DNS servers, then the router will happily use ALL of the available DNS servers, DoT or NON DoT.

Could limiting the router to only DoT raise issues? Perhaps. But that's where you may have to change your strategy if total stealthiness is your goal. For example, you may have to store commonly accessed hostnames in the router's local hosts file (//etc//hosts). At least it avoids the need for DNS queries over the WAN. Of course, the downside is those entries in //etc//hosts may grow stale over time. And so now you may have to keep it updated, at least once you have DoT or VPN protection for your DNS queries.

In short, you now have the information available to know what's happening, and what you can possibly do about it. Some may not care. Some may care and want to take reasonable efforts to mitigate the situation. Others may decide the situation is intolerable and go hog wild trying to stomp out all DNS leaks.

 

[SomeWhereOverTheRainBow]

As I said, there are potential chicken and egg problems when it comes to DNS. No doubt. And there may be no way to avoid it. But as you can see, if you DON'T exclude the WAN's DNS servers, then the router will happily use ALL of the available DNS server, DoT or NOT DoT.

Could limiting the router to only DoT raises issues? Perhaps. But that's where you may have to change your strategy if total stealthiness is your goal. For example, you may have to store commonly accessed hostnames in the router's local hosts file (//etc//hosts). At least it avoids the need for DNS queries over the WAN. Of course, the downside is those entries in //etc//hosts may grow stale over time. And so now you may have to keep it updated, at least once you have DoT or VPN protection for your DNS queries.

In short, you now have the information available to know what's happening, and what you can possibly do about it. Some may not care. Some may care and want to take reasonable efforts to mitigate the situation. Others may decide the situation is intolerable and go hog wild trying to stomp out all DNS leaks.

One thing that might arise is that you are breaking your router's ability to resolve hostnames locally, especially if 127.0.0.1 is not getting specified to /etc/resolv.conf. (which it does not by the routers default behavior.) So if you are using scripts that rely in resolving local client hostnames, you may be out of luck. For example the router might not be able to identify a client's hostname for scripts like diversion (if it uses this method).

 

[eibgrad]

One thing that might arise is that you are breaking your router's ability to resolve hostnames locally, especially if 127.0.0.1 is not getting specified to /etc/resolv.conf. (which it does not by the routers default behavior.) So if you are using scripts that rely in resolving local client hostnames, you may be out of luck. For example the router might not be able to identify a client's hostname for scripts like diversion (if it uses this method).

If you're referring to DoT and NOT specifying any custom DNS servers, all this does is remove the *public* DNS servers from /etc/resolv.conf, replacing it w/ 127.0.1.1 (Stubby). And as far as I can tell, any local hostnames in //etc//hosts are still accessible (using nslookup reports the use of Stubby (127.0.1.1) by default).

I can't speak to anything Diversion may do to alter these change of events (I don't use it). Third-party scripting is often problematic, esp. if it starts altering system behavior for its own benefit. As I say over and over again, DNS is a complex mess on the router.

 

[SomeWhereOverTheRainBow]

If you're referring to DoT and NOT specifying any custom DNS servers, all this does is remove the *public* DNS servers from /etc/resolv.conf, replacing it w/ 127.0.1.1 (Stubby). And as far as I can tell, any local hostnames in //etc//hosts are still accessible (using nslookup reports the use of Stubby (127.0.1.1) by default).

I can't speak to anything Diversion may do to alter these change of events (I don't use it). Third-party scripting is often problematic, esp. if it starts altering system behavior for its own benefit. As I say over and over again, DNS is a complex mess on the router.

I don't see the point in setting dns automatic to no and clearing leaving it blank, when setting use local cache as system resolver on the tools page to yes replaces the dns with 127.0.0.1 and 127.0.1.1 which is effectively doing the same thing .