[SOLVED] DNS communication issues, when DNS is on a routed LAN segment (ASUS & Merlin Firmware versions).

[Soul_]

So, not sure if this a bug or a feature, but here's the challenge:

Summary: If PiHole DNS service is sitting on a broadcast domain hosted by another router, you can no longer communicate to it, once you set it as a DNS server on ASUS router.

Setup and observations:
L3 Topology is as follows:

WAN<->AX88U(192.168.1.0/24)<->CiscoASR(10.0.0.0/8)<->PiHole

Now if my DNS on AX88U is anything other than the PiHole in question. I can ping, SSH, HTTP, and reach PiHole just fine, even for DNS resolution.
But the moment I configure AX88U to use PiHole as DNS, I can't even ping PiHole anymore. The ingress on ASR no longer sees AX88U sending any frames to reach the pihole. It is as if, the moment you assign the PiHole as DNS server, AX88U forgets how to route to it.

However, if Pihole is on AX88U local segment(192.168.1.0/24), all works well.

Conclusion: It seems that something has changed in latest version, which disallows a DNS Server to be on a routed LAN segment. This can be reproduced on both ASUS firmware and Merlin firmware on latest versions.


Anyone know if there is a way to fix this issue?

For now I am running virtual server port forward setup on Cisco ASR, to make it seem as if the DNS service is on the AX88U local segment. This seems to bypass the issue.

Edit: For additional clarity, home network is on 192.168.1.0/24. This is where I am pinging from. And running all other connections requests.




SOLVED: It is a "feature" in the latest version. If WAN DNS is on a routed LAN segment, it gets allocated at metric 1 to be on eth0 (WAN). To make it work, NAT and port forward from an IP on AX88U local directly connected segment to your LAN routed DNS server.


Update: I have been in touch with ASUS engineers since this issue. They were quite accommodating and happy to help. Over last few weeks, they sent a couple of new beta versions for me to test. Good news is, their latest Beta, I tested today, has fixed this issue.

So, I am anticipating they will be issuing a new GA firmware very soon with the fix included.

 

[eibgrad]

Sounds like your AX88U is missing a static route that tells it how to find the 10.0.0.0/8 network (i.e., via the WAN ip of the CiscoASR). That would explain why the port forwarding works too. That effectively acts as your static route.

 

[Soul_]

yes, exactly. It starts to seem like static route no longer is in effect, when you implement this. But I can ping other IP addresses on the segment, so static route stays in effect. Just the ip that gets set as DNS server becomes unreachable.

I am former CCNP route switch. So I hope that I am not making simple mistakes like that.

Plus in my post, I mentioned that I am able to ping pihole when it is not set as a dns server. That wouldn’t be possible without the static route, would it?

 

[eibgrad]

yes, exactly. It starts to seem like static route no longer is in effect, when you implement this. But I can ping other IP addresses on the segment, so static route stays in effect. Just the ip that gets set as DNS server becomes unreachable.

I am former CCNP route switch. So I hope that I am not making simple mistakes like that.

Plus in my post, I mentioned that I am able to ping pihole when it is not set as a dns server. That wouldn’t be possible without the static route, would it?

Ping other devices on 10.0.0.0/8 from where? For all I know, your PC client is aware of where that network is located, perhaps based on dynamic routing. What matters here is the AX88U as a client. Can you ping the DNS server on 10.0.0.0/8 from THAT client? IOW, ssh into the AX88U and try to ping.

 

[eibgrad]

And btw. Consuming the entire 10.x.x.x network for the CiscoASR seems extreme. That's going to conflict w/ the OpenVPN servers, at least if you use them w/ their default networks (e.g., 10.8.0.0/24). If it happens they are active, they will override any reference to the 10.0.0.0/8 static route to the CiscoASR. Same thing w/ any OpenVPN clients (most fall into the 10.x.x.x. range).

 

[eibgrad]

If the problem persists, dump the AX88U's routing table. We might see a conflict.

ip route show table main

 

[ColinTaylor]

Are you setting the PiHole's address as the WAN DNS server? If so that will be the problem.

If you examine the routing table you will see that the WAN DNS servers have static routes out via the WAN interface. This is new and was done (I assume) to avoid problems with VPN's.

 

[eibgrad]

Are you setting the PiHole's address as the WAN DNS server? If so that will be the problem.

If you examine the routing table you will see that the WAN DNS servers have static routes out via the WAN interface. This is new and was done to avoid problems with VPN's.

Boy oh boy, I can see that breaking some things down the road, esp. scripts. Good to know.

 

[Soul_]

Are you setting the PiHole's address as the WAN DNS server? If so that will be the problem.

If you examine the routing table you will see that the WAN DNS servers have static routes out via the WAN interface. This is new and was done (I assume) to avoid problems with VPN's.

Two things:
1. this used to work just fine with previous versions.
2.Additionally, you can set wan dns to a local ip on ax88u segment and it works just fine. just not on a routed lan segment with this latest version.

 

[Soul_]

For all who are confused, home network is on 192.168.1.0/24. This is where I am pinging from. And running all other connections requests.

 

[ColinTaylor]

Two things:
1. this used to work just fine with previous versions.
2.Additionally, you can set wan dns to a local ip on ax88u segment and it works just fine. just not on a routed lan segment with this latest version.

Indeed.

 

[Soul_]

If the problem persists, dump the AX88U's routing table. We might see a conflict.

ip route show table main

Yep, you are correct. In the latest version, it does force WAN DNS /32 to be on eth0, regardless of where the static route says it is. If it was just a higher than 1 metric, I could have added a lower metric route to counter this.

Looks like it is a "feature". So be it, glad to get the clarity with your help. Looks like my port forwarding setup will become permanent now.

 

[Soul_]

And btw. Consuming the entire 10.x.x.x network for the CiscoASR seems extreme. That's going to conflict w/ the OpenVPN servers, at least if you use them w/ their default networks (e.g., 10.8.0.0/24). If it happens they are active, they will override any reference to the 10.0.0.0/8 static route to the CiscoASR. Same thing w/ any OpenVPN clients (most fall into the 10.x.x.x. range).

Agreed. I am just not using open vpn. But if it did, I would need to truncate some segments out for sure. Thank you.

 

[Soul_]

Update: I have been in touch with ASUS engineers since this issue. They were quite accommodating and happy to help. Over last few weeks, they sent a couple of new beta versions for me to test. Good news is, their latest Beta, I tested today, has fixed this issue.

So, I am anticipating they will be issuing a new GA firmware very soon with the fix included.