Split Tunnel Weirdness

[ColinTaylor]

The router config looks fine. I can't help but think this ping problem is some kind of filtering, firewall or routing issue on the server. Obviously TCP traffic is reaching the server, from both external and internal clients. So unless you've got another internet gateway the external traffic must be making its way via the router to the server, only the source IP address would be different.

Just a random thought....the router uses 22 as its ssh port. My server uses 2222 as its ssh port. Worth considering?

I can't think of any reason why that would be a problem.

 

[jim99]

OK, so here's the plan for tomorrow. First, I will shut down the firewall in the server, and see if that changes anything. If it does, I will take firewall logs with the firewall on and off and debug from there. If as I suspect, doing so makes no change, I will reset the router back to factory default, and flash it back to stock. I will then run it as a simple router, with no VPN capability, to replicate the existing router.

If it won't do that, I shall contact the Asus support team, because my existing router does that acceptably and any router should do.

Assuming that I can make it behave as a simple router on stock firmware, I will then set up the VPN client (still on stock) and check that works. I know this will be of limited use under stock firmware capabilities, because it will stop external access to my servers (dns will still point at my static address, but my servers under stock capabilities will appear at the VPN termination), but I should be able to achieve internal (LAN) access, and that gives me useful debug info. (suppose I could change my DNS to point to the VPN termination, but that would be a horrible, and probably unreliable kludge which I shall avoid at all costs!)

Having got that far, and assuming it behaves itself, I will flash the router with the most recent version of Merlin, flush the config, and reconfigure it, again initially as a simple router with no VPN. This should work, or I will have exposed disparity between stock and Merlin fundamental operation.

Finally, I will set up the VPN and see if that works.

There are a lot of go/no-go points in the process which will help me understand what is working and exactly what isn't.

And to you guys out there who have been so supportive today (especially Colin), I thank you! You have come up with some good suggestions, and forced me to marshal my thoughts and describe the issue clearly, which is a major help when working on a problem like this where its all too easy to disappear down a rabbit hole. If any of the steps above could be changed for the better, let me know your thoughts.

I will update when I have got some concrete results

 

[jim99]

OK, you are going to be displeased....I started this morning by ssh-ing into the server and ensuring I had the commands right to shut the firewall down and bring it back up...I wanted to get it down and up with as short a window as possible, for obvious reasons. If you are familiar with Linux, you will have come across shorewall, and I simply did a shorewall stop/start, making the firewall stop and start in less than a minute. That worked OK, so I went on to the router, only to find it was now working perfectly.

Checked the ping to the server, it was fine, 0.6ms averge, when it was 100% lost before the shorewall restart. Checked all the virtual servers could be accessed by name, that was perfect too. It looks like a simple restart of the server firewall sorted the problem. I was so convinced that it was a router problem, because it worked OK with the old router, that I left checking the server. Sorry guys, it seems I've wasted a lot of your time .

 

[ColinTaylor]

Displeased? I'm delighted! I was beginning to doubt my sanity.

But seriously, I'm glad that you found the solution.

I have used shorewall but only very briefly about 15 years ago.

 

[jim99]

Its reproducable: I had to power off the router to put it in its final position and to tidy the wires. It came up with no pings to the server, and no access to the virtual hosts. I restarted shorewalll, and bingo, off we go! Well, I learned something, but I'm not sure what

 

[jim99]

I hope this will be the final update! I got into debugging shorewall, but as some of you out there know, shorewall is only a manager (with a freiendly face) of the minefield that is iptables. Something was happening when I reset shorewall that put iptables back to a good status. Further contemplation and a lot of coffee lead me to believe that fail2ban might be the culprit. The fail2ban process rummages around in a bunch of logs and grabs addresses that it thinks are suspect, and rewrites iptables to block the suspicious addresses. Something in the way that the new router works (I'm not about to put wireshark on a live server) was upsetting fail2ban, and it was blocking the router's address. That was the root cause: the blocked address led to other untowards events, and basically resulted in a broken server. There is a file in fail2ban (/etc/fail2ban/jail.local) that allows you to fine tune the process, and one of the things you can do is ignore addresses. I set it to ignore the router address, rebooted the server, and I haven't seen the problems since. Of course, it will all resurface (sod's law) just immediately I hit the "post" button on this.

I'm writing this in the hope that it will prevent someone else spending 3 days, off and on, trying to sort out why their Linux server keeps losing connection, and because it may be useful to Colin, who kept me sane a lot of the time when trying to get this @*&^%$ working!