What's new

IPSec Server Issue - RT-AX86U

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

Xtropy

New Around Here
I just got this router yesterday and struggling to get IPSec VPN working. OpenVPN is working fine but the server is not accepting any connections (nothing in the logs as well) with IPSec. Firewall is off and NAT Passthrough is enabled. Looking at the logs, I see the following which I think are causing the issue:

Jun 2 13:17:32 00[DMN] Starting IKE charon daemon (strongSwan 5.7.2, Linux 4.1.52, aarch64)
Jun 2 13:17:32 00[KNL] unable to set IPSEC_POLICY on socket: Operation not supported (95)
Jun 2 13:17:32 00[NET] installing IKE bypass policy failed
Jun 2 13:17:32 00[KNL] unable to set IPSEC_POLICY on socket: Operation not supported (95)
Jun 2 13:17:32 00[NET] installing IKE bypass policy failed
Jun 2 13:17:32 00[KNL] unable to set IPSEC_POLICY on socket: Operation not supported (95)
Jun 2 13:17:32 00[NET] installing IKE bypass policy failed
Jun 2 13:17:32 00[KNL] unable to set IPSEC_POLICY on socket: Operation not supported (95)
Jun 2 13:17:32 00[NET] installing IKE bypass policy failed
Jun 2 13:17:32 00[CFG] loading ca certificates from '/etc/ipsec.d/cacerts'

Everything else completes but with the bypasses failing, I assume that's not allowing traffic. Anything I can do to allow these policies to apply?
 
I just got this router yesterday and struggling to get IPSec VPN working. OpenVPN is working fine but the server is not accepting any connections (nothing in the logs as well) with IPSec. Firewall is off and NAT Passthrough is enabled. Looking at the logs, I see the following which I think are causing the issue:

Jun 2 13:17:32 00[DMN] Starting IKE charon daemon (strongSwan 5.7.2, Linux 4.1.52, aarch64)
Jun 2 13:17:32 00[KNL] unable to set IPSEC_POLICY on socket: Operation not supported (95)
Jun 2 13:17:32 00[NET] installing IKE bypass policy failed
Jun 2 13:17:32 00[KNL] unable to set IPSEC_POLICY on socket: Operation not supported (95)
Jun 2 13:17:32 00[NET] installing IKE bypass policy failed
Jun 2 13:17:32 00[KNL] unable to set IPSEC_POLICY on socket: Operation not supported (95)
Jun 2 13:17:32 00[NET] installing IKE bypass policy failed
Jun 2 13:17:32 00[KNL] unable to set IPSEC_POLICY on socket: Operation not supported (95)
Jun 2 13:17:32 00[NET] installing IKE bypass policy failed
Jun 2 13:17:32 00[CFG] loading ca certificates from '/etc/ipsec.d/cacerts'

Everything else completes but with the bypasses failing, I assume that's not allowing traffic. Anything I can do to allow these policies to apply?
What is your router situation? Are you double nat (meaning isp router, and then your router behind?
 
No, I just have a cable modem (no routing/switching capability) direct to the router.

I was hoping for some SSH commands to maybe manually apply the IPSEC policies? I've narrowed down the following .conf file that seems to be the initiator of the policy: /tmp/etc/strongswan.d/charon/kernel-netlink.conf

Or is there a way to reset the entire IPSEC stack to defaults? I am about to hard reset the router but seems like taking a sledgehammer to a fly when this is the only issue experienced.
 
When I disabled port forwarding, I was able to connect. I ended up deleting all my port forwarding lines and re-doing them (with the exact same info) and I can connect stably now. Something appears to have glitched.
 
What is your router situation? Are you double nat (meaning isp router, and then your router behind?
I am facing similar issue as the original poster and am running DDNS and the router is visible /pingable from the outside. OpenVPN server is running but not accepting any connections ie no reaction and nothing the logs ever since I upgraded rt's firmware to the latest AussWRT Merlin 386.2_6. Any tip is welcomed.
 
I am facing similar issue as the original poster and am running DDNS and the router is visible /pingable from the outside. OpenVPN server is running but not accepting any connections ie no reaction and nothing the logs ever since I upgraded rt's firmware to the latest AussWRT Merlin 386.2_6. Any tip is welcomed.
What ddns service are you using? Asus default?

The problem is, I don't know what is wrong. You have to provide more information in hopes that someone can catch the error. There are alot of possible reasons. There are so many reasons that process of elimination is required to properly troubleshoot, unless you got suspicious logs that can be combed through.
 
What ddns service are you using? Asus default?

The problem is, I don't know what is wrong. You have to provide more information in hopes that someone can catch the error. There are alot of possible reasons. There are so many reasons that process of elimination is required to properly troubleshoot, unless you got suspicious logs that can be combed through.
Yes I am using Asus default DDNS service. The hostname and rt are pingable and traceable on the net. Interestingly I donćt have that issue on my AC68U router. What other info would help you. Here is a screenshot of the svr settings.
 

Attachments

  • vpn.png
    vpn.png
    426 KB · Views: 167
Yes I am using Asus default DDNS service. The hostname and rt are pingable and traceable on the net. Interestingly I donćt have that issue on my AC68U router. What other info would help you. Here is a screenshot of the svr settings.
I think I found the problem. My previous routers hostname (for the lack of better words we can refer to it as H1) and new router (the AX86U) hostname (H2) in Asus's db have the same IP address assigned (the current in use that as some of us know in many locations in US tend to remain often the same for years unless local providers make some major infrastructure changes). As it stands when the OpenVPN client makes the call to the H2's OpenVPN server Asus's DDNS db is likely resolving both H1 and H2 to the same IP (I checked by doing nslookup on both bost name and got the exact same IP, the whole H1 has been decommissioned on my end past 3 months), and the request goes nowhere since the client is looking for the correct hostname but in back resolution confusion happens and the pockets drop. I contacted Asus Support and requested that the H1 is permanently removed from the db since my dumb(bleep for the 3 letters) didn't just go about hostname transfer during this router swap, no I rather came up with a new hostname failing to recall that my dynamic IP sits often for months and in some instances for years literally the same as issued by the ISP's DHCP svr on the CMTS (yes the providers is a darn cable company). Sitting on the phone with Asus support that is located in a country half a World away (no need to name it, it would be unfair as it is not their fault that Asus greed hires cheap labor) was an interesting experience. Long story short, after near 45 min of back and forth they finally got it and took all the hostnames, mac's and IP info and pledged to update the db by removing the old hostname within the next 24 to 48 hours (I'll believe it when I see it). If they don't I will just drop Asus's DDNS and provision another with another alternative provider.

I will report the outcome as the next comment from me if that is the resolution, so in the future possibly someone else benefits from it as it could be that it is not an issue with the firmware upgrade, rather my own doing. However, interestingly it worked until the code upgrade, which leads me to believe something was cashed somewhere, but I can't put my finger onto what exactly would be where cacheing happened.
 
I think I found the problem. My previous routers hostname (for the lack of better words we can refer to it as H1) and new router (the AX86U) hostname (H2) in Asus's db have the same IP address assigned (the current in use that as some of us know in many locations in US tend to remain often the same for years unless local providers make some major infrastructure changes). As it stands when the OpenVPN client makes the call to the H2's OpenVPN server Asus's DDNS db is likely resolving both H1 and H2 to the same IP (I checked by doing nslookup on both bost name and got the exact same IP, the whole H1 has been decommissioned on my end past 3 months), and the request goes nowhere since the client is looking for the correct hostname but in back resolution confusion happens and the pockets drop. I contacted Asus Support and requested that the H1 is permanently removed from the db since my dumb(bleep for the 3 letters) didn't just go about hostname transfer during this router swap, no I rather came up with a new hostname failing to recall that my dynamic IP sits often for months and in some instances for years literally the same as issued by the ISP's DHCP svr on the CMTS (yes the providers is a darn cable company). Sitting on the phone with Asus support that is located in a country half a World away (no need to name it, it would be unfair as it is not their fault that Asus greed hires cheap labor) was an interesting experience. Long story short, after near 45 min of back and forth they finally got it and took all the hostnames, mac's and IP info and pledged to update the db by removing the old hostname within the next 24 to 48 hours (I'll believe it when I see it). If they don't I will just drop Asus's DDNS and provision another with another alternative provider.

I will report the outcome as the next comment from me if that is the resolution, so in the future possibly someone else benefits from it as it could be that it is not an issue with the firmware upgrade, rather my own doing. However, interestingly it worked until the code upgrade, which leads me to believe something was cashed somewhere, but I can't put my finger onto what exactly would be where cacheing happened.

Nah the DDNS update didn't fix it. Still not able to connect.
 
Nah the DDNS update didn't fix it. Still not able to connect.
Ok. Fixed the problem. So the RC were two things (all my doing not the firmware or VPN server code or anything else)
1. As described before, DDNS record issue
2. Incorrect port forwarding settings on the providers WAN access router
As usual, human error.
 

Similar threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top