What's new

RT-AC86U, can't get IPsec server to work

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

NoSync

Regular Contributor
Hi,
I can't get the IPsec server to work on my RT-AC86U. I'm currently on 384.5 (I briefly tried this on 384.4 before as well, but didn't have time to investigate further), and after configuring the server the devices I tried to use (Mac from my office's wifi, iPhone from the same wifi and cellular network) just won't connect. The error I get is "Negotiation with the VPN server failed". The wifi network has got IPsec pass-through enabled, but again, same behavior with cellular.

The RT-AC86U gets a public IP via PPPoE. OpenVPN works flawlessly with the same clients.

When I start the connection from the client (in this case it's the iPhone via cellular) this is what I see when running ipsec statusall (176.200.114.236 is the current IP of my iPhone on the cellular network):

Code:
Status of IKE charon daemon (weakSwan 5.2.1, Linux 4.1.27, aarch64):
  uptime: 11 minutes, since Jun 07 12:31:27 2018
  malloc: sbrk 1466368, mmap 0, used 329776, free 1136592
  worker threads: 3 of 8 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 1
  loaded plugins: charon aes des rc2 sha1 sha2 md4 md5 random nonce x509 revocation constraints acert pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl fips-prf agent xcbc cmac hmac attr kernel-netlink resolve socket-default stroke updown eap-identity eap-md5 eap-mschapv2 eap-tls eap-peap xauth-generic
Virtual IP pools (size/online/offline):
  10.10.10.0/24: 254/0/0
Listening IP addresses:
  192.168.10.1
  192.168.10.2
  192.168.11.1
  169.254.113.110
  <REDACTED>
  10.16.0.1
  10.30.16.122
Connections:
 Host-to-Net:  <REDACTED>...%any  IKEv1, dpddelay=10s
 Host-to-Net:   local:  [<REDACTED>] uses pre-shared key authentication
 Host-to-Net:   remote: uses pre-shared key authentication
 Host-to-Net:   remote: uses XAuth authentication: any
 Host-to-Net:   child:  0.0.0.0/0 === dynamic TUNNEL, dpdaction=clear
Security Associations (0 up, 1 connecting):
   (unnamed)[4]: CONNECTING, <REDACTED>[%any]...176.200.114.236[%any]
   (unnamed)[4]: IKEv1 SPIs: 242a90f70807c796_i 0ddfcb6020a63262_r*
   (unnamed)[4]: IKE proposal: AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048
   (unnamed)[4]: Tasks passive: ISAKMP_VENDOR MAIN_MODE

The clients do reach the server but can't go through, and I can't find anything in the logs except for this unhelpful line:

Code:
Jun  7 12:42:34 06[IKE] 176.200.114.236 is initiating a Main Mode IKE_SA

No errors whatsoever.

Any ideas of what I can try next? I searched the forums but couldn't find anything relevant.

Thanks.
 
type in shell
Code:
ipsec stroke loglevel ike 2
and see the ipsec server log for more accurate log.

There was another user who reported ipsec server didn't work on his iphone.
While ipsec server really work on my android.
 
type in shell
Code:
ipsec stroke loglevel ike 2
and see the ipsec server log for more accurate log.

There was another user who reported ipsec server didn't work on his iphone.
While ipsec server really work on my android.

Thanks Odkrys,
that actually had me solve the problem already. For future reference, this is what I received:

Code:
Jun  7 13:12:58 07[IKE] received NAT-T (RFC 3947) vendor ID
Jun  7 13:12:58 07[IKE] received draft-ietf-ipsec-nat-t-ike vendor ID
Jun  7 13:12:58 07[IKE] received draft-ietf-ipsec-nat-t-ike-08 vendor ID
Jun  7 13:12:58 07[IKE] received draft-ietf-ipsec-nat-t-ike-07 vendor ID
Jun  7 13:12:58 07[IKE] received draft-ietf-ipsec-nat-t-ike-06 vendor ID
Jun  7 13:12:58 07[IKE] received draft-ietf-ipsec-nat-t-ike-05 vendor ID
Jun  7 13:12:58 07[IKE] received draft-ietf-ipsec-nat-t-ike-04 vendor ID
Jun  7 13:12:58 07[IKE] received draft-ietf-ipsec-nat-t-ike-03 vendor ID
Jun  7 13:12:58 07[IKE] received draft-ietf-ipsec-nat-t-ike-02 vendor ID
Jun  7 13:12:58 07[IKE] received draft-ietf-ipsec-nat-t-ike-02\n vendor ID
Jun  7 13:12:58 07[IKE] received XAuth vendor ID
Jun  7 13:12:58 07[IKE] received Cisco Unity vendor ID
Jun  7 13:12:58 07[IKE] received FRAGMENTATION vendor ID
Jun  7 13:12:58 07[IKE] received DPD vendor ID
Jun  7 13:12:58 07[IKE] 176.200.114.236 is initiating a Main Mode IKE_SA
Jun  7 13:12:58 07[IKE] IKE_SA (unnamed)[7] state change: CREATED => CONNECTING
Jun  7 13:12:58 07[IKE] sending strongSwan vendor ID
Jun  7 13:12:58 07[IKE] sending XAuth vendor ID
Jun  7 13:12:58 07[IKE] sending DPD vendor ID
Jun  7 13:12:58 07[IKE] sending NAT-T (RFC 3947) vendor ID
Jun  7 13:12:58 06[IKE] remote host is behind NAT
Jun  7 13:12:58 08[IKE] message parsing failed
Jun  7 13:12:58 08[IKE] ID_PROT request with message ID 0 processing failed
Jun  7 13:13:01 02[IKE] message parsing failed
Jun  7 13:13:01 02[IKE] ID_PROT request with message ID 0 processing failed
Jun  7 13:13:04 08[IKE] message parsing failed
Jun  7 13:13:04 08[IKE] ID_PROT request with message ID 0 processing failed
Jun  7 13:13:08 02[IKE] message parsing failed
Jun  7 13:13:08 02[IKE] ID_PROT request with message ID 0 processing failed
Jun  7 13:13:21 06[IKE] message parsing failed
Jun  7 13:13:21 06[IKE] ID_PROT request with message ID 0 processing failed
Jun  7 13:13:28 02[IKE] IKE_SA (unnamed)[7] state change: CONNECTING => DESTROYING

Thanks to some googling, it appears that the error "ID_PROT request with message ID 0 processing failed" is related to the pre-shared key. The one I was using is a simple sentence with spaces and capitals, yet changing it to something else *without spaces* seems to fix it.

To make a long story short: spaces in the pre-shared key break IPsec connections to the RT-AC86U, at least on macOS and iOS. It would be great if somebody could test it on Android, Linux and Windows.
 
Last edited:
There was another user who reported ipsec server didn't work on his iphone.
That might be me... Still does not work.

I hope to perform a factory reset and start from scratch one of these days.
 
That might be me... Still does not work.

I hope to perform a factory reset and start from scratch one of these days.

You are specific case I think.
Is your ISP honest ?
Some ISP block well known vpn ports.
 
Performed a factory reset and set up my router from scratch.

Unfortunately with same result: I can set up a VPN connection from my iPhone (over 4G), but cannot access the router or the internet over that VPN connection...
 
Last edited:
I too am having this issue with Mac/iOS connection to IPSEC VPN on my AC86U. However, when I SSH into my router and type
Code:
ipsec stroke loglevel ike 2

I get the following error:
/usr/lib/ipsec/stroke: error while loading shared libraries: libpthread.so.0: cannot open shared object file: No such file or directory

My preshared key has no spaces, just Caps and Lowercase letters.
 
Huh, after executing the suggested command I can access the router and the internet over this VPN connection...

Code:
ipsec stroke loglevel ike 2
 
I get the following error:
Did you turn on IPSec server ?

look up the file.
Code:
find / -name libpthread.so*
firmware must have two for 32bit and 64bit
Code:
/lib/aarch64/libpthread.so.0
/lib/libpthread.so.0
 
Last edited:
Huh, after executing the suggested command I can access the router and the internet over this VPN connection...

Code:
ipsec stroke loglevel ike 2
weird, the command just change log level, doesn't affect to daemon.
 
Did you turn on IPSec server ?

look up the file.
Code:
find / -name libpthread.so*
firmware must have two for 32bit and 64bit
Code:
/lib/aarch64/libpthread.so.0
/lib/libpthread.so.0

Here's what I'm seeing:
Code:
admin@RT-AC86U:/tmp/home/root# find / -name libpthread.so*
/lib/aarch64/libpthread.so.0
/lib/libpthread.so.0
admin@RT-AC86U:/tmp/home/root# ipsec stroke loglevel ike 2
/usr/lib/ipsec/stroke: error while loading shared libraries: libpthread.so.0: cannot open shared object file: No such file or directory
admin@RT-AC86U:/tmp/home/root#
 
ldd /usr/lib/ipsec/stroke

Code:
admin@RT-AC86U:/tmp/home/root# ldd /usr/lib/ipsec/stroke
    linux-vdso.so.1 (0x0000007f9f1d7000)
    libstrongswan.so.0 => /usr/lib/ipsec/libstrongswan.so.0 (0x0000007f9f153000)
    libpthread.so.0 => not found
    libdl.so.2 => not found
    libm.so.6 => not found
    libc.so.6 => not found
    libpthread.so.0 => not found
    libdl.so.2 => not found
    libm.so.6 => not found
    libc.so.6 => not found
 
Code:
admin@RT-AC86U:/tmp/home/root# ldd /usr/lib/ipsec/stroke
    linux-vdso.so.1 (0x0000007f9f1d7000)
    libstrongswan.so.0 => /usr/lib/ipsec/libstrongswan.so.0 (0x0000007f9f153000)
    libpthread.so.0 => not found
    libdl.so.2 => not found
    libm.so.6 => not found
    libc.so.6 => not found
    libpthread.so.0 => not found
    libdl.so.2 => not found
    libm.so.6 => not found
    libc.so.6 => not found
I don't know why your stroke is broken.
Try re-flash the latest firmware.
 
I don't know why your stroke is broken.
Try re-flash the latest firmware.

Unfortunately reflashing the firmware didn't help. I'm still getting the same stroke errors. I'm running firmware 3.0.0.4.384_21045 and it is configured in a mesh with another AC86U.

I did notice that if I reboot the router, it will allow an IPSec connection from my iPhone for a short time, but after a few minutes if I try to connect from another device I start getting the same communication errors.
 
Unfortunately reflashing the firmware didn't help. I'm still getting the same stroke errors. I'm running firmware 3.0.0.4.384_21045 and it is configured in a mesh with another AC86U.

I did notice that if I reboot the router, it will allow an IPSec connection from my iPhone for a short time, but after a few minutes if I try to connect from another device I start getting the same communication errors.

Try on Merlin. Stock and Merlin firmware have different files.
And IPSec server allows only one connection per user.
 

Similar threads

Latest threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top