OpenVPN iOS: Server poll timeout / Strange connection behavior

[1h0dl3r]

Hello fellows,

i am currently experiencing a strange behavior of either my VPS server or my iPhone.

I recently setup a new VPS and used this script to setup a OpenVPN server. The VPS runs a standard installation of debian 10 and, for testing purposes, no firewall is active.

I can connect effortlessly by uploading the config file to my ASUS DSL-AC68U router.
But i cannot connect to the VPN using my iPhone. The server does not recognize any incoming connections and the OpenVPN app is caught in an event(wait) loop.
However, what astonishes me is that i am in fact able to connect once i try to initiate a ssh (login attempt is enough) prior to connecting to the VPN itself. Of course both services on the same server:

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
openvpn 654 nobody 6u IPv4 22381 0t0 UDP xx.xx.xx.xx:123
sshd 682 root 3u IPv4 22740 0t0 TCP *:22 (LISTEN)

Apps:
-OpenVPN for VPN
-WebSSH for SSH

When i reboot the phone, i always have to initiate the ssh connection (attempt) before connecting to the vpn. I also tried it on another iPhone. Same Problem.

I have no problem connecting to other VPN servers, neither by using PIAs native VPN app nor by using the same OpenVPN app with custom configs for e.g. my home server. So i guess my cellular provider is not blocking anything.

Any suggestions on this? Any help is appreciated!

Server config:

local xx.xx.xx.xx (remote ip)
port 123
proto udp
dev tun
ca ca.crt
cert server.crt
key server.key
dh dh.pem
auth SHA512
tls-crypt tc.key
topology subnet
server 10.8.0.0 255.255.255.0
push "redirect-gateway def1 bypass-dhcp"
ifconfig-pool-persist ipp.txt
push "dhcp-option DNS xx.xx.xx.xx"
keepalive 10 120
cipher AES-256-CBC
user nobody
group nogroup
persist-key
persist-tun
verb 3
crl-verify crl.pem
explicit-exit-notify

Client config:

client
dev tun
proto udp
remote xx.xx.xx.xx 123
resolv-retry infinite
nobind
persist-key
persist-tun
remote-cert-tls server
auth SHA512
cipher AES-256-CBC
ignore-unknown-option block-outside-dns
block-outside-dns
verb 3
<ca>
-----BEGIN CERTIFICATE-----
xxx
-----END CERTIFICATE-----
</ca>
<cert>
-----BEGIN CERTIFICATE-----
xxx
-----END CERTIFICATE-----
</cert>
<key>
-----BEGIN PRIVATE KEY-----
xxx
-----END PRIVATE KEY-----
</key>
<tls-crypt>
-----BEGIN OpenVPN Static key V1-----
xxx
-----END OpenVPN Static key V1-----
</tls-crypt>

UNsuccessful connection attempt using OpenVPN iOS app:

2021-03-12 16:47:02 1

2021-03-12 16:47:02 ----- OpenVPN Start -----
OpenVPN core 3.git::58b92569 ios arm64 64-bit

2021-03-12 16:47:02 OpenVPN core 3.git::58b92569 ios arm64 64-bit
2021-03-12 16:47:02 Frame=512/2048/512 mssfix-ctrl=1250

2021-03-12 16:47:02 UNUSED OPTIONS

4 [resolv-retry] [infinite]
5 [nobind]
6 [persist-key]
7 [persist-tun]
11 [ignore-unknown-option] [block-outside-dns]
12 [block-outside-dns]
13 [verb] [3]

2021-03-12 16:47:02 EVENT: RESOLVE
2021-03-12 16:47:02 Contacting [xx.xx.xx.xx]:123/UDP via UDP
2021-03-12 16:47:02 EVENT: WAIT
2021-03-12 16:47:02 Connecting to [xxx.xxx.com]:123 (xx.xx.xx.xx) via UDPv4
2021-03-12 16:47:13 Server poll timeout, trying next remote entry...
2021-03-12 16:47:13 EVENT: RECONNECTING
2021-03-12 16:47:13 Contacting [xx.xx.xx.xx]:123/UDP via UDP
2021-03-12 16:47:13 EVENT: WAIT
2021-03-12 16:47:13 Connecting to [xx.xxx.com]:123 (xx.xx.xx.xx) via UDPv4

Successful connection attempt using OpenVPN iOS app after ssh login attempt using WebSSH iOS app:

2021-03-12 16:45:40 ----- OpenVPN Start -----
OpenVPN core 3.git::58b92569 ios arm64 64-bit

2021-03-12 16:45:40 OpenVPN core 3.git::58b92569 ios arm64 64-bit

2021-03-12 16:45:40 Frame=512/2048/512 mssfix-ctrl=1250

2021-03-12 16:45:40 UNUSED OPTIONS
4 [resolv-retry] [infinite]
5 [nobind]
6 [persist-key]
7 [persist-tun]
11 [ignore-unknown-option] [block-outside-dns]
12 [block-outside-dns]
13 [verb] [3]

2021-03-12 16:45:40 EVENT: RESOLVE
2021-03-12 16:45:40 Contacting [xx.xx.xx.xx]:123/UDP via UDP
2021-03-12 16:45:40 EVENT: WAIT
2021-03-12 16:45:40 Connecting to [xxxx.xxxx.com]:123 (xx.xx.xx.xx) via UDPv4
2021-03-12 16:45:41 EVENT: CONNECTING

2021-03-12 16:45:41 Tunnel Options:V4,dev-type tun,link-mtu 1601,tun-mtu 1500,proto UDPv4,cipher AES-256-CBC,auth SHA512,keysize 256,key-method 2,tls-client

2021-03-12 16:45:41 Creds: UsernameEmpty/PasswordEmpty

2021-03-12 16:45:41 Peer Info:
IV_VER=3.git::58b92569
IV_PLAT=ios
IV_NCP=2
IV_TCPNL=1
IV_PROTO=2
IV_IPv6=0
IV_AUTO_SESS=1
IV_GUI_VER=net.openvpn.connect.ios_3.2.3-3760
IV_SSO=openurl

2021-03-12 16:45:41 VERIFY OK: depth=1, /CN=ChangeMe
2021-03-12 16:45:41 VERIFY OK: depth=0, /CN=server
2021-03-12 16:45:41 SSL Handshake: CN=server, TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 2048 bit RSA
2021-03-12 16:45:41 Session is ACTIVE
2021-03-12 16:45:41 EVENT: GET_CONFIG
2021-03-12 16:45:41 Sending PUSH_REQUEST to server...

2021-03-12 16:45:41 OPTIONS:
0 [redirect-gateway] [def1] [bypass-dhcp]
1 [dhcp-option] [DNS] [xx.xx.xx.xx]
2 [route-gateway] [10.8.0.1]
3 [topology] [subnet]
4 [ping] [10]
5 [ping-restart] [120]
6 [ifconfig] [10.8.0.2] [255.255.255.0]
7 [peer-id] [1]
8 [cipher] [AES-256-GCM]
9 [block-ipv6]


2021-03-12 16:45:41 PROTOCOL OPTIONS:
cipher: AES-256-GCM
digest: NONE
compress: NONE
peer ID: 1

2021-03-12 16:45:41 EVENT: ASSIGN_IP
2021-03-12 16:45:41 NIP: preparing TUN network settings
2021-03-12 16:45:41 NIP: init TUN network settings with endpoint: xx.xx.xx.xx
2021-03-12 16:45:41 NIP: adding IPv4 address to network settings 10.8.0.2/255.255.255.0
2021-03-12 16:45:41 NIP: adding (included) IPv4 route 10.8.0.0/24
2021-03-12 16:45:41 NIP: redirecting all IPv4 traffic to TUN interface
2021-03-12 16:45:41 NIP: adding DNS xx.xx.xx.xx
2021-03-12 16:45:41 NIP: blocking all IPv6 traffic
2021-03-12 16:45:41 Connected via NetworkExtensionTUN

2021-03-12 16:45:41 EVENT: CONNECTED xxx.xxx.com:123 (xx.xx.xx.xx) via /UDPv4 on NetworkExtensionTUN/10.8.0.2/ gw=[/]

2021-03-12 16:45:43 EVENT: DISCONNECTED

2021-03-12 16:45:43 Raw stats on disconnect:
BYTES_IN : 3619
BYTES_OUT : 3942
PACKETS_IN : 11
PACKETS_OUT : 17
TUN_BYTES_IN : 300
TUN_PACKETS_IN : 5

2021-03-12 16:45:43 Performance stats on disconnect:
CPU usage (microseconds): 69208
Tunnel compression ratio (downlink): inf
Network bytes per CPU second: 109250
Tunnel bytes per CPU second: 4334

Incoming connection to VPS after ssh login attempt:

Mar 12 16:41:01 server openvpn[654]: xx.xxx.xxx.xx:24917 TLS: Initial packet from [AF_INET]xx.xxx.xxx.xx:24917, sid=c375a3cd 9eccbdd1
Mar 12 16:41:01 server openvpn[654]: xx.xxx.xxx.xx:24917 VERIFY OK: depth=1, CN=ChangeMe
Mar 12 16:41:01 server openvpn[654]: xx.xxx.xxx.xx:24917 VERIFY OK: depth=0, CN=r
Mar 12 16:41:01 server openvpn[654]: xx.xxx.xxx.xx:24917 peer info: IV_VER=3.git::58b92569
Mar 12 16:41:01 server openvpn[654]: xx.xxx.xxx.xx:24917 peer info: IV_PLAT=ios
Mar 12 16:41:01 server openvpn[654]: xx.xxx.xxx.xx:24917 peer info: IV_NCP=2
Mar 12 16:41:01 server openvpn[654]: xx.xxx.xxx.xx:24917 peer info: IV_TCPNL=1
Mar 12 16:41:01 server openvpn[654]: xx.xxx.xxx.xx:24917 peer info: IV_PROTO=2
Mar 12 16:41:01 server openvpn[654]: xx.xxx.xxx.xx:24917 peer info: IV_IPv6=0
Mar 12 16:41:01 server openvpn[654]: xx.xxx.xxx.xx:24917 peer info: IV_AUTO_SESS=1
Mar 12 16:41:01 server openvpn[654]: xx.xxx.xxx.xx:24917 peer info: IV_GUI_VER=net.openvpn.connect.ios_3.2.3-3760
Mar 12 16:41:01 server openvpn[654]: xx.xxx.xxx.xx:24917 peer info: IV_SSO=openurl
Mar 12 16:41:01 server openvpn[654]: xx.xxx.xxx.xx:24917 Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 2048 bit RSA
Mar 12 16:41:01 server openvpn[654]: xx.xxx.xxx.xx:24917 [r] Peer Connection Initiated with [AF_INET]xx.xxx.xxx.xx:24917
Mar 12 16:41:01 server openvpn[654]: r/xx.xxx.xxx.xx:24917 MULTI_sva: pool returned IPv4=10.8.0.2, IPv6=(Not enabled)
Mar 12 16:41:01 server openvpn[654]: r/xx.xxx.xxx.xx:24917 MULTI: Learn: 10.8.0.2 -> r/xx.xxx.xxx.xx:24917
Mar 12 16:41:01 server openvpn[654]: r/xx.xxx.xxx.xx:24917 MULTI: primary virtual IP for r/xx.xxx.xxx.xx:24917: 10.8.0.2
Mar 12 16:41:01 server openvpn[654]: r/xx.xxx.xxx.xx:24917 PUSH: Received control message: 'PUSH_REQUEST'
Mar 12 16:41:01 server openvpn[654]: r/xx.xxx.xxx.xx:24917 SENT CONTROL [r]: 'PUSH_REPLY,redirect-gateway def1 bypass-dhcp,dhcp-option DNS xx.xxx.xx.xx,route-gateway 10.8.0.1,topology subnet,ping 10,ping-restart 120,ifconfig 10.8.0.2 255.255.255.0,peer-id 1,cipher AES-256-GCM' (status=1)
Mar 12 16:41:01 server openvpn[654]: r/xx.xxx.xxx.xx:24917 Data Channel: using negotiated cipher 'AES-256-GCM'
Mar 12 16:41:01 server openvpn[654]: r/xx.xxx.xxx.xx:24917 Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Mar 12 16:41:01 server openvpn[654]: r/xx.xxx.xxx.xx:24917 Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Mar 12 16:41:04 server openvpn[654]: r/xx.xxx.xxx.xx:24917 SIGTERM[soft,remote-exit] received, client-instance exiting

 

[eibgrad]

I don't know if it would make a difference, but just in case, are you *sure* the phone is using the cellular network when this occurs, and NOT perhaps wifi? I would try both ways, making sure to only have one or the other (cellular vs. wifi) connected at the time. I just want to see if perhaps there is a difference and if this might provide a clue. Because it does sound pretty weird.

 

[1h0dl3r]

Thank you for that hint.

And yes, it makes a difference. When i am connected to (any) Wifi, i can connect to the VPN without any problems. The problem only seems to exist when using the cellular network (Telekom Germany).

What i do not understand is, why i can connect to other VPN servers with that same cellular connection and the same specs (like mssfix-ctrl 1250, mtu 1601,tun-mtu 1500).

Furthermore, when i open a hotspot on the iPhone and try to connect from that cellular connection by using the PC it SOMETIMES works:

Syslog on error with open hotspot on PC:

Mar 12 17:46:56 nm-openvpn[35332]: TCP/UDP: Preserving recently used remote address: [AF_INET]xx.xx.xx.x:123
Mar 12 17:46:56 nm-openvpn[35332]: UDP link local: (not bound)
Mar 12 17:46:56 nm-openvpn[35332]: UDP link remote: [AF_INET]xx.xx.xx.xx:123
Mar 12 17:46:56 nm-openvpn[35332]: NOTE: chroot will be delayed because of --client, --pull, or --up-delay
Mar 12 17:46:56 nm-openvpn[35332]: NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Mar 12 17:47:56 nm-openvpn[35332]: TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Mar 12 17:47:56 nm-openvpn[35332]: TLS Error: TLS handshake failed
Mar 12 17:47:56 nm-openvpn[35332]: SIGUSR1[soft,tls-error] received, process restarting
Mar 12 17:47:57 NetworkManager[1041]: <warn>  [1615567677.3658] vpn-connection[0x5606bee5e350,dbf84d5d-2bdb-41e2-8ff0-9d6272b1cd91,"r",0]: VPN connection: connect timeout exceeded.
Mar 12 17:47:57 nm-openvpn-serv[35328]: Connect timer expired, disconnecting.

Syslog on success with open hotspot on PC:

Mar 12 17:57:52 nm-openvpn[38234]: OpenVPN 2.4.9 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Aug 18 2020
Mar 12 17:57:52 nm-openvpn[38234]: library versions: OpenSSL 1.1.1f  31 Mar 2020, LZO 2.10
Mar 12 17:57:52 nm-openvpn[38234]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Mar 12 17:57:52 nm-openvpn[38234]: TCP/UDP: Preserving recently used remote address: [AF_INET]xx.xx.xx.xx:123
Mar 12 17:57:52 nm-openvpn[38234]: UDP link local: (not bound)
Mar 12 17:57:52 nm-openvpn[38234]: UDP link remote: [AF_INET]xx.xx.xx.xx:123
Mar 12 17:57:52 nm-openvpn[38234]: [server] Peer Connection Initiated with [AF_INET]xx.xx.xx.xx:123
Mar 12 17:57:53 nm-openvpn[38234]: TUN/TAP device tun0 opened

Thanks.

 

[eibgrad]

I noticed your config uses port 123, which is the well-known NTP (time) protocol. Is this the actual port you're using, or are you masking it for the purposes of this forum? Because many ports in the 1-1024 range are blocked by ISPs, or perhaps already actively being used by the server for NTP. If it is 123, try something else, like 10000. Even if it's not the source of the immediate problem, it's best to stay away from the well-known ports to avoid issues.

 

[1h0dl3r]

I know. I already tried different ports like 1194, 11994, 10000, 4000. Does not work as well.

Originally i though that the problem was UDP relevated, because TCP works right out of the box with any port on the cellular network.

However, I do not understand why the connection works with UDP when i try to ssh into the server prior to initializing the VPN connection. Does this even make sense?

 

[eibgrad]

So this problem is further limited to only UDP? TCP works?! Any other details we need to know?

 

[eibgrad]

If using UDP, try adding the following directive to the OpenVPN server config file and restarting it.

multihome

P.S. I see you're using the local directive in the OpenVPN server config file, so this shouldn't be necessary, but try it anyway.

P.S. I also see you masked the IP on that local directive, but the masking says "(remote IP)"? I'm not sure what that means to you, but that has to be the *local* IP of the OpenVPN server. And local as in the VPS's public IP, not the *private* local network that's also available on the VPS.

 

[1h0dl3r]

I have made a video of how it looks like first hand.

Yes, TCP works out of the box. But "multihome" did not help.

multihome

Yes, the local directive is the public IP of the VPS.

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 11:11:0:48:b2:5c brd ff:ff:ff:ff:ff:ff
    inet xx.xx.xx.xx/24 brd xx.xx.xx.xx scope global eth0
       valid_lft forever preferred_lft forever
4: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 100
    link/none
    inet 10.8.0.1/24 brd 10.8.0.255 scope global tun0
       valid_lft forever preferred_lft forever

The IP should not be the problem as it works when not using the cellular.

It seems like the ssh "knock" opens the udp door. But i do not know why that is and why i have no problem when being connected to Wifi. The router connects to the VPN without any problems (see attached screenshot).

 

[1h0dl3r]

Before ssh knock: Nothing
After ssh knock: UDP packets are incoming

tcpdump udp port 123:

21:09:05.506973 IP xx.xx.xx.xx.123 > tmo-xxx-xxx.customers.d1-online.com.1872: NTPv1, unspecified, length 88
21:09:05.506991 IP xx.xx.xx.xx.123 > tmo-xxx-xxx.customers.d1-online.com.1872: NTPv1, unspecified, length 76
21:09:05.507220 IP xx.xx.xx.xx.123 > tmo-xxx-xxx.customers.d1-online.com.1872: NTPv1, unspecified, length 656
21:09:05.507257 IP xx.xx.xx.xx.123 > tmo-xxx-xxx.customers.d1-online.com.1872: NTPv1, unspecified, length 107

 

[eibgrad]

Sounds like a firewall issue on the VPS. Or some other software that's functioning similarly. And the VPS is one thing only you have access to and can examine. Try dumping the process table and see if anything unusual is running.

 

[1h0dl3r]

I have also tried a fresh Ubuntu 20.04. The problem persists.

Kernel:

Linux ffm 5.4.0-66-generic #74-Ubuntu SMP Wed Jan 27 22:54:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

OS:

NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"

Route:

default via xx.xx.xx.254 dev eth0
10.8.0.0/24 dev tun0 proto kernel scope link src 10.8.0.1
xx.xx.xx.0/24 dev eth0 proto kernel scope link src xx.xx.xx.xx
xx.xx.xx.254 dev eth0 scope link

IPtables:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination 
ACCEPT     udp  --  anywhere             anywhere             udp dpt:ntp


Chain FORWARD (policy ACCEPT)
target     prot opt source               destination 
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     all  --  10.8.0.0/24          anywhere   


Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Th status-server.log just shows my connected router:

TITLE,OpenVPN 2.4.7 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Sep  5 2019
TIME,Fri Mar 12 21:57:21 2021,1615582641
HEADER,CLIENT_LIST,Common Name,Real Address,Virtual Address,Virtual IPv6 Address,Bytes Received,Bytes Sent,Connected Since,Connected Since (time_t),Username,Client ID,Peer ID
CLIENT_LIST,router,xx.xx.xx.xx:37771,10.8.0.3,,16802429,322847772,Fri Mar 12 20:46:32 2021,1615578392,UNDEF,2,0
HEADER,ROUTING_TABLE,Virtual Address,Common Name,Real Address,Last Ref,Last Ref (time_t)
ROUTING_TABLE,10.8.0.3,router,xx.xx.xx.xx:37771,Fri Mar 12 21:57:19 2021,1615582639
GLOBAL_STATS,Max bcast/mcast queue length,0
END

Server date and time:

journalctl -au openvpn
-- Logs begin at Fri 2021-03-12 09:04:13 CET, end at Fri 2021-03-12 22:01:06 CET. --
Mär 12 12:50:47 ffm systemd[1]: Starting OpenVPN service...
Mär 12 12:50:47 ffm systemd[1]: Finished OpenVPN service.

shell  2,14s user 0,67s system 0% cpu 5:24:31,14 total
children  2,47s user 2,67s system 0% cpu 5:24:31,14 total

Fr 12. Mär 22:01:17 CET 2021

 

[eibgrad]

Well at this point I'm out of ideas. Seems we know one thing for sure; it all works *except* for the smartphone, when using cellular and udp. And if that isn't strange enough, if you make an ssh attempt (not even a need to get connected), now even that works.

All I can assume it's something quirky in iOS. Esp. since in every other scenario, regardless of client platform, protocol (tcp or udp), cellular vs. wifi, it works.

Might be time to start searching the Apple forums for an explanation as well.

P.S. If you want, I have an Apple smartphone as well (iPhone SE 11), and we could see if I have the same problem. Maybe there's some application conflict on your phone that I don't have. If I got connected w/o a problem, then that would be my assumption. But that's up to you. As I said, I'm otherwise out of ideas.

 

[1h0dl3r]

Thank you for your time and help, appreciate it!

What really grinds my gears is that i cannot even exclude anything.
I'll order a VPS somewhere else just for testing purposes (although my home server works with the same setup/config) and also wrote to technical support. Maybe they have an idea although i'm not expecting any support for a cheap VPS.

 

[1h0dl3r]

Update:
"I can confirm that this would have been a result of the default DDoS protection configuration. We've adjusted the configuration in order to resolve the issue."

It works now.