VPN Brain Teaser...

[sfx2000]

Here's a brain teaser for you VPN fans... and a lesson learned

use case - OpenVPN is set up for client/server, with server being routable, sitting behind a router, with ovpn and ssh ports forwarded - basically offsite access into the LAN resources

Here's a rough graphic

Laptop ----- MobileHotSpot -------- Router-------- Server (OVPN/SSH/etc..)
(LocalIP)       166.170.a.b         68.0.d.e       192.168.1.85
172.20.20.8                         TCP/22         assigns OPVPN IP's
(OvpnIP)                            UDP/1194
10.0.8.5                            (port forward from backend host)

OpenDNS is pushed to client for VPN access - otherwise uses the DHCP assigned DNS from hotspot provider when VPN not in use

Router is running unbound DNS server as a caching/resolver on FreeBSD 10.2

Backend is statically configured to use Google Public DNS - and is set to no block any outgoing connections (multiple services supporting the LAN side)

LAN side is heavy on mDNS deployment, and IPv6 is running inside in a link-local config, OpenVPN is configed as IPv4 only, and IPv6 is disabled on the router for WAN side.

Scenario:

Laptop connects to Ovpn endpoint via vpn.example.net at 68.0.d.e, assigned 10.0.8.5 as remote. OvenVPN see the 166.170.a.b address, pushes the right routing info, DHCP, DNS, etc - no problem

Laptop connects to OpenSSH vis user@192.168.1.85, no problem - backend host sees the connection from 10.0.8.5 - This is expected behavior

Now keep in mind - all IPv4 traffic on the client is configured transit the OpenVPN tunnel

Laptop then does an ssh connection to user@example.com - guess what happens?

Yep, the SSH server now sees the connection coming not from 10.0.8/5. but from 166.170.a.b - which was unexpected as it exposed the Hotspot IP connected from (the hotspot NAT's the laptop)

Now here's where it gets weird... internally, the router also has SSH running, so inside the LAN, the router is resolving example.com to 192.168.1.1, which again is expected behavior, so it doesn't go to the backend server - to get the to backend server from LAN side, either user@192.168.1.85, or user@hostname.local

What happened? Is it the client doing the wrong thing, or the router/gw, or the OpenVPN server?

How to correct?

(I did eventually sort out what happened, but this is an excellent thought exercise on what can happen with VPN if one is not careful, not only in the OpenVPN setup, but the Router/GW setup)

 

[sfx2000]

So in a nutshell - how did the hotspot's IP get exposed inside the LAN as a public IP, and not a private IP?

 

[kvic]

So in a nutshell - how did the hotspot's IP get exposed inside the LAN as a public IP, and not a private IP?

DNS leakage on opvn client side..?

 

[sfx2000]

DNS leakage on opvn client side..?

That's what I thought at first... so double checked things there - Tunnelblick 3.6.3 on El Cap (OSX 10.11), so rounded up a couple of other devices, and same results...

 

[sfx2000]

Another hint - this is on the Ovpn Server side - docker0 and virbr0 are docker and kvm respectively - box has a wifi card in it, but is disabled via rfkill, and has never associated with in house Ap's in any event.

$ ip route
default via 192.168.1.1 dev enp4s0  proto static  metric 100 
10.8.0.0/24 via 10.8.0.2 dev tun0 
10.8.0.2 dev tun0  proto kernel  scope link  src 10.8.0.1 
169.254.0.0/16 dev tun0  scope link  metric 1000 
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1 linkdown 
192.168.1.0/24 dev enp4s0  proto kernel  scope link  src 192.168.1.20  metric 100 
192.168.122.0/24 dev virbr0  proto kernel  scope link  src 192.168.122.1 linkdown

$ nmcli device status

DEVICE      TYPE      STATE        CONNECTION 
docker0     bridge    connected    docker0    
virbr0      bridge    connected    virbr0     
enp4s0      ethernet  connected    homenet    
tun0        tun       connected    tun0       
wlp2s0      wifi      unavailable  --         
lo          loopback  unmanaged    --         
virbr0-nic  tun       unmanaged    --

<MAC addr managled>

$ ifconfig
docker0   Link encap:Ethernet  HWaddr 02:42:b0:11:11:11  
          inet addr:172.17.0.1  Bcast:0.0.0.0  Mask:255.255.0.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

enp4s0    Link encap:Ethernet  HWaddr 74:e6:e2:11:11:11  
          inet addr:192.168.1.20  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::d9db:fdf7:1111:1111/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1623423 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1000936 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1816150076 (1.8 GB)  TX bytes:407477730 (407.4 MB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:33554 errors:0 dropped:0 overruns:0 frame:0
          TX packets:33554 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:18602152 (18.6 MB)  TX bytes:18602152 (18.6 MB)

tun0      Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
          inet addr:10.8.0.1  P-t-P:10.8.0.2  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:95779 errors:0 dropped:0 overruns:0 frame:0
          TX packets:104297 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100 
          RX bytes:29304680 (29.3 MB)  TX bytes:101293854 (101.2 MB)

virbr0    Link encap:Ethernet  HWaddr 00:00:00:00:00:00  
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

 

[sfx2000]

Note that the OpenVPN server is NAT'ed, it's not facing the WAN directly, just TCP/22 and UCP/1194 forwarded via the router...

So from Internal, we're connecting to the server via ssh on enp4s0, and when in ovpn, we're using tun0 - getting close to what the problem ended up being..

The fix was fun once the problem was understood...

 

[kvic]

That's what I thought at first... so double checked things there - Tunnelblick 3.6.3 on El Cap (OSX 10.11), so rounded up a couple of other devices, and same results...

Lots of smoke and mirror..lol. Back to your first post:

OpenDNS is pushed to client for VPN access - otherwise uses the DHCP assigned DNS from hotspot provider when VPN not in use

Laptop then does an ssh connection to user@example.com - guess what happens?

You didn't mention if you intended to setup a split tunnel or not. Also if you force redirection of all traffic through VPN. Though you pushed a public DNS, OpenDNS to clients..

Assume you didn't force all traffic through VPN, then when you ssh user@example.com, the host name is resolved by OpenDNS to 68.0.d.e. The SSH connection goes outside VPN and reached your router and then forwarded to your server. Hence, your server sees the client from a public IP of the Wifi Hotspot.

Assume you did enforce all traffic through VPN. When you ssh user@example.com, OpenDNS (through VPN) resolves to 68.0.d.e. Now the ssh connection goes through VPN and reached your server. It's routed back to the router which then port forwarded back to the server's SSHD (convoluted and inefficient IMO..). You seem to imply this scenario. I don't quite get why the server's SSHD see a client with the public IP of the Wifi Hotspot...perhaps NAT codes did some wonder.

Perhaps we shall wait for more participants..before you open the riddle.

 

[sfx2000]

Enforcing all traffic thru VPN... and you're really close

 

[sfx2000]

It was a implicit hairpin - coming from tun0, the server resolved example.com to the public IP, and then we get into what some call the 'port fail' issue that the VPN community raised sometime back..

The quick fix was to add a local zone entry into my unbound dns server..

/var/unbound/unbound.conf

local-zone: "example.com" redirect
local-data: "example.com A 192.168.1.20"

So basically everything was working _as designed_ - just had to add the local zone to tighten things up because of the port-forward.

I'm sure someone could do a quick IP tables solution as well, but I found DNS fix to be a more elegant approach, as my requirement is to keep the openssh server also public.

neat eh?

Just goes to show - configs can get one into trouble, esp. if folks are doing cut-paste without fully understanding what they're doing - for the most part, my config was good, no DNS leak issues, except for the 'intranet' where sshd and the vpn server were both in the same zone.

 

[kvic]

The quick fix was to add a local zone entry into my unbound dns server..

/var/unbound/unbound.conf

local-zone: "example.com" redirect
local-data: "example.com A 192.168.1.20"

So basically everything was working _as designed_ - just had to add the local zone to tighten things up because of the port-forward.

You have OpenDNS pushed to VPN clients. I wonder how this could resolve the issue if you don't push the router as name server instead..

Another way is to direct SSH requests from tun0 to local SSHD through iptables. This also reduces bouncing packets between the router and the server.

Have you looked into why SSHD see's public IP of the client?

Doesn't look like the same as VPN "port fail"...which I just heard of it from you...google up..and two groups of people seem to have different interpretations of what "port fail' is..both seem valid but totally different mechanisms of failures.

 

[sfx2000]

The way is to direct SSH requests from tun0 to local SSHD through iptables. This also reduces bouncing packets between the router and the server.

Doing the local-zone, local-data approach keeps data from even hitting IP tables in the router, as the local lookup resolves direct to the server...

Have you looked into why SSHD see's public IP of the client?

It saw the IP of the hot spot - looking at the pcap's, it the routing info in the packet itself.. and this is with client and at the server - it's pretty clear in this case - but it's an edge case of having port forwarded the ssh port.

Don't want/need to modify FW rules on the router, as there are many times I use the ssh direct from the WAN side...

Like I've mentioned before - VPN doesn't guarantee anonymity, as clients and servers are going to do what they do - it will protect the payload inside...

 

[kvic]

Like I've mentioned before - VPN doesn't guarantee anonymity, as clients and servers are going to do what they do - it will protect the payload inside...

That's true..

I'm surprised to find out why VPN so popular in places like North America and Europe. Keep spy away from the government maybe one little reason. People seem to think they won't get caught during bittorent!

 

[sfx2000]

That's true..

I'm surprised to find out why VPN so popular in places like North America and Europe. Keep spy away from the government maybe one little reason. People seem to think they won't get caught during bittorent!

Anyways - this is/was an edge case - was fun to look into, rest of testing went fine, everything else was seeing the IP of the server end of the link (not the server IP as it's nat'ed, but the public/wan side IP of the router.