RT-AC68U: Unable to add 2nd OpenVPN client.

[Pete]

Hi,

I am using a TM-1900 router (800 MHz), upgraded to AC68U. Firmware 380.65.

I configured OpenVPN with NordVPN.com. OpenVPN clients on the router 1 and 5 works fine. However, if I add a third client, it does not route the connections as per the "Policy Rules". Instead, the connections are routed as per the client 5.

Update: 2nd OpenVPN client is not working. Only one client is routing all the requests.

Is there anything I am missing?

 

[RMerlin]

Is there anything I am missing?

If you have overlapping routes, then the first ones will have priority.

 

[Pete]

Thanks.

There is no overlapping routes. VPN1 directs a few specific IPs through a VPN server. VPN5 routes rest of the IPs to a different VPN server.

Looks like the policy rules are not processed as expected.

 

[Martineau]

Thanks.

There is no overlapping routes. VPN1 directs a few specific IPs through a VPN server. VPN5 routes rest of the IPs to a different VPN server.

Looks like the policy rules are not processed as expected.

You will need to check the routes/rules for inconsistencies.

So issue the three commands (where X is the VPN instance Number) after you manually start VPN X one after the other.

ip route show table main
ip route show table 11x
ip rule

the output of 'ip rule' determines the order in which your Policy routing rules are applied, and the individual table 11x should only refer to tun1x routes.

 

[Pete]

Thanks.

Here is the output for your reference.

All the traffic is flowing through VPN1, not through VPN5.

admin@RT-AC68U-17F8:/tmp/home/root# ip route show table main
202.83.21.24 via 10.245.0.1 dev eth0  metric 1
107.191.33.51 via 10.244.0.1 dev ppp0
10.244.0.1 dev ppp0  proto kernel  scope link
202.83.21.28 via 10.245.0.1 dev eth0  metric 1
185.145.38.237 via 10.244.0.1 dev ppp0
192.168.68.0/24 dev br0  proto kernel  scope link  src 192.168.68.1
10.8.8.0/24 dev tun11  proto kernel  scope link  src 10.8.8.35
10.8.8.0/24 dev tun15  proto kernel  scope link  src 10.8.8.57
10.245.0.0/16 dev eth0  proto kernel  scope link  src 10.245.184.189
127.0.0.0/8 dev lo  scope link
0.0.0.0/1 via 10.8.8.1 dev tun11
128.0.0.0/1 via 10.8.8.1 dev tun11
default via 10.244.0.1 dev ppp0
default via 10.245.0.1 dev eth0  metric 1

admin@RT-AC68U-17F8:/tmp/home/root# ip route show table 111
202.83.21.24 via 10.245.0.1 dev eth0  metric 1
10.244.0.1 dev ppp0  proto kernel  scope link
107.191.33.51 via 10.244.0.1 dev ppp0
202.83.21.28 via 10.245.0.1 dev eth0  metric 1
192.168.68.0/24 dev br0  proto kernel  scope link  src 192.168.68.1
10.8.8.0/24 dev tun11  proto kernel  scope link  src 10.8.8.35
10.245.0.0/16 dev eth0  proto kernel  scope link  src 10.245.184.189
127.0.0.0/8 dev lo  scope link
0.0.0.0/1 via 10.8.8.1 dev tun11
128.0.0.0/1 via 10.8.8.1 dev tun11
default via 10.8.8.1 dev tun11
default via 10.245.0.1 dev eth0  metric 1

admin@RT-AC68U-17F8:/tmp/home/root# ip route show table 115
202.83.21.24 via 10.245.0.1 dev eth0  metric 1
10.244.0.1 dev ppp0  proto kernel  scope link
107.191.33.51 via 10.244.0.1 dev ppp0
202.83.21.28 via 10.245.0.1 dev eth0  metric 1
185.145.38.237 via 10.244.0.1 dev ppp0
192.168.68.0/24 dev br0  proto kernel  scope link  src 192.168.68.1
10.8.8.0/24 dev tun11  proto kernel  scope link  src 10.8.8.35
10.245.0.0/16 dev eth0  proto kernel  scope link  src 10.245.184.189
127.0.0.0/8 dev lo  scope link
0.0.0.0/1 via 10.8.8.1 dev tun11
128.0.0.0/1 via 10.8.8.1 dev tun11
default via 10.8.8.1 dev tun11
default via 10.245.0.1 dev eth0  metric 1

admin@RT-AC68U-17F8:/tmp/home/root# ip rule
0:      from all lookup local
10101:  from all to 65.213.75.2 lookup ovpnc1
10102:  from all to 4.34.104.184 lookup ovpnc1
10103:  from all to 198.58.15.6 lookup ovpnc1
10104:  from all to 204.245.22.99 lookup ovpnc1
10801:  from 192.168.68.3 lookup main
10802:  from 192.168.68.4 lookup main
10803:  from 192.168.68.87 lookup main
10901:  from 192.168.68.0/24 lookup ovpnc5
32766:  from all lookup main
32767:  from all lookup default

 

[Martineau]

Here is the output for your reference.
All the traffic is flowing through VPN1, not through VPN5

For most users, the RPDB tables need only contain one 'default' rule

e.g.

admin@RT-AC68U-17F8:/tmp/home/root# ip route show table 111

default via 10.8.8.1 dev tun11

admin@RT-AC68U-17F8:/tmp/home/root# ip route show table 115

default via 10.8.8.1 dev tun11

The 'error' is shown in table 115...... as you can see it uses the tun11 device instead of the tun15 device!

/usr/bin/vpnrouting.sh is incorrectly (or by design) cloning the table 'main' routes based on these two conflicting routes:

10.8.8.0/24 dev tun11  proto kernel  scope link  src 10.8.8.35
10.8.8.0/24 dev tun15  proto kernel  scope link  src 10.8.8.57

 

[Pete]

Thanks for your quick analysis.

Anyone else is facing similar issue?

Any resolution?

 

[Martineau]

Any resolution?

Well, given the /usr/bin/vpnrouting.sh is incredibly stupid...i.e. it knows when creating say RPDB table 115, that the 'default dev' statement should always match (at least for the current Policy routing RPDB design) the last two digits of the table number but it doesn't alert the user to a misconfigured table!

So you could hack the script (or manually edit/correct table 115 etc.), however, first I would advise that you contact your VPN provider to see if multiple concurrent Client connections from the same physical device are allowed.

With my VPN provider they always manage to ensure that each VPN client is on a separate subnet
e.g. VPN Client 1 and VPN Client 2

10.200.195.0/24 dev tun11  proto kernel  scope link  src 10.200.195.6
10.200.196.0/24 dev tun12  proto kernel  scope link  src 10.200.196.68

so there is no conflicting VPN route - consequently table 112 correctly uses the tun12 device.
Obviously if you had two separate VPN providers then hopefully this would also solve your issue!

 

[Pete]

Thanks for your support.

I have tried with different VPN provider for the second VPN connection and worked as expected.

Do I have to rollback to previous version to overcome this script error?

Thanks.

 

[Pete]

If you have overlapping routes, then the first ones will have priority.

Based on Martineau's comments, there seems to be a known bug/limitation in the vpnrouting.sh script.

Any thoughts or resolution?

Thanks.

 

[Martineau]

Any thoughts or resolution?

What happens if you run this script after both your VPN Clients have connected?

#!/bin/sh

# Remove these from table main
ip route delete 0.0.0.0/1
ip route delete 128.0.0.0/1

ip route | grep tun1 | sort -k 2.1 | while read LINE
 do
 
  IPADDR=$(echo $LINE  | awk '{print $NF}')
  DEV=$(echo $LINE | awk '{print $3}')
  ID=${DEV:3:2}
  TABLE="1"$ID
 
  # Modify the RPDB table used by the VPN Client
  echo -e "\n\n\tProcessing Routing table...." $TABLE"\n"
  ip route flush table $TABLE 2> /dev/null > /dev/null      # Destroy table 111-ovpnc1,115-ovpnc5 etc.
  #ip route del default table $TABLE 2> /dev/null > /dev/null
  ip route show table $TABLE
  ip route add default via $IPADDR dev $DEV table $TABLE  # Add the 'default route for the VPN Client via the correct device interface!
  ip route flush cache
 
  # Show what we have done and the test results...
  echo -e "\n\n\tRouting table ovpnc"${DEV:4:1}"\n"
  ip route show table $TABLE
  echo -e "\n\tVPN Client ${DEV:4:1} endpoint I/P is" $(curl -s --interface $DEV http://ipecho.net/plain)"\n"
 
 done
exit 0

Basically this hack uses the I/P addresses from the output of the 'ip route' command

e.g.

10.8.8.0/24 dev tun11  proto kernel  scope link  src xxx.xxx.xxx.xxx
10.8.8.0/24 dev tun15  proto kernel  scope link  src yyy.yyy.yyy.yyy

to create a 'corrected' appropriately unique 'default' entry in the RPDB tables

RPDB table ovpnc1
default via xxx.xxx.xxx.xxx dev tun11

RPDB table ovpnc5
default via yyy.yyy.yyy.yyy dev tun15

 

[Pete]

The session hangs after executing the line " ip route flush table $TABLE >2 >dev>null " for the second time. Router needs to be restarted to start ssh connection

 

[Martineau]

The session hangs after executing the line " ip route flush table $TABLE >2 >dev>null " for the second time. Router needs to be restarted to start ssh connection

Well I guess you'll have to debug the script to find out what the value of $TABLE is the second time around!

EDIT: Updated original script to provide debug messages.

 

[Pete]

Looks better, but did not solve the issue.

admin@RT-AC68U-17F8:/tmp/home/root# ./fixroute.sh


        Processing Routing table.... 111

10.244.0.1 dev ppp0  proto kernel  scope link
104.192.2.218 via 10.244.0.1 dev ppp0
192.168.68.0/24 dev br0  proto kernel  scope link  src 192.168.68.1
10.8.8.0/24 dev tun11  proto kernel  scope link  src 10.8.8.69
10.245.0.0/16 dev eth0  proto kernel  scope link  src 10.245.214.178
127.0.0.0/8 dev lo  scope link
0.0.0.0/1 via 10.8.8.1 dev tun11
128.0.0.0/1 via 10.8.8.1 dev tun11
default via 10.8.8.1 dev tun11
default via 10.245.0.1 dev eth0  metric 1


        Routing table ovpnc1

10.244.0.1 dev ppp0  proto kernel  scope link
104.192.2.218 via 10.244.0.1 dev ppp0
192.168.68.0/24 dev br0  proto kernel  scope link  src 192.168.68.1
10.8.8.0/24 dev tun11  proto kernel  scope link  src 10.8.8.69
10.245.0.0/16 dev eth0  proto kernel  scope link  src 10.245.214.178
127.0.0.0/8 dev lo  scope link
0.0.0.0/1 via 10.8.8.1 dev tun11
128.0.0.0/1 via 10.8.8.1 dev tun11
default via 10.8.8.69 dev tun11
default via 10.245.0.1 dev eth0  metric 1

        VPN Client 1 endpoint I/P is 104.192.2.218



        Processing Routing table.... 115

10.244.0.1 dev ppp0  proto kernel  scope link
104.192.2.218 via 10.244.0.1 dev ppp0
185.151.58.113 via 10.244.0.1 dev ppp0
192.168.68.0/24 dev br0  proto kernel  scope link  src 192.168.68.1
10.8.8.0/24 dev tun11  proto kernel  scope link  src 10.8.8.69
10.245.0.0/16 dev eth0  proto kernel  scope link  src 10.245.214.178
127.0.0.0/8 dev lo  scope link
0.0.0.0/1 via 10.8.8.1 dev tun11
128.0.0.0/1 via 10.8.8.1 dev tun11
default via 10.8.8.1 dev tun11
default via 10.245.0.1 dev eth0  metric 1


        Routing table ovpnc5

10.244.0.1 dev ppp0  proto kernel  scope link
104.192.2.218 via 10.244.0.1 dev ppp0
185.151.58.113 via 10.244.0.1 dev ppp0
192.168.68.0/24 dev br0  proto kernel  scope link  src 192.168.68.1
10.8.8.0/24 dev tun11  proto kernel  scope link  src 10.8.8.69
10.245.0.0/16 dev eth0  proto kernel  scope link  src 10.245.214.178
127.0.0.0/8 dev lo  scope link
0.0.0.0/1 via 10.8.8.1 dev tun11
128.0.0.0/1 via 10.8.8.1 dev tun11
default via 10.8.8.237 dev tun15
default via 10.245.0.1 dev eth0  metric 1

        VPN Client 5 endpoint I/P is 185.151.58.113



        Processing Routing table.... 18.

Error: argument "18." is wrong: table id value is invalid

Error: argument "18." is wrong: "table" value is invalid

Error: an inet address is expected rather than "tun11".


        Routing table ovpnc.

Error: argument "18." is wrong: table id value is invalid


        VPN Client . endpoint I/P is



        Processing Routing table.... 18.

Error: argument "18." is wrong: table id value is invalid

Error: argument "18." is wrong: "table" value is invalid

Error: an inet address is expected rather than "tun11".


        Routing table ovpnc.

Error: argument "18." is wrong: table id value is invalid


        VPN Client . endpoint I/P is

admin@RT-AC68U-17F8:/tmp/home/root#

ip route table list:

admin@RT-AC68U-17F8:/tmp/home/root# ip route

10.244.0.1 dev ppp0  proto kernel  scope link
185.151.58.113 via 10.244.0.1 dev ppp0
104.192.2.218 via 10.244.0.1 dev ppp0
192.168.68.0/24 dev br0  proto kernel  scope link  src 192.168.68.1
10.8.8.0/24 dev tun11  proto kernel  scope link  src 10.8.8.69
10.8.8.0/24 dev tun15  proto kernel  scope link  src 10.8.8.237
10.245.0.0/16 dev eth0  proto kernel  scope link  src 10.245.214.178
127.0.0.0/8 dev lo  scope link
0.0.0.0/1 via 10.8.8.1 dev tun11
128.0.0.0/1 via 10.8.8.1 dev tun11
default via 10.244.0.1 dev ppp0
default via 10.245.0.1 dev eth0  metric 1

Does the lines below needs to be addressed as well?

10.8.8.0/24 dev tun11  proto kernel  scope link  src 10.8.8.69
10.8.8.0/24 dev tun15  proto kernel  scope link  src 10.8.8.237

Another FYI:

admin@RT-AC68U-17F8:/tmp/home/root# ip route | grep tun1 | sort -k 2.1
10.8.8.0/24 dev tun11  proto kernel  scope link  src 10.8.8.69
10.8.8.0/24 dev tun15  proto kernel  scope link  src 10.8.8.237
0.0.0.0/1 via 10.8.8.1 dev tun11
128.0.0.0/1 via 10.8.8.1 dev tun11

 

[Pete]

After some tweaking, my ip route looks like this. I believe it is working as expected.

admin@RT-AC68U-17F8:/tmp/home/root# ip route show table 115
10.244.0.1 dev ppp0  proto kernel  scope link
185.151.58.113 via 10.244.0.1 dev ppp0
192.168.68.0/24 dev br0  proto kernel  scope link  src 192.168.68.1
10.8.8.0/24 dev tun15  proto kernel  scope link  src 10.8.8.237
10.245.0.0/16 dev eth0  proto kernel  scope link  src 10.245.214.178
127.0.0.0/8 dev lo  scope link
0.0.0.0/1 via 10.8.8.1 dev tun15
128.0.0.0/1 via 10.8.8.1 dev tun15
default via 10.8.8.237 dev tun15

admin@RT-AC68U-17F8:/tmp/home/root# ip route show table 111
10.244.0.1 dev ppp0  proto kernel  scope link
104.192.2.218 via 10.244.0.1 dev ppp0
192.168.68.0/24 dev br0  proto kernel  scope link  src 192.168.68.1
10.8.8.0/24 dev tun11  proto kernel  scope link  src 10.8.8.69
10.245.0.0/16 dev eth0  proto kernel  scope link  src 10.245.214.178
127.0.0.0/8 dev lo  scope link
0.0.0.0/1 via 10.8.8.1 dev tun11
128.0.0.0/1 via 10.8.8.1 dev tun11
default via 10.8.8.69 dev tun11
default via 10.245.0.1 dev eth0  metric 1
admin@RT-AC68U-17F8:/tmp/home/root#

 

[Martineau]

After some tweaking, my ip route looks like this. I believe it is working as expected.

ip route show table 115
default via 10.8.8.237 dev tun15

ip route show table 111
default via 10.8.8.69 dev tun11
default via 10.245.0.1 dev eth0  metric 1    <<- ???

Usually the tables should not have multiple (possibly conflicting) 'default' statements.

Also for most users, only the single 'default' statement needs to be defined in the table.

 

[Martineau]

Another FYI:

admin@RT-AC68U-17F8:/tmp/home/root# ip route | grep tun1 | sort -k 2.1
10.8.8.0/24 dev tun11  proto kernel  scope link  src 10.8.8.69
10.8.8.0/24 dev tun15  proto kernel  scope link  src 10.8.8.237
0.0.0.0/1 via 10.8.8.1 dev tun11
128.0.0.0/1 via 10.8.8.1 dev tun11

Table main should not contain the following

0.0.0.0/1 via 10.8.8.1 dev tun11
128.0.0.0/1 via 10.8.8.1 dev tun11

as vpnrouting.sh should strip them. These two statements override the 'default' statement, i.e. table main should route traffic via the 'eth0' device

default via 10.245.0.1 dev eth0

and not the 'tun11' device if Selective routing is required.

I suspect you may have a conflicting custom statement that is the root cause of your issues....
i.e. can't recall if 'topology subnet' is one that breaks the Selective Routing etc.

EDIT: Updated script to remove the inappropriate '0.0.0.0/1 and 128.0.0.0/1' routes from table main, before listing the active tun1X devices. Hopefully this was the cause of the spurious attempt to process table '18.' ...twice!!!

 

[RMerlin]

OpenVPN 2.4 gives us access to the new pull-filter parameter for clients. Have you thought about trying something like this on your clients?

pull-filter ignore route 0.0.0.0/1
pull-filter ignore route 128.0.0.0/1

Another potential way to deal with it is by rejecting/ignoring the "redirect-gateway def1", which is what defines these two routes.

These are just ideas based on the OpenVPN manual, I never tested either of them. Might be a way to prevent the server from pushing those "route-everything" rules, it will depend on how the parameters are pushed to you by the remote server.

 

[RMerlin]

I just tested with pull-filter ignore "redirect-gateway def1" and it does prevent the client from adding the 0.0.0.0/1 and 128.0.0.0/1 routes.

 

[RMerlin]

/usr/bin/vpnrouting.sh is incorrectly (or by design) cloning the table 'main' routes based on these two conflicting routes:

Would it actually make more sense to only clone routes that are specific to the tunnel interface? I.e, in vpnrouting.sh:

init_table(){
        logger -t "openvpn-routing" "Creating VPN routing table"
        ip route flush table $VPN_TBL

# Fill it with copy of existing main table
        ip route show table main dev $dev | while read ROUTE
        do
                ip route add table $VPN_TBL $ROUTE dev $dev
        done
}

This might however mean that any user-defined static route would be missing from that client's table.