Splitting Wireguard between Router and client

[archiel]

Hi @ZebMcKayhan Almost a year on from my initial look at the Flow Cache restrictions with using WireGuard (WGM) see https://www.snbforums.com/threads/session-manager-4th-thread.81187/page-8#post-833178 I finally have a fast enough ISP for it to matter. With the latest rMerlin it looks like just running the WGM server component is fine whether Flow Control is enabled or disabled, however trying to use the client component still generates the Router kernel: [0;33;41m[ERROR mcast] bcm_mcast_blog_process,819: blog allocation failure[0m errors if I have Flow Control enabled and while I can flush these using scribe, within an hour or so the connection speed drops back as if FC was disabled (fc status shows enabled) and I can only get back to full speed after a reboot.

What I can do is run WGM just as server on the router and then run a Wireguard client to my VPN provider directly on the Ubuntu client VM (same one as originally designated as a client on WGM). This allows the non-WireGuard clients to function at full speed.

On the original setup, I also routed Unbound through the VPN (on WGM) so as to mask my local IP addresses. In addition I have been using passthru for some of the server clients (phone, laptop) and what I would like to know is whether I can continue to do this now that WireGuard is split and if so, where do I start? Do you have any time to help / any suggestions as to what to do next?

n. b. I know I can resolve the local IP as DNS issue by moving away from unbound, but if there is a way of keeping it, I would prefer this.

 

[ZebMcKayhan]

With the latest rMerlin it looks like just running the WGM server component is fine whether Flow Control is enabled or disabled, however trying to use the client component still generates the Router kernel: [0;33;41m[ERROR mcast] bcm_mcast_blog_process,819: blog allocation failure[0m errors if I have Flow Control enabled

Have you tried the built in fw client?? Are you sure it's not ipv6 yet?
On a side note, the fc bypass in newer routers only seems to limit upload speed, while keeping your full download speed. But this may not apply to your AX88 though.

what I would like to know is whether I can continue to do this now that WireGuard is split

Sure, I don't see why not... you may need some firewall rules though.

where do I start?

This could be done in so many ways though... to ease the burden of scripting too much I'm thinking something like setting up a dummy client in wgm with bogus data, then hijack that policy route table and point the default route to be to your Ubuntu machine instead. Then you could still use wgm for policy rules. Even passthrough rules if you wish... and unbound of course.
Not sure if wgm would give us trouble with the bogus client though.

Do you have any time to help

Relatively... I usually have some minutes on the evening, not always though. But I'll try with what I can.

any suggestions as to what to do next?

Start with getting the Ubuntu client up and running. Then setup a dummy client in wgm. Start testing with ipv4 only and remove wgm default route in the policy table and replace with a route via the Ubunty client.
Get some client you could use for testing and start to setup a wgm rule for that ip to use the bogus client.

n. b. I know I can resolve the local IP as DNS issue by moving away from unbound, but if there is a way of keeping it, I would prefer this.

I don't see any reason for why Unbound should be any different, it should be able to use your Ubuntu wg connection. But if the Ubunty client borks there may not be any fallbacks.

 

[archiel]

Hi @ZebMcKayhan

Starting at the top

1. I have not tried the built in client, I could but as it is IPv4 only I can not see how it will help other than to indicate that it is something in the WGM implementation. Could try this to see what happens - will have a look for how to set this up.
2. I completely removed IPv6 during my testing last year, and it had no effect at all. The bcm_mcast_blog_process errors and the eventual speed drop occurred regardless. so I had concluded that this was not the issue.
3. in regard to the effect of fc disable, it is definitely on both sides on this router - the line is a notional 1 Gb up/down, in practice around 930/930 with fc enabled and dropping to around 420/550 when disabled.
4. The ubuntu client is up and running, though using the VPN provider's standard install, so both IP addresses and DNS servers are different to those in the WGM client - even though it is pointed at the same node.
5. In respect of the dummy client, as I already have have a fully functioning client under WGM, including the routing to unbound, passthrough and ipset linking, could it work to 'break' this so that it no longer works (e.g. change the public / private keys, point at a wrong address, or right address, wrong port) - if so what would you suggest I try first.

Thanks,

Archie

 

[ZebMcKayhan]

in regard to the effect of fc disable, it is definitely on both sides on this router - the line is a notional 1 Gb up/down, in practice around 930/930 with fc enabled and dropping to around 420/550 when disabled.

I know, but on later models (not sure which) a different kind of bypass is added where wg client IP gets added to

/proc/blog/skip_wireguard_network

And fc is kept enabled. Users have found that it mainly limits upload and not so much download. I don't have speeds to test myself.

The ubuntu client is up and running, though using the VPN provider's standard install, so both IP addresses and DNS servers are different to those in the WGM client - even though it is pointed at the same node.

Ubuntu client dns would not matter unless it somehow intercepts or prevents forwarding of dns requests. You have ip forwarding turned on and firewall setup on the Ubuntu client, right?
You will also need to masquerade forwarded packets in the firewall nat table.

In respect of the dummy client, as I already have have a fully functioning client under WGM, including the routing to unbound, passthrough and ipset linking, could it work to 'break' this so that it no longer works (e.g. change the public / private keys, point at a wrong address, or right address, wrong port) - if so what would you suggest I try first.

I'm thinking you could create a new one and as things started working you could migrate bit by bit to not cause any downtime?

if so what would you suggest I try first.

See if wgm accept your dummy client and try to replace the default routes in policy route table.

 

[archiel]

I know, but on later models (not sure which) a different kind of bypass is added where wg client IP gets added to

/proc/blog/skip_wireguard_network

And fc is kept enabled. Users have found that it mainly limits upload and not so much download. I don't have speeds to test myself.



Ubuntu client dns would not matter unless it somehow intercepts or prevents forwarding of dns requests. You have ip forwarding turned on and firewall setup on the Ubuntu client, right?
You will also need to masquerade forwarded packets in the firewall nat table.




I'm thinking you could create a new one and as things started working you could migrate bit by bit to not cause any downtime?



See if wgm accept your dummy client and try to replace the default routes in policy route table.

This probably in the wrong order, but

I have the Wireguard 'client' running on my Ubuntu VM and the WGM 'server' running on the router with fc enabled. Downtime is not really an issue for now as while I have lost passthru, when I leave home and want to use the server can just stop the client on the ubuntu VM, disable fc and start the WGM client.

I modified the WGM client (wg13) Endpoint so traffic is sent to an incorrect port, and it starts normally (tested as start wg13 and start wg13 debug) and if I run List I can see

transfer: 0 B received, 57.09 KiB sent         0 Days, 00:35:51 since Tue Jan 9 18:07:59 2024 >>>>>>

As expected there are no bcm_mcast_blog_process errors (no traffic) and so the connection speed does not drop. I did need to stop unbound from trying to run its lookups over the dummy VPN (again as expected).

So now to the bits I really don't understand - how to route the unbound and passthru traffic to the WireGuard client running on the ubuntu VM.

1. When you say 'try to replace the default routes in policy route table' what am I looking for and how do I replace?

The ubuntu VM is running in desktop mode, so the firewall (ufw) is installed but not active, equally ip forwarding is disabled (net.ipv4.ip_forward = 0). The VM is sitting behind the router firewall and on the same subnet (10.50.60.xxx) as the other home devices, so I had not thought I needed to enable these.

2. From you comments above, I assume I now need to do this, is this correct?

3. Once enabled, how do I go about 'You will also need to masquerade forwarded packets in the firewall nat table'? I am assuming that here you are referring to the nat table on the VM once the firewall is enabled.

 

[ZebMcKayhan]

When you say 'try to replace the default routes in policy route table' what am I looking for and how do I replace?

It's been so long that I can't remember which route table number wgm used. If you create a rule in wgm for some ip to use your dummy client, you can check which route table your dummy client uses by

ip rule

Locate your rule ip and see which number. It should say from <rule ip> lookup xyz

When you know what xyz is you can look how the route table looks:

ip route show table xyz

In there it should be a route called default via... or it could be 127.0.0.0/1 and 0.0.0.0/1
These are the ones you need to supersede.

The ubuntu VM is running in desktop mode, so the firewall (ufw) is installed but not active, equally ip forwarding is disabled (net.ipv4.ip_forward = 0). The VM is sitting behind the router firewall and on the same subnet (10.50.60.xxx) as the other home devices, so I had not thought I needed to enable these.

Now that you wish to have this machine recieve lan packets and forward to its wireguard client ip forwarding needs to be turned on. Otherwise it won't forward anything.

As you now open a tunnel out of your network (and also in) it may be a good idea to enable the firewall so you don't leave your lan open. Your router firewall will be bypassed in the tunnel.

From you comments above, I assume I now need to do this, is this correct?

Yep.

Once enabled, how do I go about 'You will also need to masquerade forwarded packets in the firewall nat table'? I am assuming that here you are referring to the nat table on the VM once the firewall is enabled.

Yes, you need to add a MASQUERADE rule to the POSTROUTING nat table so all forwarded packets gets the source address rewritten so it appears to come from the ubuntu machine itself. Something like:

iptables -t nat -I POSTROUTING ! -s <wg ipv4> -o <wg iface> -j MASQUERADE

As you enable the firewall you will also need to add accept -i <lan if> to -o <wg if> in FORWARD filter table and -i <wg if> -o <lan if> state RELATED, ESTABLISHED in the other way. And drop everything else. There are several tutorials on this online but I don't even know what firewall your machine is using. I only know iptables.

 

[ZebMcKayhan]

When I setup my cloud server I did something similar. I didn't bother with RELATED, ESTABLISHED since it was only wg to wg trusted data but you may find some value there: https://github.com/ZebMcKayhan/WireguardManager?tab=readme-ov-file#setup-private-server-via-cloud-server Scroll down to setup cloud server part[/URL].

For internet out cloud server there is a similar rule set as you need.

 

[archiel]

When I start wg13 and run <ip rule> it shows the following extra lines

9931:   from 192.168.3.1 lookup 123
9993:   from all fwmark 0x4000/0x4000 lookup 123

and

admin@Router:/tmp/home/root# ip route show table 123
0.0.0.0/1 dev wg13  scope link
10.50.60.0/24 dev br0  proto kernel  scope link  src 10.50.60.1
128.0.0.0/1 dev wg13  scope link

So my next questions are

what do I need to supercede,
with what,
and how

In the meantime I will setup the firewall and ip-forwarding, presumably for both ipv4 and ipv6, just to check that whether it effects anything else on the LAN connected to the VM. I am not around tomorrow, so I will let you know how I get on on Thursday. Many thanks Archie

ADDED

I just found the following:

ufw is a front-end for netfilter/iptables, the Linux mechanism for routing and filtering internet traffic.

ufw is completely optional and it's possible to create firewall and routing tables directly using the iptables commands. Some people prefer the syntax of ufw, which is supposed to make it a bit easier.

So iptables is installed on the VM and UFW is available as tool for configuring, or it can be done directly.

 

[ZebMcKayhan]

So my next questions are

what do I need to supercede,
with what,
and how

I guess the most straight forward way is to continue to supersede using even more specific routes instead of attempting to delete the existing routes, wgm already does this to supersede the default /0 route with two /1 routes. for /2 we would need to add 4 routes:

0.0.0.0/2
64.0.0.0/2
128.0.0.0/2
192.0.0.0/2

if I remember the syntax correct it would be something like:

ip route add 0.0.0.0/2 via <ubuntu machine ipv4> dev br0 table 123
ip route add 64.0.0.0/2 via <ubuntu machine ipv4> dev br0 table 123
ip route add 128.0.0.0/2 via <ubuntu machine ipv4> dev br0 table 123
ip route add 192.0.0.0/2 via <ubuntu machine ipv4> dev br0 table 123

try them out from the ssh prompt so there are no error messages and the routes appear ok in the route table before putting them into some auto-script.

now, this would mean any ip set to use this routing table would send all internet data to the ubuntu machine instead of to your WAN. then it would be up to the ubuntu machine to forward it to the wireguard tunnel and naturally relay replies back to the router for further routing back to the interface where is came from.

 

[archiel]

A slight detour:

While I am am still investigating spitting WireGuard, the starting point for this was that I wish to have IPv6 on the WG clients, which meant I cannot use the built in WireGuard.
However the WGM implementation does not support FC bypass on my current router (RT-AX88U), at least I have not found a way to get this to work. So what if I upgrade my router to a RT-AX88U Pro (I could do with 2.5 GB port on LAN and WAN, and my RT-AX88U would replace the RT-AX58U as the mesh node))

Does anyone know if WGM supports FC bypass on the RT-AX88U Pro?

 

[ZebMcKayhan]

Does anyone know if WGM supports FC bypass on the RT-AX88U Pro?

The methode is supported since here: https://www.snbforums.com/threads/session-manager-4th-thread.81187/post-830683

But don't know how well it have been tested.

 

[archiel]

The methode is supported since here: https://www.snbforums.com/threads/session-manager-4th-thread.81187/post-830683

But don't know how well it have been tested.

Went back to this and noted what @Martineau said at the time

So when starting a 'client' Peer in policy mode, it will populate the 'blog' files (if they are available within the firmware e.g. v388.2b1? and the model is supported) and will intelligently remove the entries if deemed appropriate.

When I first start WGM, flow cache is disabled. I then enable and initially any WG clients are speed limited while the other lan clients run at full speed. At the same time there are no

Router kernel: [0;33;41m[ERROR mcast] bcm_mcast_blog_process,819: blog allocation failure

errors. Which I assume is what is intended.

However after an interval (rarely more than 20 minutes) the blog errors commence and the speed on all the clients drops.

Looking at my setup I see

Spoiler: WGM about Configuration

        Router RT-AX88U Firmware (v388.6_beta1)

        [✔] Entware Architecture arch=aarch64


        v4.19b5 WireGuard® Session Manager (Change Log: https://github.com/MartineauUK/wireguard/commits/dev/wg_manager.sh)
        MD5=3d482ec26756bc93a5c2c40905f26570 /jffs/addons/wireguard/wg_manager.sh

                v4.19.2 (wg_client)
                v4.17.1 (wg_server)

        [ℹ ] WireGuard® Kernel module/User Space Tools included in Firmware (1.0.20210124) but 3rd-Party modules installed...

        [✔] WireGuard® Module LOADED Thu Apr  6 15:28:06 IST 2023

        MD5=70a85a1bed5f6313add595e2a95423c4 wireguard-kernel_1.0.20220627-RT-AX88U_aarch64-3.10.ipk
        MD5=3c3fef331578bcd20714a148b96257f8 wireguard-tools_1.0.20210914-1_aarch64-3.10.ipk

        [✔] WebUI Addon Enabled

        [✔] DNSmasq is listening on ALL WireGuard® interfaces 'wg*'

        [✔] firewall-start is monitoring WireGuard® Firewall rules

        [✖] WAN KILL-Switch is DISABLED (use 'vx' command for info)
        [✖] UDP monitor is DISABLED

        [✔] Flow Cache is ENABLED (WireGuard© VPN Bypass available)

        [✔] IPv6 Service is dhcp6
        [ℹ ] IPv6,2a02:6b67:xxxx:yyyy::1,v1.1,,,See http://ip6.me/docs/ for api documentation

        [✔] Reverse Path Filtering ENABLED

        [✔]Use 3rd-party Entware/Userspace Tools modules is ALLOWED

        [✔] Endpoint DDNS re-fresh monitor ACTIVE

        [✔] Cron schedule #WireGuard_DB# (0 7 * * 6) to trim older than 99 days from WireGuard® SQL Database ENABLED

        [✔] Use of 'Pg-Up' Key for command retrieval is ENABLED

        [✔] Statistics gathering is ENABLED

and looking at /proc/blog I can see

admin@Router:/tmp/home/root# ls -l /proc/blog
-r--r--r--    1 admin    root             0 Jan 14 21:27 skip_wireguard_network
-r--r--r--    1 admin    root             0 Jan 14 21:27 skip_wireguard_port

What I do not understand is what 'intelligently remove the entries if deemed appropriate.' is meant to mean (should the blog errors not be appearing?) and how to test what is failing. While I use scribe, so all my logs are split up, I cannot find any other unusual entries either shortly before or concurrent with the 'blog' errors kicking off.

 

[archiel]

While I have managed to add the firewall (ufw) and ip forwarding (ipv4 and ipv6) to my ubuntu VM I am getting stuck on replicating its connectivity to the rest of the LAN devices (10.50.60.xxx). While I could add particular services by allowing the their ports, I cannot see how to allow network access from my other machines.

For example on VM Ubu1 (10.50.60.150) folder /data/Store is shared as Store (with Guest access). As such from other windows devices I can access it as \\10.50.60.150\Store or \\Ubu1\Store. However once the firewall is up, the Windows machines can no longer connect, though I can still ping on 10.50.60.150 or Ubu1.

What am I missing?

Update: just needed to specify the IPv4 addresses of the windows devices that needed access (or the subnet).

 

[ZebMcKayhan]

What I do not understand is what 'intelligently remove the entries if deemed appropriate.' is meant to mean (should the blog errors not be appearing?) and how to test what is failing.

My interpretation is that added rules are removed when stopped unless explicitly needed by other peers.

Even if your system have the files I'm still not sure if the are implemented/used by fw properly. @RMerlin have previously stated ax88u does not have blog skip network implementation. Don't know if anything have changed.

, I cannot find any other unusual entries either shortly before or concurrent with the 'blog' errors kicking off.

As far as I have understand Flow-cache is all about flow-learning which I guess is a method to determine active flows to accelerate. It may take ~20min for it to decide. This is just guesswork, most people with ax88u don't seem to have these issues.

Update: just needed to specify the IPv4 addresses of the windows devices that needed access (or the subnet).

so, did you solve it? As you are running as container you may need to tweak something to forward all data to container. Note, the container ip differs from your lan ip.

 

[archiel]

so, did you solve it? As you are running as container you may need to tweak something to forward all data to container. Note, the container ip differs from your lan ip.

Puzzled by this, the VM IPs as seen on the LAN or the Router are the same as the VM IPs seen when running ifconfig from the VM itself.

The next stage is to try adding the routes from the ssh prompt when WGM client is up, and if no errors add them to the wg-xx-up.sh script. Presumably I add the equivalent IPv6 entries.

 

[archiel]

Adding the ipv4 as you advised worked with no error messages

ip route add 0.0.0.0/2 via <ubuntu machine ipv4> dev br0 table 123
ip route add 64.0.0.0/2 via <ubuntu machine ipv4> dev br0 table 123
ip route add 128.0.0.0/2 via <ubuntu machine ipv4> dev br0 table 123
ip route add 192.0.0.0/2 via <ubuntu machine ipv4> dev br0 table 123

and ip route show table 123 now shows

admin@Router:/tmp/home/root# ip route show table 123
0.0.0.0/2 via 10.50.60.150 dev br0
0.0.0.0/1 dev wg13  scope link
10.50.60.0/24 dev br0  proto kernel  scope link  src 10.50.60.1
64.0.0.0/2 via 10.50.60.150 dev br0
128.0.0.0/2 via 10.50.60.150 dev br0
128.0.0.0/1 dev wg13  scope link
192.0.0.0/2 via 10.50.60.150 dev br0

Is this what I should be expecting?

Two more questions

First

Any thoughts on how I deal with IPv6. given the changeable nature of the IPv6 address on the VM?
Current ip -6 route show table 123

2a02:6b67:e9c1:xxxx::/56 dev br0  proto kernel  metric 256  pref medium
::/1 dev wg13  metric 1024  pref medium
fe80::/64 dev br0  proto kernel  metric 256  pref medium
8000::/1 dev wg13  metric 1024  pref medium

And

Why do you think the container (VM) IP addresses should be different from the lan IPs? For all effective purposes the the VM is just another device on the same subnet as the windows machines, phones, printers, etc.

Thanks, Archie

 

[ZebMcKayhan]

Is this what I should be expecting?

Yep, looks good!

Any thoughts on how I deal with IPv6. given the changeable nature of the IPv6 address on the VM?

Uuh, not really... perhaps a script to get the prefix from nvram? And I think you can use it's slaac suffix (the one derived from mac). Or better yet, use the link-local address.

Why do you think the container (VM) IP addresses should be different from the lan IPs? For all effective purposes the the VM is just another device on the same subnet as the windows machines, phones, printers, etc.

I've used portainer on OMV and I think the container got its own ip, but I could be wrong.

 

[ZebMcKayhan]

Any thoughts on how I deal with IPv6. given the changeable nature of the IPv6 address on the VM?
Current ip -6 route show table 123

2a02:6b67:e9c1:xxxx::/56 dev br0 proto kernel metric 256 pref medium
::/1 dev wg13 metric 1024 pref medium
fe80::/64 dev br0 proto kernel metric 256 pref medium
8000::/1 dev wg13 metric 1024 pref medium

For ipv6 it would be

ip -6 route add 0000::/2 via <ubuntu link-local> dev br0 table 123
Ip -6 route add 4000::/2 via <ubuntu link-local> dev br0 table 123
ip -6 route add 8000::/2 via <ubuntu link-local> dev br0 table 123
ip -6 route add C000::/2 via <ubuntu link-local> dev br0 table 123

You could still use mac adress ipset in wgm for client routing if you wish.

 

[archiel]

For ipv6 it would be

ip -6 route add 0000::/2 via <ubuntu link-local> dev br0
Ip -6 route add 4000::/2 via <ubuntu link-local> dev br0
ip -6 route add 8000::/2 via <ubuntu link-local> dev br0
ip -6 route add C000::/2 via <ubuntu link-local> dev br0

You could still use mac adress ipset in wgm for client routing if you wish.

I will try this - link local is much easier as it is fixed, SLAAC on the router is using privacy extensions, so the MAC address is not used for the global suffix(es).

Currently if I test my dummy client on the router, it seems to break the network connectivity on the VM. The possible culprits are the ipset (on the router), the firewall and the VM wg client - I will run through the variations to see where the issue is.

Going back to the routing on the VM

Yes, you need to add a MASQUERADE rule to the POSTROUTING nat table so all forwarded packets gets the source address rewritten so it appears to come from the ubuntu machine itself. Something like:

iptables -t nat -I POSTROUTING ! -s <wg ipv4> -o <wg iface> -j MASQUERADE

Looking at the output from ifconfig with the wg client up i have 3 interfaces eth0: (LAN), lo: (loopback) and azirevpn-nl-ams: (VPN) so for IPv4 and IPv6 I assume I should try

iptables -t nat -I POSTROUTING ! -s 10.0.65.xx -o azirevpn-nl-ams -j MASQUERADE
ip6tables -t nat -I POSTROUTING ! -s 2a0e:1c80:1337::xxxx -o azirevpn-nl-ams -j MASQUERADE

where 10.0.65.xx and 2a0e:1c80:1337::xxxx are the inet and inet6 addresses shown for the VPN.

And in regard to

As you enable the firewall you will also need to add accept -i <lan if> to -o <wg if> in FORWARD filter table and -i <wg if> -o <lan if> state RELATED, ESTABLISHED in the other way. And drop everything else. There are several tutorials on this online but I don't even know what firewall your machine is using. I only know iptables.

The VM is using iptables, and presumably <lan if> and <wg if> are eth0 and azirevpn-nl-ams respectively, but as I do not follow the rest I will have a look at the cloud example to see if that help.

 

[archiel]

Regarding

ip -6 route add 0000::/2 via <ubuntu link-local> dev br0
Ip -6 route add 4000::/2 via <ubuntu link-local> dev br0
ip -6 route add 8000::/2 via <ubuntu link-local> dev br0
ip -6 route add C000::/2 via <ubuntu link-local> dev br0

do I also need to add table 123 after br0?