questions about building wireless mesh LAN

[drush8]

Hello everyone! For some research purpose, I need to build up a enormous Manhattan wireless mesh local access network (no need to connect to Internet), in this LAN, every router communicate with each other through 5GHZ, devices join to the router through 2.4GHZ(like the figure attached shows, all lines mean connection wirelessly, not physical wire). And I need to have the ability to design the specific route for different devices by myself. Thus some modes like Al mesh or repeater should not work. In order to figure out how to build this LAN, I did a small experiment, a triangle-shaped structure. There are three wireless AC86U router in Router mode link to each other through 5GHZ with the help of WDS setting. For simplicity I call them router50, 60, and 70 (router50 control the subnet of 192.168.50.0/24, so does router60 and 70). I want to realize such a scenario: deviceA connect to router 60 wirelessly, has ip address manually set to 192.168.60.66, deviceB connect to router 70 wirelessly, has ip address manually set to 192.168.70.77. The two device should communicated to each other through the route: A--60--50--70--B. Now here is the problem:

After I set up the routing table for three routers successfully, It is fine that A and B can ping to each other. But when I use iperf3 or udp transmission python script, it fails. Looks like basic icmp communication is well but packets of application layer meet some problem. What interesting thing is that after I close the stp for the virtual bridge br0 of router60, A and B then can communicate with each other through iperf3 tcp transmission well! But the iperf3 udp transmission still fail. Looks like STP and WDS may lead to some unexplainable problems?

Here is the detail:
all routers are AC86U ASUS routers. All have the 386.11 merlin firmware. All open the WDS for 5Ghz and are in hybrid mode (means both AP and WDS enabled), and have add each other mac addressed in to the list. For 5GhZ, all router SSID is set the same and encryption/security method is off. For 2.4G three routers still have the different SSID and encryption is on which means if the device want to connect to that router it has to enter the password. I have find some threads showing that developers have not recommend to use WDS anymore because of the security reason, but now I have no choice but WDS can realize my mesh structure. All of them close the firewall. All of them charge different subnets(router60 charges 192.168.60.0/24...).

The routing rule adding commends look like this:
for router50: (through command line)
ip route add 192.168.60.1 dev br0
ip route add 192.168.70.1 dev br0
ip route add 192.168.60.66 via 192.168.60.1 dev br0
ip route add 192.168.70.77 via 192.168.70.1 dev br0

for router70: (through command line)
ip route add 192.168.50.1 dev br0
ip route add 192.168.60.66 via 192.168.50.1 dev br0

for router60: (through command line)
ip route add 192.168.50.1 dev br0
ip route add 192.168.70.77 via 192.168.50.1 dev br0

If stp of br0 in all three routers closed, then all three routers will be slow to death, ssh will get stuck...


I am not sure if I am on the right track to build the structure, and how to solve the problem that ping works but udp transmission doesn't.

 

[drinkingbird]

Hello everyone! For some research purpose, I need to build up a enormous Manhattan wireless mesh local access network (no need to connect to Internet), in this LAN, every router communicate with each other through 5GHZ, devices join to the router through 2.4GHZ(like the figure attached shows, all lines mean connection wirelessly, not physical wire). And I need to have the ability to design the specific route for different devices by myself. Thus some modes like Al mesh or repeater should not work. In order to figure out how to build this LAN, I did a small experiment, a triangle-shaped structure. There are three wireless AC86U router in Router mode link to each other through 5GHZ with the help of WDS setting. For simplicity I call them router50, 60, and 70 (router50 control the subnet of 192.168.50.0/24, so does router60 and 70). I want to realize such a scenario: deviceA connect to router 60 wirelessly, has ip address manually set to 192.168.60.66, deviceB connect to router 70 wirelessly, has ip address manually set to 192.168.70.77. The two device should communicated to each other through the route: A--60--50--70--B. Now here is the problem:

After I set up the routing table for three routers successfully, It is fine that A and B can ping to each other. But when I use iperf3 or udp transmission python script, it fails. Looks like basic icmp communication is well but packets of application layer meet some problem. What interesting thing is that after I close the stp for the virtual bridge br0 of router60, A and B then can communicate with each other through iperf3 tcp transmission well! But the iperf3 udp transmission still fail. Looks like STP and WDS may lead to some unexplainable problems?

Here is the detail:
all routers are AC86U ASUS routers. All have the 386.11 merlin firmware. All open the WDS for 5Ghz and are in hybrid mode (means both AP and WDS enabled), and have add each other mac addressed in to the list. For 5GhZ, all router SSID is set the same and encryption/security method is off. For 2.4G three routers still have the different SSID and encryption is on which means if the device want to connect to that router it has to enter the password. I have find some threads showing that developers have not recommend to use WDS anymore because of the security reason, but now I have no choice but WDS can realize my mesh structure. All of them close the firewall. All of them charge different subnets(router60 charges 192.168.60.0/24...).

The routing rule adding commends look like this:
for router50: (through command line)
ip route add 192.168.60.1 dev br0
ip route add 192.168.70.1 dev br0
ip route add 192.168.60.66 via 192.168.60.1 dev br0
ip route add 192.168.70.77 via 192.168.70.1 dev br0

for router70: (through command line)
ip route add 192.168.50.1 dev br0
ip route add 192.168.60.66 via 192.168.50.1 dev br0

for router60: (through command line)
ip route add 192.168.50.1 dev br0
ip route add 192.168.70.77 via 192.168.50.1 dev br0

If stp of br0 in all three routers closed, then all three routers will be slow to death, ssh will get stuck...


I am not sure if I am on the right track to build the structure, and how to solve the problem that ping works but udp transmission doesn't.

You need professional (not Asus) equipment, a professional site survey, and a network architect. FYI that much 2.4ghz in Manhattan will be virtually useless, the band is pretty much fully saturated anywhere you go.

WDS is old technology (some manufacturers have implemented their own custom version that is more up to date), and if you want that many nodes all daisy chained, even 5ghz backhaul is going to be a problem.

Why are you running them in router mode and adding static routes?

If wired backhaul is not an option get APs with 6ghz for backhaul and use 5ghz for clients. Still going to be a challenge and require a proper site survey and pro grade equipment.

Iperf or any bidirectional initiated traffic is failing because NAT is likely enabled in router mode, you'd have to shut off NAT or change to AP mode.

 

[degrub]

this sounds like a theoretical question rather than an actual implementation question. Is that correct ?

 

[drush8]

You need professional (not Asus) equipment, a professional site survey, and a network architect. FYI that much 2.4ghz in Manhattan will be virtually useless, the band is pretty much fully saturated anywhere you go.

WDS is old technology (some manufacturers have implemented their own custom version that is more up to date), and if you want that many nodes all daisy chained, even 5ghz backhaul is going to be a problem.

Why are you running them in router mode and adding static routes?

If wired backhaul is not an option get APs with 6ghz for backhaul and use 5ghz for clients. Still going to be a challenge and require a proper site survey and pro grade equipment.

Iperf or any bidirectional initiated traffic is failing because NAT is likely enabled in router mode, you'd have to shut off NAT or change to AP mode.

Hi drinkingbird,
Sorry for late reply. We want to add static routes because we want to test and do research on the routing algorithm for such a mesh network. Instead for router mode we have closed all possible things: firewall and the NAT. We also use tcpdump to monitor the traffic, we find most packages have been transmitted successfully, while some package (especially the ack package of TCP) seems doesn't reach. There are many wired scenario, Also sometimes the package monitored by the middle router have TTL all equal to 1. We now solve the 3 router scenario interestingly by closing STP and enabling Jumble frame. But I think it is not the root way to solve it.

And we use Asus because we only have this in lab now XD

 

[drush8]

this sounds like a theoretical question rather than an actual implementation question. Is that correct ?

Hummm, I think it is because the different implementation of WDS for different manufacturers. Maybe it is more like a practical problem relating to the kinds of the router I think?

 

[drinkingbird]

Hi drinkingbird,
Sorry for late reply. We want to add static routes because we want to test and do research on the routing algorithm for such a mesh network. Instead for router mode we have closed all possible things: firewall and the NAT. We also use tcpdump to monitor the traffic, we find most packages have been transmitted successfully, while some package (especially the ack package of TCP) seems doesn't reach. There are many wired scenario, Also sometimes the package monitored by the middle router have TTL all equal to 1. We now solve the 3 router scenario interestingly by closing STP and enabling Jumble frame. But I think it is not the root way to solve it.

And we use Asus because we only have this in lab now XD

STP is probably doing what it is supposed to be doing. You have several bridge loops in your diagram so the switch is blocking them. By disabling it, some frames will now be able to pass that couldn't before but your network is going to be totally saturated with the bridge loop traffic and the switches are going to be overloaded.

You likely need to remove some of those links to make this work correctly. Even ignoring spanning tree/bridge loops the routing is not going to work right as you'll have asymmetry and conflicts.

Basically you need to carefully plan out what connects to LAN and what connects to WAN, and in a mesh that big, likely use scripts to separate your LAN ports into different VLANs and subnets. If two devices have two layer 2 paths between each other (even if via a 3rd device) it will not function correctly.

 

[drush8]

STP is probably doing what it is supposed to be doing. You have several bridge loops in your diagram so the switch is blocking them. By disabling it, some frames will now be able to pass that couldn't before but your network is going to be totally saturated with the bridge loop traffic and the switches are going to be overloaded.

You likely need to remove some of those links to make this work correctly. Even ignoring spanning tree/bridge loops the routing is not going to work right as you'll have asymmetry and conflicts.

Basically you need to carefully plan out what connects to LAN and what connects to WAN, and in a mesh that big, likely use scripts to separate your LAN ports into different VLANs and subnets. If two devices have two layer 2 paths between each other (even if via a 3rd device) it will not function correctly.

Instead we doesn't consider the WAN connection. We build this local access network in farm and not consider this structure connecting to Internet. But from my perspective the STP only relates to local bridge configure, Why can that influence the inter-router communication? It that because WDS?

 

[drinkingbird]

Instead we doesn't consider the WAN connection. We build this local access network in farm and not consider this structure connecting to Internet. But from my perspective the STP only relates to local bridge configure, Why can that influence the inter-router communication? It that because WDS?

These asus devices are essentially layer 3 switches. All of the LAN ports are just a switch with no routing features associated with them (until they pass to the CPU and LAN port).

When you have two links between the same two switches (even if it passes through other switches) you create a spanning tree (bridge) loop. That's what STP (Spanning Tree Protocol/Protection) is there for, in case you accidentally create a bridge loop, it blocks one of the paths.

The only way to prevent bridge loops is to have separate routing/broadcast domains, which on the asus in the default config, there are only two, the LAN (all ports) and the WAN.

So as long as you're connecting LAN to WAN to LAN to WAN etc you're fine, but if you have 3 routers all connected using the LAN you have a bridge loop. Your diagram above definitely shows a whole bunch of bridge loops, which would cause STP to kick in and block some paths and with STP disabled would cause the switches to perform very poorly, if at all.

Regardless of all that, static routes aren't going to do anything for you if all you're doing is connecting the LAN ports together. There is no router in the path, thus nothing for the static routes to point to. Even if you give them all different IP subnets, you need a router to send traffic between those subnets, and you'll still have a bridge loop since that happens at layer 2, below/before IP addresses are even involved.

If you really want to set up the mesh like your diagram you're going to need to get into scripting on the asus or use some tricks with the guest wireless VLANs that get built in order to make it work. Basically you have to take the switch ports out of the same broadcast domain and make them routed ports, which is not possible via the GUI.

 

[drush8]

These asus devices are essentially layer 3 switches. All of the LAN ports are just a switch with no routing features associated with them (until they pass to the CPU and LAN port).

When you have two links between the same two switches (even if it passes through other switches) you create a spanning tree (bridge) loop. That's what STP (Spanning Tree Protocol/Protection) is there for, in case you accidentally create a bridge loop, it blocks one of the paths.

The only way to prevent bridge loops is to have separate routing/broadcast domains, which on the asus in the default config, there are only two, the LAN (all ports) and the WAN.

So as long as you're connecting LAN to WAN to LAN to WAN etc you're fine, but if you have 3 routers all connected using the LAN you have a bridge loop. Your diagram above definitely shows a whole bunch of bridge loops, which would cause STP to kick in and block some paths and with STP disabled would cause the switches to perform very poorly, if at all.

Regardless of all that, static routes aren't going to do anything for you if all you're doing is connecting the LAN ports together. There is no router in the path, thus nothing for the static routes to point to. Even if you give them all different IP subnets, you need a router to send traffic between those subnets, and you'll still have a bridge loop since that happens at layer 2, below/before IP addresses are even involved.

If you really want to set up the mesh like your diagram you're going to need to get into scripting on the asus or use some tricks with the guest wireless VLANs that get built in order to make it work. Basically you have to take the switch ports out of the same broadcast domain and make them routed ports, which is not possible via the GUI.

Hi drinkingbird,

Thanks for quick reply. That makes some sense. We do configure every router to have the different subnet masks, . However we instead don't consider the switch ports, we want a complete wireless mesh network, thus the only "switch port" should base on the 5ghz wireless network card (maybe?, the asus ac86 has eth1-4 for switch ports, eth5 for 2.4ghz, eth6 for 5ghz), so, all packages go out through that single 5ghz wireless card and all packages comes in through that. Thus the way we try to make static route is not specifying the forwarding interface or port, but specifying the gateway, the next hop router's ip, to make sure that the package goes as we want. So, with wds, Asus routers may recognize each other and do some optimized topology for layer 2 package forwarding right(skip the ip/layer3 routing). Thus we need to find some way to force the router to forward the package based on ip address


Instead what I want is to realize the functionality similar to AI mesh mode, however I can define the route by myself through configuring the specific routers.

 

[drinkingbird]

Hi drinkingbird,

Thanks for quick reply. That makes some sense. We do configure every router to have the different subnet masks, . However we instead don't consider the switch ports, we want a complete wireless mesh network, thus the only "switch port" should base on the 5ghz wireless network card (maybe?, the asus ac86 has eth1-4 for switch ports, eth5 for 2.4ghz, eth6 for 5ghz), so, all packages go out through that single 5ghz wireless card and all packages comes in through that. Thus the way we try to make static route is not specifying the forwarding interface or port, but specifying the gateway, the next hop router's ip, to make sure that the package goes as we want. So, with wds, Asus routers may recognize each other and do some optimized topology for layer 2 package forwarding right(skip the ip/layer3 routing). Thus we need to find some way to force the router to forward the package based on ip address


Instead what I want is to realize the functionality similar to AI mesh mode, however I can define the route by myself through configuring the specific routers.

The wireless radios are on the same router interface as the physical switch ports (unless you use guest wireless 1 with access LAN disabled). You still have no router in the path. Think of the wireless radios as 2 more physical LAN ports on the back of the router, grouped in with the rest.

Putting two routers on two different subnets does not make a routable segment. You are essentially connecting a bunch of access points together and expecting routing to happen, it won't, access points are Layer 2 devices. They don't care about IPs.

The only way you might be able to get this to work is using Yazfi to give different subnets to different guest wireless, and use those various guest wireless to interconnect the APs. Yazfi converts each guest wireless interface into a routed interface. But you're still going to come up short on interfaces probably, unless you use 2.4ghz for some interconnects and 5ghz for others.

Even if you get all of that working, the overhead of daisy chaining (repeating essentially) the wifi multiple times is going to reduce your throughput to a likely unusable amount.

You need to go back and look at fundamentals of routing, switching, and wireless before trying to build some theoretical routed mesh network. The asus are likely not going to be the right devices to even start to attempt this on, and using wireless as your backhaul medium for a mesh this big is not going to yield good (if any) results.

To be clear, if you set the LAN of router1 to 192.168.1.1/24 and the LAN of router 2 to 192.168.2.1/24 they will not be able to communicate with each other as that traffic never passes through a routed interface.

 

[drush8]

The wireless radios are on the same router interface as the physical switch ports (unless you use guest wireless 1 with access LAN disabled). You still have no router in the path. Think of the wireless radios as 2 more physical LAN ports on the back of the router, grouped in with the rest.

Putting two routers on two different subnets does not make a routable segment. You are essentially connecting a bunch of access points together and expecting routing to happen, it won't, access points are Layer 2 devices. They don't care about IPs.

The only way you might be able to get this to work is using Yazfi to give different subnets to different guest wireless, and use those various guest wireless to interconnect the APs. Yazfi converts each guest wireless interface into a routed interface. But you're still going to come up short on interfaces probably, unless you use 2.4ghz for some interconnects and 5ghz for others.

Even if you get all of that working, the overhead of daisy chaining (repeating essentially) the wifi multiple times is going to reduce your throughput to a likely unusable amount.

You need to go back and look at fundamentals of routing, switching, and wireless before trying to build some theoretical routed mesh network. The asus are likely not going to be the right devices to even start to attempt this on, and using wireless as your backhaul medium for a mesh this big is not going to yield good (if any) results.

To be clear, if you set the LAN of router1 to 192.168.1.1/24 and the LAN of router 2 to 192.168.2.1/24 they will not be able to communicate with each other as that traffic never passes through a routed interface.

Yeh, instead I want to search how wds work(seems different manufacturer has different method), and why asus configure most interface to have the same mac address, but it is hard to find detailed one online.
I think the problem you mention and propose is that the whole wds things works on layer 2 and router only use forwarding when use WAN port. But what produce an illusion to let me keep on working is that router 1 instead can communicate with router 2 if they are configured to different subnet but are configured with proper ip rules of what you said:"

To be clear, if you set the LAN of router1 to 192.168.1.1/24 and the LAN of router 2 to 192.168.2.1/24 they will not be able to communicate with each other as that traffic never passes through a routed interface

". As the A--60--50--70--B scenario I originally mention, there are all wireless connection with each other (I' add 50 mac addr to 60's wds list...). only 60,50,70 have the proper ip routing rules can A and B do can ping to each other. If routing rules are not correct I promise A can B cannot ping to rach other.So I think under some situation routing still makes some work, but as you say they should works on layer2, so many weired things happen..

 

[drush8]

Yazfi

Oh I would see how Yazfi work

 

[drinkingbird]

Yeh, instead I want to search how wds work(seems different manufacturer has different method), and why asus configure most interface to have the same mac address, but it is hard to find detailed one online.
I think the problem you mention and propose is that the whole wds things works on layer 2 and router only use forwarding when use WAN port. But what produce an illusion to let me keep on working is that router 1 instead can communicate with router 2 if they are configured to different subnet but are configured with proper ip rules of what you said:"

". As the A--60--50--70--B scenario I originally mention, there are all wireless connection with each other (I' add 50 mac addr to 60's wds list...). only 60,50,70 have the proper ip routing rules can A and B do can ping to each other. If routing rules are not correct I promise A can B cannot ping to rach other.So I think under some situation routing still makes some work, but as you say they should works on layer2, so many weired things happen..

WDS simply isn't going to work for you. First it is limited to 54mbit/sec by WPA encryption. Second, it is just one big L2 network of repeaters, so that 54M is going to get cut in half several times. Even if you have perfect signal (which you don't) you'll have an unusable amount of bandwidth.

If you want to set up the model you show and have it be routed, you will need to find routers/access points that support that, ones with multiple radios that support wireless to wireless bridge mode.

But honestly the system you're trying to set up, nobody would ever do that with all wireless backhaul. It is going to be awful no matter what you do.

You may be able to get occasional ping responses, sometimes forcing traffic out an interface where another subnet sits will result in some proxy-arp happening (a security risk actually) and something will work, but it won't be flexible or reliable. It is a glitch at best, it is not routing traffic, it is just looping it back in a way.

 

[drinkingbird]

Oh I would see how Yazfi work

The only way I could think to make it work is put it on one of your routers near the middle then have all the ones around it be repeaters for those subnets, joining the various different guest networks that Yazfi creates.

It is possible that the ones in repeater mode you could do some scripting to assign IPs/subnets/routes to the wireless interfaces. It is going to be a big hack though, and still won't get you all the links you want in your diagram.

Maybe in WDS mode you could find a way to do something similar via scripts, but again, the bandwidth of WDS is going to choke.