does IGMP proxy/multicast routing with no ISP profile assist igmp snooping on managed switches

[drinkingbird]

the kbytes /sec is not from sniffing, it is running this script on rt-ax88u every 5 minutes, then retrieving and
graphing it on Home Assistant.
its burst of 1000bytes/sec seem to coincide with severe network issues.
the 9th column of /proc/net/dev is labeled multicast. I sum it across all interfaces whose name start with eth.
only interfaces i'm not summing up that have non-zero value are bcmsw, bond0.
maybe i should be including one of them to get accurate multicast byte data.
This field gives me total bytes, I take the byte difference between
2 measurements divided by the time difference in seconds of the 2 measurements which is 300seconds.

cat /proc/net/dev | awk '{ printf("%s %s\n",$1,$9); };' |grep eth | awk '{s+=$2} END {printf("%s\n",s);}'

I have everything disabled in IPTV, on my rt-ax88u i don't have a screen from IGMP snooping for LAN devices

i have airtime fairness disabled, on 2.4/5ghz wifi igmp snooping enabled, my multicast rate is set at OFDM 36.
I figured 36 which I believe means 36Mbps is a good value for wifi calling, and possible video conference calls by Microsoft Teams app on my mobile device. I have terrible Cell reception in my house, wifi call is essential.
i can't disable WMM, the menu dropdown won't let me, it only has "enable" if you click on it.
I have rebooted RT-AX88u several times already when I disabled earlier part of this thread the router didn't behave properly after disabling Multicast Routing.
after reboot it worked.

I did not factory reset the last 2 FW updates, but I have the one before those 2.
after i put all the iptables/ebtable rules on amazon devices only, wireshark i finally do see iphone make mdns traffic, its not as bad as amazon devices.
I'm starting to believe its not the router fault, but too many amazon devices. Home Assistant definitely aggravated the amazon echo system like kicking a hornets nest.
i can try another factory reset next weekend and re-customize all the screens again, i have documented most of the custom changes, so that i don't restore settings after factory reset.

Hardware acceleration in the Asus may be causing you to not see all traffic when looking in the router. The only way to have confidence in what you're seeing is to use a sniffer, but you'll need to insert a tap or hub in line or use a smart switch that supports SPAN/mirroring.

Multicast rate has nothing to do with wifi calling. Anything destined for the internet is unicast. Leaving it at auto to start is a good idea, and if you run into issues with streaming you can try tweaking it. For people who don't do much multicast streaming (not all streaming utilizes multicast, only streaming between devices on the same LAN, i.e. one device sending and multiple receiving, or something that pulls in a stream and converts it to multicast before sending to other devices) setting it to the lowest value can improve performance of the wifi network. Setting it too high can cause problems. 36 is near the high end which means your devices must have a pretty good signal or they won't be able to connect or will have issues. Auto has never been a problem for me, I don't do a ton of multicast but some. WMM will typically help wifi calling but that is enabled by default (and sounds like you can't disable it anyway).

Part of the goal of a factory reset would be to NOT reapply any tweaks etc you've done, start with the basics, leave the advanced stuff at defaults, see how it behaves, add features back as needed.

 

[Pierre Nakashian]

Hardware acceleration in the Asus may be causing you to not see all traffic when looking in the router. The only way to have confidence in what you're seeing is to use a sniffer, but you'll need to insert a tap or hub in line or use a smart switch that supports SPAN/mirroring.

Multicast rate has nothing to do with wifi calling. Anything destined for the internet is unicast. Leaving it at auto to start is a good idea, and if you run into issues with streaming you can try tweaking it. For people who don't do much multicast streaming (not all streaming utilizes multicast, only streaming between devices on the same LAN, i.e. one device sending and multiple receiving, or something that pulls in a stream and converts it to multicast before sending to other devices) setting it to the lowest value can improve performance of the wifi network. Setting it too high can cause problems. 36 is near the high end which means your devices must have a pretty good signal or they won't be able to connect or will have issues. Auto has never been a problem for me, I don't do a ton of multicast but some. WMM will typically help wifi calling but that is enabled by default (and sounds like you can't disable it anyway).

Part of the goal of a factory reset would be to NOT reapply any tweaks etc you've done, start with the basics, leave the advanced stuff at defaults, see how it behaves, add features back as needed.

For manual multicast analysis, i use wireshark over my notebook wifi in promiscuous mode. For trends i use the automated 5min measurement tools I've setup. If hardware acceleration makes the automated trend chart faulty, i'll have to play with my new managed switch in front of RT-AX88u, it supports mirroring of port. too bad wireshark doesn't have good graphics to display bytes/sec, it has packets/sec in statistics I/O graph. but it doesn't show which devices its coming from. it has another statistics source and destination addresses where I sort by percent and look at source IPv4 addresses, that sort of gives me which devices with highest count of packets and highest percentage of traffic i just have to remember how long i collected the sample to compare it against next time.

I used to have multicast rate on AUTO, then i read a thread in this forum to set to OFDM 6 after factory reset, i changed it recently to 36, because wrongfully i was thinking it was impacting my wifi calling indirectly, i was thinking it was allowing week wifi connection stay connected instead of choosing a closer access point. i have 3 access points extending wifi range. if that part is true, than wifi devices will close connections to the weak signal access point, and get connected to the fastest closest access point and even iphone wifi call can improve improve indirectly if it connects to the closest strongest access point signal.
all my devices have a tendency to stay connected to my main RT-AX88u only, even if the signal is weakest at one corner of my house, and there is a stronger signal nearby from an access point.

people couldn't hear me every 5-10seconds. I'll switch that back to auto and see how it impacts my devices. i may not care as much as most of my streaming devices now are wired an multicast rate is in the WIFI professional settings area. I'll pay attention if devices still stubbornly stay connected to the weak signal RT-AX88u.

There are some customizations in the router that solves some issues, that I need to carry over. I understand sometimes we need factory reset to initialize new variables that are implemented in new firmware, and restore settings could wipe out the new settings that need to be left alone.

 

[drinkingbird]

For manual multicast analysis, i use wireshark over my notebook wifi in promiscuous mode. For trends i use the automated 5min measurement tools I've setup. If hardware acceleration makes the automated trend chart faulty, i'll have to play with my new managed switch in front of RT-AX88u, it supports mirroring of port. too bad wireshark doesn't have good graphics to display bytes/sec, it has packets/sec in statistics I/O graph. but it doesn't show which devices its coming from. it has another statistics source and destination addresses where I sort by percent and look at source IPv4 addresses, that sort of gives me which devices with highest count of packets and highest percentage of traffic i just have to remember how long i collected the sample to compare it against next time.

I used to have multicast rate on AUTO, then i read a thread in this forum to set to OFDM 6 after factory reset, i changed it recently to 36, because wrongfully i was thinking it was impacting my wifi calling indirectly, i was thinking it was allowing week wifi connection stay connected instead of choosing a closer access point. i have 3 access points extending wifi range. if that part is true, than wifi devices will close connections to the weak signal access point, and get connected to the fastest closest access point and even iphone wifi call can improve improve indirectly if it connects to the closest strongest access point signal.
all my devices have a tendency to stay connected to my main RT-AX88u only, even if the signal is weakest at one corner of my house, and there is a stronger signal nearby from an access point.

people couldn't hear me every 5-10seconds. I'll switch that back to auto and see how it impacts my devices. i may not care as much as most of my streaming devices now are wired an multicast rate is in the WIFI professional settings area. I'll pay attention if devices still stubbornly stay connected to the weak signal RT-AX88u.

There are some customizations in the router that solves some issues, that I need to carry over. I understand sometimes we need factory reset to initialize new variables that are implemented in new firmware, and restore settings could wipe out the new settings that need to be left alone.

Not staying not to use your customizations, but best to see if issues have been resolved and those customizations (which could be causing other issues) may no longer be needed. So that's why I say add one thing back at a time, if it doesn't resolve whatever issue you're seeing, take it off and try another thing, etc. A lot of times during troubleshooting you can inadvertently leave things customized that didn't actually help but end up causing other issues.

Mirroring the port between the switch and router will at least give you an accurate view of what is happening on the wired LAN, and you can analyze wireless as you have been. There are plenty of tools out there that you can feed a Wireshark pcap into in order to get nice graphs etc, and they will be realtime instead of 5 min average which is practically useless.

As far as stubborn devices, the multicast rate setting can be a roundabout way of dealing with it, but not the ideal solution and can cause things to be even worse. Using roaming assistant is a better way to deal with that, or see if the problematic devices allow you to tweak the roaming aggressiveness.

Some don't like roaming assistant but I've found it to work perfectly fine (I have it set to -70 which is a fairly safe number, you can even go -60 or -50 depending on your environment and how aggressive you want it to be).

 

[Pierre Nakashian]

Not staying not to use your customizations, but best to see if issues have been resolved and those customizations (which could be causing other issues) may no longer be needed. So that's why I say add one thing back at a time, if it doesn't resolve whatever issue you're seeing, take it off and try another thing, etc. A lot of times during troubleshooting you can inadvertently leave things customized that didn't actually help but end up causing other issues.

Mirroring the port between the switch and router will at least give you an accurate view of what is happening on the wired LAN, and you can analyze wireless as you have been. There are plenty of tools out there that you can feed a Wireshark pcap into in order to get nice graphs etc, and they will be realtime instead of 5 min average which is practically useless.

As far as stubborn devices, the multicast rate setting can be a roundabout way of dealing with it, but not the ideal solution and can cause things to be even worse. Using roaming assistant is a better way to deal with that, or see if the problematic devices allow you to tweak the roaming aggressiveness.

Some don't like roaming assistant but I've found it to work perfectly fine (I have it set to -70 which is a fairly safe number, you can even go -60 or -50 depending on your environment and how aggressive you want it to be).

I changed roaming assistant back to on, and turned changed multicast rate back to auto.
some devices are still stubborn, mostly old 2.4ghz devices 1 Logitech remote hub and 2 old amazon products.

I do have a question what process manages wifi? not the guest network.
I want to try to restart this specific service/process next time without rebooting.
I had a few processes stopped working properly (my own sslh process), I use sslh as a way of multiplexing a number of different protocol incoming request to 1 listening socket 443. at the time i was connecting vpn over stunnel through sslh listening port.

i killed it and restarted the sslh process. next couple of hours wifi calling was terrible, i turned off each network device around the house, nothing helped, until I rebooted RT-AX88u.

while wifi calling was horrible Microsoft team call on a notebook over hard wired lan was fine. the metrics on RT-ax88u was fine, cswch/sec, proc/sec, memory, loads 5min. Fine means the trend was the same all day.

i did try some initial tests on managed switch to mirror traffic to RT-AX88u, and found a new menu in
wireshark to try "udp multicast streams" it looks like there are a number devices that multicast to this address 239.255.255.250 and on this destination port 1900 (UPNP SSDP) that do consume High Mbits bandwith. surveillance is uploading 1 camera footage to cloud. Today I have a wemo smart plug probably downloading firmware update. hopefully its not caught in a loop like last time where download fails it restarts over and over.

I'll factory reset RT-AX88u over this weekend. I have moved all amazon wireless devices to guest wifi now.

 

[drinkingbird]

I changed roaming assistant back to on, and turned changed multicast rate back to auto.
some devices are still stubborn, mostly old 2.4ghz devices 1 Logitech remote hub and 2 old amazon products.

I do have a question what process manages wifi? not the guest network.
I want to try to restart this specific service/process next time without rebooting.
I had a few processes stopped working properly (my own sslh process), I use sslh as a way of multiplexing a number of different protocol incoming request to 1 listening socket 443. at the time i was connecting vpn over stunnel through sslh listening port.

i killed it and restarted the sslh process. next couple of hours wifi calling was terrible, i turned off each network device around the house, nothing helped, until I rebooted RT-AX88u.

while wifi calling was horrible Microsoft team call on a notebook over hard wired lan was fine. the metrics on RT-ax88u was fine, cswch/sec, proc/sec, memory, loads 5min. Fine means the trend was the same all day.

i did try some initial tests on managed switch to mirror traffic to RT-AX88u, and found a new menu in
wireshark to try "udp multicast streams" it looks like there are a number devices that multicast to this address 239.255.255.250 and on this destination port 1900 (UPNP SSDP) that do consume High Mbits bandwith. surveillance is uploading 1 camera footage to cloud. Today I have a wemo smart plug probably downloading firmware update. hopefully its not caught in a loop like last time where download fails it restarts over and over.

I'll factory reset RT-AX88u over this weekend. I have moved all amazon wireless devices to guest wifi now.

UPNP/SSDP is similar to MDNS. Devices advertising capabilities to each other, and UPNP can also tell a router to open ports, cameras frequently use this capability for remote viewing. Totally normal to see it on a local LAN. While it is possible for it to consume a lot of bandwidth, it is unlikely on a home network with even a moderate amount of devices. It also would never go out the WAN since again, there is no multicast on the internet. However if your VPN, stunnel, SSLH are not configured properly you could be attempting to send the multicast out through there which could result in problems.

Someone here can probably advise on how to bounce the wifi process (you can also try cycling it with the button on the router, if yours has one), I don't remember what the process is called on your router. However it sounds like you have a lot of stuff customized and hacked together. SSLH and/or VPN could certainly be interfering with your wifi calling and even causing some other slowness you've been experiencing, especially if not configured exactly right (and even then, it will be processed in CPU not hardware, so if something high bandwidth or high PPS gets caught in it, it will definitely bog down).

Trying to troubleshoot the router is not the answer here, you need to troubleshoot your customizations as it is almost guaranteed that is what is causing your problems. Start clean, add back only the custom stuff you determine to be 100% necessary, and make sure everything is still working properly after adding each thing back.

 

[Pierre Nakashian]

UPNP/SSDP is similar to MDNS. Devices advertising capabilities to each other, and UPNP can also tell a router to open ports, cameras frequently use this capability for remote viewing. Totally normal to see it on a local LAN. While it is possible for it to consume a lot of bandwidth, it is unlikely on a home network with even a moderate amount of devices. It also would never go out the WAN since again, there is no multicast on the internet. However if your VPN, stunnel, SSLH are not configured properly you could be attempting to send the multicast out through there which could result in problems.

Someone here can probably advise on how to bounce the wifi process (you can also try cycling it with the button on the router, if yours has one), I don't remember what the process is called on your router. However it sounds like you have a lot of stuff customized and hacked together. SSLH and/or VPN could certainly be interfering with your wifi calling and even causing some other slowness you've been experiencing, especially if not configured exactly right (and even then, it will be processed in CPU not hardware, so if something high bandwidth or high PPS gets caught in it, it will definitely bog down).

Trying to troubleshoot the router is not the answer here, you need to troubleshoot your customizations as it is almost guaranteed that is what is causing your problems. Start clean, add back only the custom stuff you determine to be 100% necessary, and make sure everything is still working properly after adding each thing back.

I may have stumbled on an intermittent problem on the router that could be the culprit I had forgotten about, my USB3 connected HDD I just noticed the swap memory suddenly is turned off, I run the swapon manually and after a certain time it gets turned off again. This is after I performed a complete factory reset of the router this morning and reset the configuration manually, but the JFFS partition I did reimport.
I switched the HDD from USB3 mode to USB2 mode and the swap again is getting turned off. Reboot on my old router made the USB HDD intermittently not mount, On this router it mounts but I have no explanation why the swap would just get turned off.

I also checked the multicast traffic over vpn, it is not transmitted from my LAN or public IP to the notebook running openvpn client. For the reverse I'll just put an ebtables command very similar to the guest network.

If this is my HDD dying maybe I should try a usb stick. All my customized utilities are running off of this HDD as well.

 

[sfx2000]

UPNP/SSDP is similar to MDNS.

I wouldn't couple mDNS/Avahi/Boujour to UPNP/SSDP...

They are different tools to different ends...

 

[drinkingbird]

I wouldn't couple mDNS/Avahi/Boujour to UPNP/SSDP...

They are different tools to different ends...

Different tools yes, different ends, not really. Both are used for discovering and mapping a network without a central authority to track that information (which is why they're relatively inefficient). That's why I said they are "similar" since both use multicast to announce capabilities and query for other devices (yes UPNP has some extra protentional functionality but the SSDP portion is pretty much the same).

User was asking what that multicast group was for, just letting them know it is similar to the other MC group they were seeing for MDNS and that's why they were seeing similar behavior.

 

[drinkingbird]

I may have stumbled on an intermittent problem on the router that could be the culprit I had forgotten about, my USB3 connected HDD I just noticed the swap memory suddenly is turned off, I run the swapon manually and after a certain time it gets turned off again. This is after I performed a complete factory reset of the router this morning and reset the configuration manually, but the JFFS partition I did reimport.
I switched the HDD from USB3 mode to USB2 mode and the swap again is getting turned off. Reboot on my old router made the USB HDD intermittently not mount, On this router it mounts but I have no explanation why the swap would just get turned off.

I also checked the multicast traffic over vpn, it is not transmitted from my LAN or public IP to the notebook running openvpn client. For the reverse I'll just put an ebtables command very similar to the guest network.

If this is my HDD dying maybe I should try a usb stick. All my customized utilities are running off of this HDD as well.

Well unfortunately at this point it sounds like step by step troubleshooting is not something you want to do if you're restoring your whole JFFS with scripts etc. Your router should be able to handle whatever MDNS or SSDP traffic is on your network (with the exception of a device misbehaving, but the volume of traffic you were seeing did not point to that). Your problem likely lies in something else you have customized or configured. It also sounds like you have multiple layers of problems going on.

Your suggestion to just configure ebtables to block traffic that shouldn't be there in the first place is a good example. Reminds me of the old "we couldn't fix your brakes so we make your horn louder" joke. Just layering on "fixes" isn't really the right way to go about this. This isn't a powerful linux PC, this is a cheap router with SOC in it that is optimized to do a few specific things.

 

[Pierre Nakashian]

Well unfortunately at this point it sounds like step by step troubleshooting is not something you want to do if you're restoring your whole JFFS with scripts etc. Your router should be able to handle whatever MDNS or SSDP traffic is on your network (with the exception of a device misbehaving, but the volume of traffic you were seeing did not point to that). Your problem likely lies in something else you have customized or configured. It also sounds like you have multiple layers of problems going on.

Your suggestion to just configure ebtables to block traffic that shouldn't be there in the first place is a good example. Reminds me of the old "we couldn't fix your brakes so we make your horn louder" joke. Just layering on "fixes" isn't really the right way to go about this. This isn't a powerful linux PC, this is a cheap router with SOC in it that is optimized to do a few specific things.

I always backup JFFS before a complete factory reset after firmware update. I then set all my custom settings manually lastly I restore my JFFS. This is my standard practice. The memory profile now looks better. Memory now is at or below 50%. Before factory reset it climbed to around 60%.

>>Your router should be able to handle whatever MDNS or SSDP traffic is on your network

Is that still true even if you have 100 active devices mixes of wifi/lan. I'm not an expert on this, just researching alot, and I could be taking this info out of context, but I've read some multicast traffic are many to many. Many to many with 100 devices is a lot of combinations of traffic. I used the link below to figure out if I had 100 devices and if they communicated either in all possible permutation or all possible combination with each other, just splitting my 1 LAN devices into 2 VLANs where half couldn't see the other half, the number of permutations or number of combinations drops by half (the sum of 2 vlans is half of 1 LAN). My biggest wish list is if ASUS supported VLAN natively. My hunch is If I Split 1 LAN network (100 devices) into smaller VLAN groups should make the RT-AX88u handle the traffic with less resources.

formula for permutation & combination

Thanks to many threads in this site and lots of youtube videos on vlan, I researched and experimented for 6 days, how to setup several VLANS in RT-AX88u, and 1 vlan in my access points RC-AC68u, and configured the 2 vlans in my TP-LINK managed switches i just purchased in between RT-AX88u and RT-AC68u to put the chatty wired/wireless devices such as Amazon, google devices into an isolated VLAN 100 comprised of guest wifi and hard wired devices. IGMP snooping is also turned on in the managed switches. This was a very hard probably tweaks are still needed. I do now have 3 subnets vlan100, vlan200 and default for the remainder vlan 1 which I've read doesn't even have a tag. The assignment of devices to VLAN is either via them joining to the guest wifi, or managed switch configuration web ui on assigning a specific port to a vlan untagged with a specific PVID.

I did have some problems , my JFFS was border line full, the features I had turned on in the past but are turned off now, such as traffic analyzer, web history, left behind very large files. I deleted them, I'm down to 39% used, I reached 100% full while I was connected over vpn, and I couldn't even delete the files. But got this resolved now.

I also researched wifi calling on iphone, it seems to be a problem independent of the RT-AX88u. when wifi calling causes the other party to hear me as a robotic voice, several calls in a row, just turning off the iphone then on fixed it today. So I reset the iphone network settings as a precaution. We'll see how long that holds out. but i can't blame my network anymore.

I'm at the end of troubleshooting. Now just going to see if my vlan setup can be optimized, not something for this thread though.

 

[drinkingbird]

I always backup JFFS before a complete factory reset after firmware update. I then set all my custom settings manually lastly I restore my JFFS. This is my standard practice. The memory profile now looks better. Memory now is at or below 50%. Before factory reset it climbed to around 60%.

>>Your router should be able to handle whatever MDNS or SSDP traffic is on your network

Is that still true even if you have 100 active devices mixes of wifi/lan. I'm not an expert on this, just researching alot, and I could be taking this info out of context, but I've read some multicast traffic are many to many. Many to many with 100 devices is a lot of combinations of traffic. I used the link below to figure out if I had 100 devices and if they communicated either in all possible permutation or all possible combination with each other, just splitting my 1 LAN devices into 2 VLANs where half couldn't see the other half, the number of permutations or number of combinations drops by half (the sum of 2 vlans is half of 1 LAN). My biggest wish list is if ASUS supported VLAN natively. My hunch is If I Split 1 LAN network (100 devices) into smaller VLAN groups should make the RT-AX88u handle the traffic with less resources.

formula for permutation & combination

Thanks to many threads in this site and lots of youtube videos on vlan, I researched and experimented for 6 days, how to setup several VLANS in RT-AX88u, and 1 vlan in my access points RC-AC68u, and configured the 2 vlans in my TP-LINK managed switches i just purchased in between RT-AX88u and RT-AC68u to put the chatty wired/wireless devices such as Amazon, google devices into an isolated VLAN 100 comprised of guest wifi and hard wired devices. IGMP snooping is also turned on in the managed switches. This was a very hard probably tweaks are still needed. I do now have 3 subnets vlan100, vlan200 and default for the remainder vlan 1 which I've read doesn't even have a tag. The assignment of devices to VLAN is either via them joining to the guest wifi, or managed switch configuration web ui on assigning a specific port to a vlan untagged with a specific PVID.

I did have some problems , my JFFS was border line full, the features I had turned on in the past but are turned off now, such as traffic analyzer, web history, left behind very large files. I deleted them, I'm down to 39% used, I reached 100% full while I was connected over vpn, and I couldn't even delete the files. But got this resolved now.

I also researched wifi calling on iphone, it seems to be a problem independent of the RT-AX88u. when wifi calling causes the other party to hear me as a robotic voice, several calls in a row, just turning off the iphone then on fixed it today. So I reset the iphone network settings as a precaution. We'll see how long that holds out. but i can't blame my network anymore.

I'm at the end of troubleshooting. Now just going to see if my vlan setup can be optimized, not something for this thread though.

Not saying you can't restore JFFS eventually just that you should try things before doing it, take a phased approach, that's how you troubleshoot.

Splitting up broadcast domains can help for some things and hurt for others. If your router is now having to route traffic between subnets, that is more load on the router (and less throughput potentially). The router is still seeing all the MDNS and broadcasts, it may reduce the chain effect of MDNS (one device queires, other devices respond) but unlikely it is a massive difference.

Multicast is one to many but if other hosts then respond it can behave similar to a many to many or broadcast. But your router is seeing and dealing with broadcasts all the time, since it does not route multicast it should not really be impacted by it. Bridging it between wifi and wired, or between wifi bands might stress it some but those MDNS packets are so small and, unless something is misconfigured, not as numerous as you might think.

After you reset things and before restoring custom stuff, did it seem to be working better? It is more likely that something is catching and forwarding (or trying to forward) those multicast packets into a black hole

 

[Pierre Nakashian]

Not saying you can't restore JFFS eventually just that you should try things before doing it, take a phased approach, that's how you troubleshoot.

Splitting up broadcast domains can help for some things and hurt for others. If your router is now having to route traffic between subnets, that is more load on the router (and less throughput potentially). The router is still seeing all the MDNS and broadcasts, it may reduce the chain effect of MDNS (one device queires, other devices respond) but unlikely it is a massive difference.

Multicast is one to many but if other hosts then respond it can behave similar to a many to many or broadcast. But your router is seeing and dealing with broadcasts all the time, since it does not route multicast it should not really be impacted by it. Bridging it between wifi and wired, or between wifi bands might stress it some but those MDNS packets are so small and, unless something is misconfigured, not as numerous as you might think.

After you reset things and before restoring custom stuff, did it seem to be working better? It is more likely that something is catching and forwarding (or trying to forward) those multicast packets into a black hole

The factory reset and JFFS restore was a step I took, to put the router into a clean known state. I wasn't using it to troubleshoot but rather to make sure the firmware upgrade put it into the expected known state. But noting your suggestion for next time.

I definitely see top command load average went up by about .5 it is now at 3.45
but top cpu %idle is now more consistent at 95%, fewer spikes.

After I reset the router but before I fully implemented VLAN setup the Multicast was more sporadically volatile but I have to wait another week to give an accurate assessment on this. Using wireshark, It seems the amazon devices become dormant at times, when they wake up Multicast metrics go up. Home assistant was irritating this group before, I shielded that device from inside the raspberry pie. The VLAN setup has a higher overhead, but seem to shield it against these chatty IOT devices, (not just 1, I own a dozen amazon devices, another 6 google devices). Its like someone kicking a hornet nest they all just get active and settle down eventually. Right now the amazon devices are not even the top 10 in wireshark statistics->"udp multicast streams"

Ironically the amazon fire tv cube 2nd gen with its 100mbit interface seems to be very sensitive to the network multicast traffic in my LAN some of it caused by its sibling devices, streaming 4k videos even from Prime videos with my 1gbit internet fiber plan (uplink/downlink is 1gbit) has unbearable frequent pauses/buffering, but when I put the Fire tv Cube behind the RT-AC68u Access point and setup a vlan to shield the Fire TV Lan interface even from the other ports on the access point I can stream 4k content from amazon all day with no issues. Not much network cache on the amazon device I guess.

As an experiment I connected the Fire TV cube directly into the managed switch, but still in the VLAN 100, it still had unbearable frequent video pauses, until I unplugged the Surveillance from the RT-AX88u port completely. Now that amazon devices are quieter, wireshark showed the Lorex NVR surveillance system is the next highest device spewing some sort of multicast traffic at destination address 239.255.255.250 destination port 3702. Plugging the NVR system to the managed switch (without vlan for now) instead of the RT-AX88u, Fire TV cube still experiences some video pauses, but at a reduced level. Somehow the RT-AX88u is being impacted by the surveillance traffic, I never considered this device as a culprit. moving the NVR connection from RT-AX88u to the managed switch also dropped the context switches on RT-AX88u from 6000 to 3700.

Below is a screenshot of the wireshark "UDP Multicast Streams" sorted by Average BW.
192.168.2.51 is the surveillance NVR system, 192.168.2.30 is the Home assistant raspberry pie fetching some of the video snapshots to display on its web portal. The next highest Average BW not on screenshot is my notebook running wireshark only 2301 bps a sharp dive from 1068Kbps as the next higher reading.

Is the managed switch IGPMP snooping what's helping here? traffic not seen by any device except by subscribers in the LAN? I will create a custom VLAN just for this device later to make its effects even less to others. I"ll see if Fire TV cube can just be plugged in the managed switch without video pauses.

The buffering symptom is the worst symptom in my network, it's managed by custom vlan scripts inside the RT-AC68u. I don't believe the Iphone issue is related to my network setup. The extreme attention I'm giving my network is because every 2-3months my network goes down , when that happens I have to login to all the access points and run "service stop_mdns" for network becomes functional again to my entire house. I think that shouldn't happen anymore with my managed smart switches placed between RT-AX88u and my RT-AC68u access points and the switches using IGMP snooping now and VLAN setup.

 

[drinkingbird]

The factory reset and JFFS restore was a step I took, to put the router into a clean known state. I wasn't using it to troubleshoot but rather to make sure the firmware upgrade put it into the expected known state. But noting your suggestion for next time.

I definitely see top command load average went up by about .5 it is now at 3.45
but top cpu %idle is now more consistent at 95%, fewer spikes.

After I reset the router but before I fully implemented VLAN setup the Multicast was more sporadically volatile but I have to wait another week to give an accurate assessment on this. Using wireshark, It seems the amazon devices become dormant at times, when they wake up Multicast metrics go up. Home assistant was irritating this group before, I shielded that device from inside the raspberry pie. The VLAN setup has a higher overhead, but seem to shield it against these chatty IOT devices, (not just 1, I own a dozen amazon devices, another 6 google devices). Its like someone kicking a hornet nest they all just get active and settle down eventually. Right now the amazon devices are not even the top 10 in wireshark statistics->"udp multicast streams"

Ironically the amazon fire tv cube 2nd gen with its 100mbit interface seems to be very sensitive to the network multicast traffic in my LAN some of it caused by its sibling devices, streaming 4k videos even from Prime videos with my 1gbit internet fiber plan (uplink/downlink is 1gbit) has unbearable frequent pauses/buffering, but when I put the Fire tv Cube behind the RT-AC68u Access point and setup a vlan to shield the Fire TV Lan interface even from the other ports on the access point I can stream 4k content from amazon all day with no issues. Not much network cache on the amazon device I guess.

As an experiment I connected the Fire TV cube directly into the managed switch, but still in the VLAN 100, it still had unbearable frequent video pauses, until I unplugged the Surveillance from the RT-AX88u port completely. Now that amazon devices are quieter, wireshark showed the Lorex NVR surveillance system is the next highest device spewing some sort of multicast traffic at destination address 239.255.255.250 destination port 3702. Plugging the NVR system to the managed switch (without vlan for now) instead of the RT-AX88u, Fire TV cube still experiences some video pauses, but at a reduced level. Somehow the RT-AX88u is being impacted by the surveillance traffic, I never considered this device as a culprit. moving the NVR connection from RT-AX88u to the managed switch also dropped the context switches on RT-AX88u from 6000 to 3700.
View attachment 45206
Below is a screenshot of the wireshark "UDP Multicast Streams" sorted by Average BW.
192.168.2.51 is the surveillance NVR system, 192.168.2.30 is the Home assistant raspberry pie fetching some of the video snapshots to display on its web portal. The next highest Average BW not on screenshot is my notebook running wireshark only 2301 bps a sharp dive from 1068Kbps as the next higher reading.

Is the managed switch IGPMP snooping what's helping here? traffic not seen by any device except by subscribers in the LAN? I will create a custom VLAN just for this device later to make its effects even less to others. I"ll see if Fire TV cube can just be plugged in the managed switch without video pauses.

View attachment 45204

The buffering symptom is the worst symptom in my network, it's managed by custom vlan scripts inside the RT-AC68u. I don't believe the Iphone issue is related to my network setup. The extreme attention I'm giving my network is because every 2-3months my network goes down , when that happens I have to login to all the access points and run "service stop_mdns" for network becomes functional again to my entire house. I think that shouldn't happen anymore with my managed smart switches placed between RT-AX88u and my RT-AC68u access points and the switches using IGMP snooping now and VLAN setup.

MDNS traffic will spike when a device wakes up or is told to discover other devices. It will send an MDNS discovery and all other devices that support MDNS will respond. Normally, that traffic should stay on the LAN and be completely switched with very little impact to network devices. The Layer 3 router does not need to do anything with those packets other than ignore them, since the Asus is not multicast enabled (at least not through the L3 path from LAN to WAN). It is possible the router is listening for MDNS and responding/participating (a normal/strict router would not, but this is a home router with various features added, so it is possible). It is also possible that if you have VPN, diversion, etc enabled, that the traffic is attempting to forward to those services and causing extra load on the router. If the router is IGMP enabled, it is maintaining a list of clients that are participating in that group, and sending IGMP messages back and forth, but those are pretty light activity. I guess if this table got too big it could overload a simple router like the Asus.

The amazon device that is having issues when seeing MDNS packets, it is likely not due to bandwidth (the stream is 15-25 megs and you have 100 meg interface) but processing and responding to those MDNS that is causing it to stutter - the CPU in the device probably just can't handle decoding the stream and dealing with all the MDNS traffic too. Make sure that amazon device is actually negotiating to 100/Full on your switch and the cabling is good, as that is another possibility. That 2nd gen cube is fairly old at this point, and 4K may be more than it was intended to handle alongside its other functions. CPU just can't handle it. Just a guess.

IGMP snooping likely is not helping much/at all with MDNS. Given that MDNS is essentially a broadcast, it will go to all hosts that support MDNS. IGMP snooping only "prunes" back multicast off a port if the device is not requesting/participating in that group (where all MDNS enabled devices will be participating, so it would not prune them). And the traffic is so brief and small, even the ports that don't need that data (non-MDNS enabled ones) aren't being impacted much by it. While snooping may help some, it can hurt MDNS and UPNP if there is no IGMP router maintaining the group memberships.

IGMP snooping would mainly help if you had a box that streams multicast video or other data to several other network devices (but not all of them), IGMP snooping will prune the stream from the ports that don't need that data. Note that for IGMP and IGMP snooping to function, there does need to be a Layer 3 device (router) that is IGMP aware. I'm assuming the Asus is, and can handle the necessary IGMP reports (since it does have the option of IGMP snooping in the router), but I don't know that for certain, it may only perform that function when you have IPTV enabled from your ISP. If the Asus is not handling the IGMP reports (nor is any other device on your network) then IGMP snooping won't function on your smart switches either, enabling it will have no effect. But then, most multicast would not be working anyway, or it may work then die out after a timer expires. A box that is designed to stream multicast data out may have an IGMP router feature in it to handle the IGMP queries and reports, in that case multicast and IGMP snooping will function correctly. Even some computers will have this ability (depending on the software installed and network config), but the IGMP router must be on the same VLAN as the devices participating in multicast. This is why I say having more VLANs could actually make things worse, if you isolate your IGMP router from devices that are using multicast, they will no longer function correctly and may repeatedly attempt to establish multicast causing extra traffic and instability.

But from your description it sounds like it isn't the router or network hardware that is having issues with the MDNS, but specific devices. Isolating those devices would help in that case, however you'll lose their ability to discover devices that are on other VLANs. If you have clusters of device types, i..e. cameras, amazon TV, google home, etc you could isolate each type into its own VLAN (assuming you don't need them to see/discover each other). But be mindful of the note above where every VLAN must have an IGMP aware router in it if it needs to handle multicast traffic (IF the asus is, then since it is participating in every VLAN that should cover it).

For the NVR, 239.255.255.250 is UPNP/SSDP, it is similar to MDNS and the device is advertising its capabilities and potentially also asking your router to open ports for it. It is known that IGMP snooping interferes with UPNP especially if there is no IGMP router managing membership. So that interference could explain the reduced traffic you see when plugging it into your switch.

Multicast troubleshooting is hard enough with an enterprise router/switch/ap, doing it on the Asus is going to be a lot harder, and trial and error will play a big part.

Unfortunately in this case where I'm not sure if Asus (or even your particular Asus model, since there are two main chipsets which vary a lot) actually supports IGMP on the LAN when IPTV on the WAN is not in use, makes it difficult to give solid answers to a lot of these questions, hence the "if" or "possible" or "maybe" used a lot. Maybe Merlin or someone else knows the answer to this.

 

[Pierre Nakashian]

>>> If the router is IGMP enabled, it is maintaining a list of clients that are participating in that group, and sending IGMP messages back and forth, but those are pretty light activity. I guess if this table got too big it could overload a simple router like the Asus

I am seeing IGMP query from RT-AX88u, and reports back by my IOT devices. But this traffic is very light, per /var/mcpd.conf it is every 20 seconds, thats roughly how often i saw it pop in wireshark.

2nd gen Fire TV cube is connected at 100mbps, internet speed is slightly above 100mbps. So the stutter problem I mitigated by using RT-AC68u VLAN scripts was keeping the MDNS packets away from the Fire TV cube. Using that Idea on the managed switch where I also placed the Fire TV cube on the same VLAN 100, I also just turned on managed Switch packet filtering against multicast/broadcast/UL Frame to the lowest setting It allows me to. It helped just a little bit, the stuttering was not eliminated as I can't eliminate incoming multicast completely by the managed switch. The RT-AC68u VLAN script to isolate the port worked well when I didn't have any other devices in the same VLAN. The script uses commands robocfg, vconfig, brctl, ifconfig, nvram but no ebtables command. Its pretty low maintenance. Putting all Amazon devices into the same isolated VLAN didn't help the Fire TV cube devices, even seeing the other amazon device multicast traffic and a small handful google devices is still slowing the Fire TV cube cpu.

>>IGMP snooping would mainly help if you had a box that streams multicast video or other data to several other network devices

The surveyance video stream was being sent from multiple Cameras on a POE hub to a computer running blue-iris, I'm in the process of retiring the Lorex NVR it was still connected. when the POE hub is connected to RT-AX88u, the context switches on RT-AX88u jumps by 2000. Connecting to the managed Switch instead without VLAN setup and removing the Lorex NVR from my network, context switches still jumped by 1800 on the RT-AX88u. Finally creating a new VLAN 300, placing the Blue-iris computer and cameras to the same VLAN 300 with PVID=300 help eliminate the entire 2000 context switches from RT-AX88u. That last part I also removed Home assistant receiving any feed, it seems to use resources even when I'm not viewing its web portal page. Wireshark did see both the blue IRIS computer and Home Assistant using high Bandwith MBps and no other device had Average BW bps above 1435bps, so IGMP snooping is working there, but it didn't shield the RT-AX88u from the 2000 context switches. traffic was going through the RT-AX88u rather than bypass it even when I plugged in the cameras to the managed switch with both cameras and blue iris computer being on the default native vlan 1 (no tag). My managed switches are just layer 2, they don't know ip addresses, they know vlan pvid, so vlan 1 which doesn't have a tag doesn't know how to bypass the router??.

>>For the NVR, 239.255.255.250 is UPNP/SSDP, it is similar to MDNS and the device is advertising its capabilities and potentially also asking your router to open ports for it. It is known that IGMP snooping interferes with UPNP especially if there is no IGMP router managing membership. So that interference could explain the reduced traffic you see when plugging it into your switch.

So the impact to the RT-AX88u high context switches is the UPNP feature of the router constantly being asked to open ports, interesting, thats why wireshark saw so many 239.255.255.250 from the same Lorex NVR with different source ports but same destination port 3702. Right now only Home assistant is still left using High bandwidth, even though I disabled cameras from its portal. it is also using UPNP/SSDP. 192.168.2.30 is Home assistant, followed by wireshark on my notebook.

I have to keep the cameras and the Blue-iris computer in an isolated VLAN 300, so the manage switches do all the work sending data to from without involving RT-AX88u router. I may also have to experiment turning off UPNP on the RT-AX88u, instead I'll try to create a static firewall rule If I can't stop Home Assistant doing those repeated Multicast calls with High Bandwidth like the ones listed above. The Lorex NVR calls similar to the ones above increased the Context Switches on the RT-AX88u, the lowest context switch is now at 3400, historically I've seen it as low as 1600. I have to find the culprit integration in Home assistant and turn it off.

>>Multicast troubleshooting is hard enough with an enterprise router/switch/ap, doing it on the Asus is going to be a lot harder, and trial and error will play a big part.

wireshark Statistics->"UDP Multicast streams" seem to be the tool to use for multicast troubleshooting in my case, I've identified targets and I just try a few changes and see if my context switches metric drop, or just the device not being listed as the highest Average Bandwidth culprit in wireshark.

Thanks for the explanations

 

[drinkingbird]

>>> If the router is IGMP enabled, it is maintaining a list of clients that are participating in that group, and sending IGMP messages back and forth, but those are pretty light activity. I guess if this table got too big it could overload a simple router like the Asus

I am seeing IGMP query from RT-AX88u, and reports back by my IOT devices. But this traffic is very light, per /var/mcpd.conf it is every 20 seconds, thats roughly how often i saw it pop in wireshark.

2nd gen Fire TV cube is connected at 100mbps, internet speed is slightly above 100mbps. So the stutter problem I mitigated by using RT-AC68u VLAN scripts was keeping the MDNS packets away from the Fire TV cube. Using that Idea on the managed switch where I also placed the Fire TV cube on the same VLAN 100, I also just turned on managed Switch packet filtering against multicast/broadcast/UL Frame to the lowest setting It allows me to. It helped just a little bit, the stuttering was not eliminated as I can't eliminate incoming multicast completely by the managed switch. The RT-AC68u VLAN script to isolate the port works best. The script uses commands robocfg, vconfig, brctl, ifconfig, nvram but no ebtables command. Its pretty low maintenance. Putting all Amazon devices into the same isolated VLAN didn't help the Fire TV cube devices, even seeing the other amazon device multicast traffic and a small handful google devices is still slowing the Fire TV cube cpu.

>>IGMP snooping would mainly help if you had a box that streams multicast video or other data to several other network devices

The surveyance video stream was being sent from multiple Cameras on a POE hub to a computer running blue-iris, I'm in the process of retiring the Lorex NVR it was still connected. when the POE hub is connected to RT-AX88u, the context switches on RT-AX88u jumps by 2000. Connecting to the managed Switch instead without VLAN setup and removing the Lorex NVR from my network, context switches still jumped by 1800 on the RT-AX88u. Finally creating a new VLAN 300, placing the Blue-iris computer and cameras to the same VLAN 300 with PVID=300 help eliminate the entire 2000 context switches from RT-AX88u. That last part I also removed Home assistant receiving any feed, it seems to use resources even when I'm not viewing its web portal page. Wireshark did see both the blue IRIS computer and Home Assistant using high Bandwith MBps and no other device had Average BW bps above 1435bps, so IGMP snooping is working there, but it didn't shield the RT-AX88u from the 2000 context switches. traffic was going through the RT-AX88u rather than bypass it even when I plugged in the cameras to the managed switch with both cameras and blue iris computer being on the default native vlan 1 (no tag). My managed switches are just layer 2, they don't know ip addresses, they know vlan pvid, so vlan 1 which doesn't have a tag doesn't know how to bypass the router??.

>>For the NVR, 239.255.255.250 is UPNP/SSDP, it is similar to MDNS and the device is advertising its capabilities and potentially also asking your router to open ports for it. It is known that IGMP snooping interferes with UPNP especially if there is no IGMP router managing membership. So that interference could explain the reduced traffic you see when plugging it into your switch.

So the impact to the RT-AX88u high context switches is the UPNP feature of the router constantly being asked to open ports, interesting, thats why wireshark saw so many 239.255.255.250 from the same Lorex NVR with different source ports but same destination port 3702. Right now only Home assistant is still left using High bandwidth, even though I disabled cameras from its portal. it is also using UPNP/SSDP. 192.168.2.30 is Home assistant, followed by wireshark on my notebook.
View attachment 45237

I have to keep the cameras and the Blue-iris computer in an isolated VLAN 300, so the manage switches do all the work sending data to from without involving RT-AX88u router. I may also have to experiment turning off UPNP on the RT-AX88u, instead I'll try to create a static firewall rule If I can't stop Home Assistant doing those repeated Multicast calls with High Bandwidth like the ones listed above. The Lorex NVR calls similar to the ones above increased the Context Switches on the RT-AX88u, the lowest context switch is now at 3400, historically I've seen it as low as 1600. I have to find the culprit integration in Home assistant and turn it off.

>>Multicast troubleshooting is hard enough with an enterprise router/switch/ap, doing it on the Asus is going to be a lot harder, and trial and error will play a big part.

wireshark Statistics->"UDP Multicast streams" seem to be the tool to use for multicast troubleshooting in my case, I've identified targets and I just try a few changes and see if my context switches metric drop, or just the device not being listed as the highest Average Bandwidth culprit in wireshark.

Thanks for the explanations

My guess would be the context switches you are seeing are the router processing MDNS and UPNP, and probably some IGMP stuff. They're getting kicked to the CPU since that is not a function of hardware. 2000 may or may not be an issue for your router, no idea. IGMP reports every 20 seconds sounds right.

Disabling uPNP on the router is always a good idea. I never leave it running. However that won't stop the devices from sending those packets. It should help reduce CPU calls and possibly response traffic since the router can now ignore it. You'd need to go on the device(s) and disable it if possible to completely eliminate it. Any port forwarding needed should be done manually, not via uPNP (for security reasons). Less convenient, but uPNP can be a big security hole. Malware happily takes advantage of it. Even the latest and greatest, most secure uPNP implementations are not secure or safe.

Good to know the router is IGMP enabled (apparently). That should allow multicast and IGMP snooping on all your switches to behave and work properly.

If your 88u supports robocfg then configuring VLANs is very easy. I guess it is a non-HND router.

Anything in VLAN 1 no matter how many switches you have can make it to the router. If it is a broadcast, multicast, or if the destination is a MAC residing on the router, it will hit it. Traffic between two devices directly on the switch won't hit the router, even if they are in VLAN 1. That's the point of switches. Anything in a tagged VLAN that is configured on both the switch and the router will behave the same way (assuming you tag it facing the router from the switch). The difference is you can filter stuff from going between those VLANs, and broadcast and multicast will mostly get blocked by default.

Placing the Fire TV in its own VLAN by itself and trunking that VLAN to the router should eliminate most/all MDNS traffic. Of course it won't be able to see/discover other devices. Or you could leave it on VLAN 1 but move the devices that seem to generate a lot of MDNS traffic off to their own VLAN(s).

Yes using wireshark and testing out different configs is what I mean by "trial and error". To really attack and troubleshoot multicast you need to be able to see mroutes, igmp/PIM memberships (doubt the asus supports PIM, that's for routing multicast between routers or VLANs), etc right on the device itself. Whether that can be done on the Asus OS, not sure, probably not much if any.

Just keep in mind, more VLANs means more work for the router to do if those devices need to talk to each other. Now instead of switching traffic between two devices on the same VLAN, you're routing it between two different VLANs. Routing performance, even with hardware acceleration, is going to be much less than switching.

Even your cameras/blue iris being in an "isolated" VLAN, unless you don't send that VLAN to the router, will not behave differently than if they were just in VLAN 1 (other than potentially reducing the amount of MDNS/uPNP traffic "domino effect" since it won't see and interact with as many devices using those protocols). If you don't trunk the VLAN to the router, then you'll need a second interface off the NVR to actually access it from outside that VLAN, which I'm assuming you want to be able to do. Remember, the router has a switch built in, and that switch will properly handle traffic directly between two devices without sending it to the router/CPU.

You may need to take a device by device (or device type by device type) approach. For each device/device type, see if it is doing excessive amounts of uPNP or MDNS. If so, try to disable or block UPNP from that device. You probably can't disable MDNS but you can block it potentially (as long as you realize it will not be able to do discovery of other devices). Depending what you observe with each device you can decide whether isolating it is necessary or not.

But if at this point the Fire TV is the only thing having problems, I think you're way over thinking this. Either replace it with a newer model or put just that into its own VLAN and let everything else stay in the main VLAN. I suspect it just can't handle 4k decoding along with attempting to do other things.

 

[Pierre Nakashian]

My guess would be the context switches you are seeing are the router processing MDNS and UPNP, and probably some IGMP stuff. They're getting kicked to the CPU since that is not a function of hardware. 2000 may or may not be an issue for your router, no idea. IGMP reports every 20 seconds sounds right.

Disabling uPNP on the router is always a good idea. I never leave it running. However that won't stop the devices from sending those packets. It should help reduce CPU calls and possibly response traffic since the router can now ignore it. You'd need to go on the device(s) and disable it if possible to completely eliminate it. Any port forwarding needed should be done manually, not via uPNP (for security reasons). Less convenient, but uPNP can be a big security hole. Malware happily takes advantage of it. Even the latest and greatest, most secure uPNP implementations are not secure or safe.

Good to know the router is IGMP enabled (apparently). That should allow multicast and IGMP snooping on all your switches to behave and work properly.

If your 88u supports robocfg then configuring VLANs is very easy. I guess it is a non-HND router.

Anything in VLAN 1 no matter how many switches you have can make it to the router. If it is a broadcast, multicast, or if the destination is a MAC residing on the router, it will hit it. Traffic between two devices directly on the switch won't hit the router, even if they are in VLAN 1. That's the point of switches. Anything in a tagged VLAN that is configured on both the switch and the router will behave the same way (assuming you tag it facing the router from the switch). The difference is you can filter stuff from going between those VLANs, and broadcast and multicast will mostly get blocked by default.

Placing the Fire TV in its own VLAN by itself and trunking that VLAN to the router should eliminate most/all MDNS traffic. Of course it won't be able to see/discover other devices. Or you could leave it on VLAN 1 but move the devices that seem to generate a lot of MDNS traffic off to their own VLAN(s).

Yes using wireshark and testing out different configs is what I mean by "trial and error". To really attack and troubleshoot multicast you need to be able to see mroutes, igmp/PIM memberships (doubt the asus supports PIM, that's for routing multicast between routers or VLANs), etc right on the device itself. Whether that can be done on the Asus OS, not sure, probably not much if any.

Just keep in mind, more VLANs means more work for the router to do if those devices need to talk to each other. Now instead of switching traffic between two devices on the same VLAN, you're routing it between two different VLANs. Routing performance, even with hardware acceleration, is going to be much less than switching.

Even your cameras/blue iris being in an "isolated" VLAN, unless you don't send that VLAN to the router, will not behave differently than if they were just in VLAN 1 (other than potentially reducing the amount of MDNS/uPNP traffic "domino effect" since it won't see and interact with as many devices using those protocols). If you don't trunk the VLAN to the router, then you'll need a second interface off the NVR to actually access it from outside that VLAN, which I'm assuming you want to be able to do. Remember, the router has a switch built in, and that switch will properly handle traffic directly between two devices without sending it to the router/CPU.

You may need to take a device by device (or device type by device type) approach. For each device/device type, see if it is doing excessive amounts of uPNP or MDNS. If so, try to disable or block UPNP from that device. You probably can't disable MDNS but you can block it potentially (as long as you realize it will not be able to do discovery of other devices). Depending what you observe with each device you can decide whether isolating it is necessary or not.

But if at this point the Fire TV is the only thing having problems, I think you're way over thinking this. Either replace it with a newer model or put just that into its own VLAN and let everything else stay in the main VLAN. I suspect it just can't handle 4k decoding along with attempting to do other things.

You mentioned earlier that If your router is now having to route traffic between subnets, that is more load on the router (and less throughput potentially), in my special case the cameras and blue iris computer, they put more load on the router without VLAN, than the extra load I created on the router by creating VLAN and have video traffic bypass the router. I could have solved it maybe another way, I did notice the cameras have an option to disable multicast, UPNP checkbox in cameras seems to be already unchecked, but the camera firmware is buggy as hell, and they provide no firmware updates.

>>supports robocfg then configuring VLANs is very easy. I guess it is a non-HND router.

it is an HND router, it does not have robocfg command, I had to use commands vlanctl, ifconfig, brctl, nvram, ethswctl
the only VLAN I couldn't get it to work with vlanctl command is vlan1, so I have 2 wires going to the router, 1 untagged vlan1 that does not have ports that are on tagged ports as well, the 2nd wire for tagged traffic only, from the tagged interface, subinterfaces are created for each vlan that were created in the managed switches, subinterfaces are placed in new bridged interfaces in the router.


>>Traffic between two devices directly on the switch won't hit the router, even if they are in VLAN 1

since my camera traffic did make it to the router when they were on VLAN1 the only conclusion is they were
traffic of the following type broadcast, multicast, or if the destination was a MAC residing on the router.
So I accidently solved this problem by placing that traffic in a separate VLAN300. It did reduce context switches on the router by 33% that is net reduction.

>>Placing the Fire TV in its own VLAN by itself and trunking that VLAN to the router should eliminate most/all MDNS traffic.

Even before I bought the managed switches, I created a vlan with robocfg only inside the RT-ac68u only, without truncing the uplink cable it did help solve the stuttering, and strangely upgrading the USB network interface with 1 GBIT only worked partially in this setup, it reached 300mbits, the same interface in any other configuration (no vlans, or this new shared VLAN) the 1gbit interface reaches only 100mbit speed. It seems USB network interface speed is partially limited by the USB version supported by the Fire TV cube hardware probably 2, and something else i can't put my finger on.

>>Even your cameras/blue iris being in an "isolated" VLAN, unless you don't send that VLAN to the router,

Nothing else subscribes to the camera feed now, except on demand. Such as the blue-iris phone app, or my url https://somecustomcameraportal.freeddns.org which is sent through my router sslh listening at port 443, which recognizes the protocol and SNI name and sends it to the blue iris computer on an SSL port managed by NGINX which then forward it to the blue iris web port. Blue iris is NVR essentially. Blue-iris phone app also uses the external url or when at home uses the vlan1 ip address to reach it, as my phone connect to my main VLAN1 wifi. there is no cloud involved. The trunc cable going to the router does include the video vlan. Wireshark saw no extra context switches like the extra 2000 before. No extra UPNP traffic coming from vlan 300.
I did have to change my network setup a little, I did create a 2nd network virtual interface on the blue iris computer that is on vlan1 not because I had network issues, blue iris software security feature of not allowing access to video feed unless you were on the same subnet it was on (no way of disable it, a bug), with 2 interfaces on the computer blue iris recognized vlan1,vlan300 subnets as allowed Local Lan access. This 2nd network interface on vlan1 is leaking a small traffic -> 239.255.255.250 dest port 1900 but no more than 6000bps. Before the smallest leak was Mpps.
This is the only port on the managed switch that is on vlan1 untagged and vlan300 tagged for both network interfaces to get access to vlan1 and vlan300.

I believe I have left the network now in a better state will only be making minor tweaks to tune things only. I have cleaned up the RT-AX88u by factory reset, my JFFS is no longer very close to running out of space. I removed my USB hard drive, which I believed was unmounting and remounting causing the processes to dye during unmount, I am using a usb 3 stick for now, I have to read other forums for recommended USB SSD brand that works great with RT-AX88u. I have net reduced context switches by 33% on the router even though it is doing more work now with VLANS. I have a workaround that I'll shortly implement for the Fire TV cube to isolate it from everything but the Fire TV recast (antenna tuner), and I believe now WIFI calling was never related to my network issues it was a known iPhone issue (requires its on network setting reset). Also I have placed drop MDNS ebtables rule inside the Home Assistant raspberry pie to prevent future multicast feedback loop caused by it every couple of months, it slightly coincides when they have an upgrade ready. I have no evidence of other sources of MDNS that are higher than Home Assistant or Higher than UPNP traffic of what my cameras used to send.