SNMP byte counters on an AC1900P (RT-AC68U)

[JefferMC]

I have a ASUS AC1900P running Merlin v386.5_2

I've been setting up a Graphite monitoring system for my little network. I'm screen scraping my AT&T provided Gateway for interface counters but I wanted to also verify them by getting the byte counts from the ASUS router as well. I discovered the MIB 1.3.6.1.2.1.2.2.1.16.x (out bytes) and 1.3.6.1.2.1.2.2.1.10.x (in bytes) where (x) is the interface id. By reading this forum and inspecting the numbers, I have decided that:

4 (eth0) is Primary WAN
6 (eth1) is the Wi-Fi 2.4 GHz band
7 (eth2) is the Wi-Fi 5 GHz band.

In general, these seem to work as I expect. However, sometimes eth0 seems a bit, er, weird if it really is the WAN interface. Since I have an Ethernet cable between the ASUS's WAN port and a LAN port of the AT&T Gateway, I expect to see mirror images of traffic (i.e. what one shows as in, the other shows as out, and vice versa). And, for the most part, I do. However, occasionally, the ASUS shows pretty much equivalent numbers for out as it's showing for in, but that traffic only appears as "out" on the Gateway's interface, so ASUS seems to be sending bytes out, equal to the bytes in, but they're not making it to the Gateway.

What this might be telling me is that eth0 is not, or not simply, WAN traffic.

Traffic also appears on interfaces 1 (lo, obviously not what I'm looking for), 8 (vlan1) and 12 (br0). My conclusion was the vlan1 would be the traffic to the 4 Ethernet ports and br0 would be the sum of 6, 7 and 8 (though I've not seen that bourn out).

Anybody got any decent ideas?

 

[drinkingbird]

The gateway is showing the same as the asus, just showing everything as "in". If anything looks like you're reading the wrong MIBs on the gateway, maybe total instead of in/out or something.

 

[JefferMC]

I'm not reading any MIBs on the Gateway at all, because I have to screen scrape to get the byte counters. Perhaps I should have put some callouts in the comparison above, but you can see periods where Gateway Out = ASUS In, and Gateway In = ASUS Out and they track perfectly. But there are other sections where the ASUS out is tracking EXACTLY the ASUS in line AND the Gateway Out line, which is why I'm looking hard at the ASUS out metric.

I could certainly have the wrong MIB or the wrong interface...

 

[drinkingbird]

I'm not reading any MIBs on the Gateway at all, because I have to screen scrape to get the byte counters. Perhaps I should have put some callouts in the comparison above, but you can see periods where Gateway Out = ASUS In, and Gateway In = ASUS Out and they track perfectly. But there are other sections where the ASUS out is tracking EXACTLY the ASUS in line AND the Gateway Out line, which is why I'm looking hard at the ASUS out metric.

I could certainly have the wrong MIB or the wrong interface...

Do a speed test (or better yet upload a very large file to onedrive/google drive/etc then download it) when nothing else is active and see how the upload and download line up, should make it pretty obvious. But considering the spikes track perfectly but are being counted the same on the gateway and differently on the Asus seems like the gateway is telling you total bytes, maybe your parsing is off by a line (blue line on the gateway could be packets or something instead of bytes).

Bear in mind hardware acceleration on the Asus is known to cause some missed data on Traffic Monitor, not sure if it affects SNMP or not. Theoretically it should not. eth0 on the 1900P should only be reporting WAN traffic. Technically eth0 is the interface to the 5 port switch (4 ports LAN under VLAN1 and 1 port WAN under VLAN2) but the way Asus has it set up, LAN traffic will show under the vlan1 interface, leaving only VLAN2 (WAN) traffic under the main interface counter. eth1 and 2 are the wireless radios. Then they create wl0.x and wl1.x for guest wireless interfaces, which are just virtual/software interfaces hanging off eth1 and 2.

To make it even more confusing, when you enable guest wireless 1 on the 386 code base, it creates eth0.501 and eth0.502 which are LAN sub-interfaces, not WAN and should not be counted against the main eth0 I don't believe.

From what I've gathered on my regular 1900, this is how it is laid out (should be the same for all the AC68 variants). Not sure if this is 100% but I think so.

eth0=interface between the 5 port switch and router. Asus moves all LAN traffic (ports 1-4) to the virtual vlan1 interface so the only thing remaining/counted here should be WAN/VLAN2 traffic. They leave vlan2 untagged on eth0 and do not remap it to a virtual router interface, and the WAN IP is assigned directly to eth0 (LAN IP is on br0 which includes vlan1). I guess in other words they strip all LAN traffic off eth0 to make it a pseudo dedicated WAN interface.
eth1=2.4ghz radio
eth2=5ghz radio
vlan1=wired LAN (ports 1-4) which is remapped/redirected from eth0. Essentially eth0.1 but without tagging, the tagging is done at the switch instead of the router.

wl0.1=2.4ghz guest wireless 1 (which on 386 code is treated differently than GW2 and 3 and gets associated with VLAN 501 and its own 192.168.101.0/24 subnet which is assigned to br1)
wl1.1=5ghz guest wireless 1 (same, but VLAN 502 and 192.168.102.0/24 on br2)

wl0.2=2.4ghz guest wireless 2 - part of main LAN subnet/VLAN 1/br1 but firewall rules are used to isolate if that option is selected.
wl1.2=5ghz guest wireless 2- part of main LAN subnet/VLAN 1/br1 but firewall rules are used to isolate if that option is selected.

wl0.3=2.4ghz guest wireless 3- part of main LAN subnet/VLAN 1/br1 but firewall rules are used to isolate if that option is selected.
wl1.3=5ghz guest wireless 3- part of main LAN subnet/VLAN 1/br1 but firewall rules are used to isolate if that option is selected.

eth0.501 = 2.4ghz guest wireless 1 (if enabled) wired backhaul for aimesh
eth0.502 = 5ghz guest wireless 1 (if enabled) wired backhaul for aimesh
These would also be used if you assigned a physical port(s) into these vlans which I have
eth1.501/502 = same but for 2.4ghz wireless backhaul on aimesh
eth2.501/502 = same but for 5ghz wireless backhaul on aimesh

br0=all LAN (wired and wireless) traffic plus Guest Wireless 2 and 3 (vlan1+eth1+eth2+wl0.2/wl1.2+wl0.3/wl1.3). This is because it uses iptables to segment guest wireless 2 and 3 and shares the same subnet. Basically br0 is all traffic except WAN and Guest Wireless 1 (on code base 386 anyway).
br1=2.4ghz guest wireless 1 including the backhauls (wl0.1+eth0.501+eth1.501+eth2.501)
br2=5ghz guest wireless 1 including the backhauls (wl1.1+eth0.502+eth1.502+eth2.502)
So the bridge interface counters aren't terribly useful, unless you aren't using guest 2 or 3 and want to see the sum of all non-guest LAN traffic which would be br0 in that case. But still probably better to get that from main interfaces and sum together when needed.

I don't believe the switch ports will have any MIBs but you can try walking it. Port 0 is the WAN.

What makes it even weirder is that they tag vlan 1 to the CPU/router and leave vlan 2 (wan) untagged to the CPU/router. VLAN 1 should never be tagged but I guess there was some reason they chose to do it that way, basically making the native VLAN of eth0 2 instead of 1, but still having 1 present.

How often are you reading the counters? Anything more than 10 seconds these days is somewhat misleading/not useful. 1 second would be nice but probably not practical/possible with the hardware we're talking about.

EDIT for future searches -
NOTE 1 - If you do not enable Guest Wireless 1 on 386 code base - there will be no 50x interfaces or VLANs. br1 and br2 are either not created or have nothing under them (don't remember which). There is no wl0.1 and wl1.1 obviously either.
NOTE 2 - If you enable GW1 with "access intranet" enabled on 386 code base - same as Note 1 above (no 50x, no br1/2) but wl0.1 and wl1.1 are created and placed under br0 like all the other interfaces. No firewall rules needed unlike the other guests since you've said to allow intranet access.

 

[JefferMC]

I'm reading byte counters every 10 seconds, both here and on the Gateway (which takes 2.5 seconds to paint that page... grrr).

I did the download test from both a wired PC and one 5 minutes later on the 5 GHz Wi-FI. Here are the charts of the Gateway WAN, the Gateway LAN to the ASUS, the ASUS eth0 (WAN), eth1 (2.4) and eth2 (5):

You can see the reversal between the Gateway WAN and its LAN port on both tests. You can see the 5 GHz test reverse at each step from in, to out, to in and back to out. But the wired test... well, the in peak got masked by an out peak that really shouldn't exist.

Seems like there's a bug in the output counter collection of the eth0.

 

[drinkingbird]

I'm reading byte counters every 10 seconds, both here and on the Gateway (which takes 2.5 seconds to paint that page... grrr).

I did the download test from both a wired PC and one 5 minutes later on the 5 GHz Wi-FI. Here are the charts of the Gateway WAN, the Gateway LAN to the ASUS, the ASUS eth0 (WAN), eth1 (2.4) and eth2 (5):

View attachment 48943

You can see the reversal between the Gateway WAN and its LAN port on both tests. You can see the 5 GHz test reverse at each step from in, to out, to in and back to out. But the wired test... well, the in peak got masked by an out peak that really shouldn't exist.

Seems like there's a bug in the output counter collection of the eth0.

Well triple check (use the MAC address) to ensure you are definitely collecting the main eth0 interface. I guess it is possible something extra is getting counted against it, maybe it is showing both VLAN1 and 2 traffic in its counters. If that is the case you probably can't isolate it (short of also collecting VLAN1 and subtracting it) so you'll just need to rely on the AT&T interface. But if you walk the Asus you might find some other counters that are useful.

It could also just be a matter of how often each device updates. SNMP interface counters are nearly instant, the web GUI may be delayed, and may be averaging. So your blue line is just getting hidden behind the wider, more accurate green line of the Asus. In fact, now that I look at it, that seems to be what may be happening. AT&T line is higher and lasts less time, Asus line is lower and longer duration. Move your blue line to the front on the asus graph and see if it mostly lines up.

When I used to do a lot of SNMP stuff I found it better to have the download be a solid green area graph and the upload be a superimposed blue or red line (in front of the green).

Or, like I mentioned, Hardware acceleration could be throwing off the counters. But usually then you see less traffic counted, not more, and I don't believe it would cause it to "change direction".

 

[JefferMC]

I decided to graph the other counters and see what I saw and to graph the difference between what I expected (Gateway eth0 in) and what I was getting from the ASUS eth0 out.

Bottom center is the difference between them. Notice how nicely it correlates to vlan1 out (upper right, green). So... the difference between them (eth0.out - vlan1.out) is plotted in bottom right (green) along with the expected (blue). That's more than acceptably close.

My opinion is that octetsOut.Eth0 is inflated by octetsOut.vlan1 . I think our state's Comptroller General (Google Richard Eckstrom) has been in this firmware.

 

[drinkingbird]

I decided to graph the other counters and see what I saw and to graph the difference between what I expected (Gateway eth0 in) and what I was getting from the ASUS eth0 out.

View attachment 48977

Bottom center is the difference between them. Notice how nicely it correlates to vlan1 out (upper right, green). So... the difference between them (eth0.out - vlan1.out) is plotted in bottom right (green) along with the expected (blue). That's more than acceptably close.

My opinion is that octetsOut.Eth0 is inflated by octetsOut.vlan1 . I think our state's Comptroller General (Google Richard Eckstrom) has been in this firmware.

So potentially subtract VLAN1 from eth0 and that may be what you need. That should only leave VLAN2 (WAN) traffic, which is counted against the main eth0 interface (I suspect that even though VLAN1 is a separate virtual interface it is falling under the main eth0 counter too, since technically it passes via that interface).

May have to toy around with other stuff too, do some testing and make sure nothing else gets counted there. If you aren't doing aimesh then shouldn't have to worry about VLAN 501 and 502, in theory.

 

[ColinTaylor]

@JefferMC You've been talking about eth0, can you confirm that your router doesn't have a vlan2 interface (with ifconfig)? The reason being that when hardware acceleration is disabled the WAN interface is eth0. When hardware acceleration is enabled the WAN interface becomes vlan2 and it is no longer possible to collect accurate traffic stats.

 

[drinkingbird]

@JefferMC You've been talking about eth0, can you confirm that your router doesn't have a vlan2 interface (with ifconfig)? The reason being that when hardware acceleration is disabled the WAN interface is eth0. When hardware acceleration is enabled the WAN interface becomes vlan2 and it is no longer possible to collect accurate traffic stats.

I have acceleration enabled and there is no vlan2 interface. Maybe that is only on HND routers? WAN does use VLAN 2 but it is untagged and uses the main eth0 (vlan 1 traffic is stripped off to a VLAN1 interface).

Theoretically subtracting the VLAN1 traffic from eth0 should work (most monitoring programs can do this), but only way to know is try it. The only other thing that would fall under eth0 is guest wireless 1 ethernet backhaul which I don't think is in use here.

 

[ColinTaylor]

I have acceleration enabled and there is no vlan2 interface. Maybe that is only on HND routers? WAN does use VLAN 2 but it is untagged and uses the main eth0 (vlan 1 traffic is stripped off to a VLAN1 interface).

Theoretically subtracting the VLAN1 traffic from eth0 should work (most monitoring programs can do this), but only way to know is try it. The only other thing that would fall under eth0 is guest wireless 1 ethernet backhaul which I don't think is in use here.

I was talking about my own RT-AC68U. The eth0 interface always exists but with hardware acceleration enabled a new vlan2 interface is created and becomes the WAN interface. Maybe your model is different than mine. Check the WAN interface with ifconfig (it will have the WAN IP) and nvram get wan0_ifname.

If the OP only has an eth0 interface then it really doesn't matter.

 

[drinkingbird]

I was talking about my own RT-AC68U. The eth0 interface always exists but with hardware acceleration enabled a new vlan2 interface is created and becomes the WAN interface. Maybe your model is different than mine. Check the WAN interface with ifconfig (it will have the WAN IP) and nvram get wan0_ifname.

If the OP only has an eth0 interface then it really doesn't matter.

Mine is RT-AC1900, definitely no VLAN2 interface under ifconfig. Odd, they should be identical, the 1900 should just be 68U rev C, and OPs 1900P should be identical as well other than CPU clock (same as the 68P). WAN IP is right on ETH0 main interface on mine, LAN IP is in BR0 (which includes VLAN1).

ifconfig | grep -i vlan
vlan1 Link encap:Ethernet HWaddr D0:17:C2:E2:7B:00

nvram show | grep -i vlan2
vlan2ports=0 5u
vlan2hwname=et0
eth_ifnames=vlan2

My GUI reports CTF enabled and I can push about 350mbit at 60% CPU with trend micro enabled so it is definitely HW accelerated.

I do have GW1 enabled with the 501/502 vlans but prior to that I don't recall anything being different, other than the new eth0.501 and 502 being added.

Running 386.7_2.

 

[ColinTaylor]

Mine is RT-AC1900, definitely no VLAN2 interface under ifconfig. Odd, they should be identical, the 1900 should just be 68U rev C, and OPs 1900P should be identical as well other than CPU clock (same as the 68P). WAN IP is right on ETH0 main interface on mine, LAN IP is in BR0 (which includes VLAN1).

ifconfig | grep -i vlan
vlan1 Link encap:Ethernet HWaddr D0:17:C2:E2:7B:00

nvram show | grep -i vlan2
vlan2ports=0 5u
vlan2hwname=et0
eth_ifnames=vlan2

My GUI reports CTF enabled and I can push about 350mbit at 60% CPU with trend micro enabled so it is definitely HW accelerated.

I do have GW1 enabled with the 501/502 vlans but prior to that I don't recall anything being different, other than the new eth0.501 and 502 being added.

Running 386.7_2.

There are some slight variations in the models. I don't know the specific details but I have CTF and FA enabled. If you're seeing that CPU load I would say hardware acceleration isn't fully enabled. I hit my ISP's limit of 660 Mbps with almost zero CPU (I didn't enable any TrendMicro stuff).

ipv6_ifname=vlan2
vlan2hwname=et0
vlan2ports=0 5
wan0_ifname=vlan2
wan_ifnames=vlan2
wandevs=vlan2

This was using John's LTS firmware based on an older Merlin branch. Maybe the current Asus/Merlin firmware is different.

 

[JefferMC]

ifconfig shows no VLAN2. I have a VLAN2 counter, but the OpStatus is down, and the counter sits at 0. I assumed that if I turned on IPTV, then this would be multicast traffic on the ports that were IPTV and LAN1 would lose the multicast traffic. I've got the traffic measuring by IP feature turned on which won't let me do hardware acceleration. I'm about to decide that it isn't useful, though; there's a lot it doesn't seem to capture and the granularity is crap.

And, yes, I can record the counter and display the difference in charts (as I am doing in the screenshots above), but it's odd that there is not just a simple WAN interface counter.

 

[drinkingbird]

There are some slight variations in the models. I don't know the specific details but I have CTF and FA enabled. If you're seeing that CPU load I would say hardware acceleration isn't fully enabled. I hit my ISP's limit of 660 Mbps with almost zero CPU (I didn't enable any TrendMicro stuff).

ipv6_ifname=vlan2
vlan2hwname=et0
vlan2ports=0 5
wan0_ifname=vlan2
wan_ifnames=vlan2
wandevs=vlan2

This was using John's LTS firmware based on an older Merlin branch. Maybe the current Asus/Merlin firmware is different.

Yeah without Trend I can get much higher throughput with barely any CPU. With acceleration disabled I get under 100M.

 

[drinkingbird]

ifconfig shows no VLAN2. I have a VLAN2 counter, but the OpStatus is down, and the counter sits at 0. I assumed that if I turned on IPTV, then this would be multicast traffic on the ports that were IPTV and LAN1 would lose the multicast traffic. I've got the traffic measuring by IP feature turned on which won't let me do hardware acceleration. I'm about to decide that it isn't useful, though; there's a lot it doesn't seem to capture and the granularity is crap.

And, yes, I can record the counter and display the difference in charts (as I am doing in the screenshots above), but it's odd that there is not just a simple WAN interface counter.

It's because it runs through the 5 port switch and shares that switch for both LAN and WAN (all ports are part of ETH0). It would seem to make more sense to have a VLAN2 virtual interface but it is probably related to having to support that VLAN for IPTV etc so they have to work around it.

 

[JefferMC]

It's because it runs through the 5 port switch and shares that switch for both LAN and WAN

To me that seems a strange architecture for a ROUTER.

 

[drinkingbird]

To me that seems a strange architecture for a ROUTER.

These home routers are far better described as L3 switches than routers. The routing functionality is layered on top. A dedicated L3 WAN port off the SOC would add cost.

 

[JefferMC]

And would hurt performance (or would gut Hardware Acceleration). I would at least think the WAN port would be a dedicated VLAN, though.

 

[drinkingbird]

And would hurt performance (or would gut Hardware Acceleration). I would at least think the WAN port would be a dedicated VLAN, though.

It is, vlan2. But it isn't tagged and thus no logical interface for it, as that would not work with most ISPs.

If the switch supported SNMP you'd be golden, but I don't think it does. Have you walked every common OID to check?

If it was really important you could find the cheapest switch with SNMP support and toss it inline. But I think etho-vlan1 is what you're looking for. Have you compared that to the ISP box yet to see if it lines up?