Dual WAN Unbalanced

[ToeStub]

I'm seeing a large bias in favor of ppp0 (WAN) in my traffic logs (though, granted, I've only had them running for a week). It's not a speed/bandwidth issue -- if anything, the underused interface is slightly faster.

Interestingly, it seems to be about a 3:1 ratio, which I believe is the default setting, but I've double checked that the ratio is set to 1:1 on the configuration page.

Any other troubleshooting I can do?

 

[RMerlin]

Load balancing works at the connection level, not at the traffic level. If you connect to four different FTP site, then the connection will be distributed based on your balance configuration, regardless of how much traffic you transfer from each of them.

 

[ToeStub]

Load balancing works at the connection level, not at the traffic level. If you connect to four different FTP site, then the connection will be distributed based on your balance configuration, regardless of how much traffic you transfer from each of them.

I understand. Still, a consistent traffic bias in favor of one interface (when both have ~the same bandwidth) over a sustained period seems indicative of a problem IMO.

I looked at the iptables:

-A balance -m statistic --mode nth --every 2 -j CONNMARK --set-xmark 0x80000000/0xf0000000
-A balance -m statistic --mode nth --every 2 --packet 1 -j CONNMARK --set-xmark 0x90000000/0xf0000000

While I'm no expert, these parameter values conflict with guidance I'm seeing around the web.

Firstly, it seems the "--every" parameter looks like it's supposed to be an index, so I'm thinking it should not be "2" for both interfaces. Also the "--packet" parameter is typically either 0 or absent.

iptables -t nat -A PREROUTING -p tcp --dport 80 -m state --state NEW -m statistic --mode nth --every 2 --packet 0 -j DNAT --to-destination 10.0.0.3:80
iptables -t nat -A PREROUTING -p tcp --dport 80 -m state --state NEW -m statistic --mode nth --every 1 --packet 0 -j DNAT --to-destination 192.168.0.2:80

(Taken from https://stackoverflow.com/questions...ng-iptables-nth-mode-of-the-statistics-module.)

Or similar:

iptables -A PREROUTING -t nat -p tcp -d 192.168.1.1 --dport 27017 \
        -m statistic --mode nth --every 3 --packet 0              \
        -j DNAT --to-destination 10.0.0.2:1234
iptables -A PREROUTING -t nat -p tcp -d 192.168.1.1 --dport 27017 \
        -m statistic --mode nth --every 2 --packet 0              \
        -j DNAT --to-destination 10.0.0.3:1234
iptables -A PREROUTING -t nat -p tcp -d 192.168.1.1 --dport 27017 \
        -j DNAT --to-destination 10.0.0.4:1234

(Taken from https://scalingo.com/blog/iptables.)

The actual documentation for IPTABLES is not great.

I'm also not sure what the CONNMARKs are for? Edit: nevermind, I see they're to identify whether a connection is new or existing.

 

[ToeStub]

Ok, so it looks like the iptables rules were correct for the nth and packet parameters. There's conflicting information about how to set it up, with some people incrementing the value of --packet, and others incrementing the value of --every. Regardless, the hit counts match, and that's the important part.

That said, it looks like anything on ports 80 and 443 and 8443 are NOT balanced, and that's likely where the discrepancy was coming from.

[1]  -A PREROUTING -i br0 -m state --state NEW -j balance
[2]  -A balance -d 192.168.0.1/24 -j RETURN
[3]  -A balance -d 10.x.x.x/32 -j RETURN
[4]  -A balance -d 10.x.x.x/32 -j RETURN
[5]  -A balance -p tcp -m tcp --dport 443 -j RETURN
[6]  -A balance -p tcp -m tcp --dport 8443 -j RETURN
[7]  -A balance -p udp -m udp --dport 443 -j RETURN
[8]  -A balance -p udp -m udp --dport 80 -j RETURN
[9]  -A balance -m connmark --mark 0x80000000/0x80000000 -j RETURN
[10] -A balance -m state --state RELATED,ESTABLISHED -j RETURN
[11] -A balance -m statistic --mode nth --every 2 --packet 0 \
       -j CONNMARK --set-xmark 0x80000000/0xf0000000
[12] -A balance -m statistic --mode nth --every 2 --packet 1 \
       -j CONNMARK --set-xmark 0x90000000/0xf0000000

I removed rules 4 (dupe), 5, 6, 7, and 8.

I also added a final rule "-j RETURN" just to get a total count of the CONNMARKed packets with iptables -vL.

Finally, I also checked the marks in /proc/net/ip_conntrack before and after making changes. Before the change, they were ~2:1 in favor of 0x80000000 (ppp0) vs 0x90000000 (ppp1). Now they're roughly the same.

If anyone knows a good reason to keep rules 5, 6, 7, and 8, please let me know.