New hEX

[tehfish]

I have done loads of debugging on my current ERL, it seems im getting the lan -> wan spikes when a new connection is made, somehow that "stalls" network traffic for 25-50ms before it then picks up, giving a large spike in network "feel" when gaming etc.

In this case it seems that the offloading mechanism is not working properly on the ERL I have (And one other I tested just now) - Im going to test the ERX monday and also fire up a pfsense router for a few hours to give that a go, I would really hate to buy a new router and then find out that it is simply my own network that are messed up, which I don't really think it is though as ping times to other machines on the network and to the lan side of the router is always <1ms.

It seems that mikrotik can selectively use hardware offload for only some certain dataflows? Is that determined on port or?

 

[System Error Message]

I have done loads of debugging on my current ERL, it seems im getting the lan -> wan spikes when a new connection is made, somehow that "stalls" network traffic for 25-50ms before it then picks up, giving a large spike in network "feel" when gaming etc.

In this case it seems that the offloading mechanism is not working properly on the ERL I have (And one other I tested just now) - Im going to test the ERX monday and also fire up a pfsense router for a few hours to give that a go, I would really hate to buy a new router and then find out that it is simply my own network that are messed up, which I don't really think it is though as ping times to other machines on the network and to the lan side of the router is always <1ms.

It seems that mikrotik can selectively use hardware offload for only some certain dataflows? Is that determined on port or?

you determine it in mangle, you can selectively use many different things. Simply go to demo.mt.lv , if you look under mangle, try to create a new rule and you will see all the options available. However for hardware acceleration the catch is to mark the connection.
Than in filter simply fasttrack the connection with the mark and state of established.

 

[tehfish]

Im gonna order up one of the rb750gr3 - Just got an ERX installed today, which provided even worse routing performance with even higher ping spikes and random issues..

Im hoping the routerboard / routeros is a better fit for my needs.

 

[System Error Message]

Im gonna order up one of the rb750gr3 - Just got an ERX installed today, which provided even worse routing performance with even higher ping spikes and random issues..

Im hoping the routerboard / routeros is a better fit for my needs.

dont forget to download and install dude on it too, its a decent free network monitoring tool, which is basically the only software you can install and run on mikrotik routers without hacking them. It can do a lot more including show whos on your network just like premium consumer routers do but better and more complicated.

 

[paraplu]

My Hex v3 died the previous night. Can't reset anymore. Weird.

Managed to restore my Hex some time back. And back on Hex again instead of the loud CCR ;-)

 

[paraplu]

Interesting note. Parallel programming has been a challenge for a long time. People as good as Mikrotik seem can't put the many cores in CCR's to efficient use. On the hEX, how evenly among the four 'cores' do you see under different usage scenario like speedtest and ipsec vpn? In ER-X as I posted in another forum, UBNT can't make use of all four 'cores' for in IPsec for one direction (..the other direction is okay).

Another minus for CCR is that its FW re-orders IPsec packets if HW accelerator is enabled. The packet re-ordering upsets majority of PC users.

The CCR out-of-order bug has been resolved a while back. VPN for windows-clients seems to work fine now. I got a hefty 250Mbps throughput with single L2TP/IPSEC connection towards the CCR.

Regarding the Hex: normal (fast-tracked established) usage divides the load evenly between all 4 threads. Around 50% CPU usage with full 500mbps up/down sequential tests (speedtest). Suggesting room for more. Gbps might be possible.

For L2TP/IPSEC (aes-128cbc) with windows, ios and android devices; hitting 180Mbps as a limit. Most of all Ethernet load on a single core, while Crypto offload activity is on another core though not a bottleneck. Apparently fast-track does not work in this VPN scenario.
Tried other methods (direct IPSEC tunneling without L2TP) but was unable to get this to work with Windows clients, in combination with Shrew Soft. OpenVPN, cough, I have not tried, knowing that this would not hit more than 50mbps.

 

[System Error Message]

Managed to restore my Hex some time back. And back on Hex again instead of the loud CCR ;-)

my CCR1036 is quieter than the ERPRO . I did modify it though.

CCR out of order bug was already fixed but mikrotik can do quite a lot with hardware acceleration but if you do need some complicated QoS, thats where the CCR will help you.

 

[kvic]

Regarding the Hex: normal (fast-tracked established) usage divides the load evenly between all 4 threads. Around 50% CPU usage with full 500mbps up/down sequential tests (speedtest). Suggesting room for more. Gbps might be possible.

Can you share the content of "/proc/interrupts" from your hEX after up for a few days with moderate usage?

For L2TP/IPSEC (aes-128cbc) with windows, ios and android devices; hitting 180Mbps as a limit. Most of all Ethernet load on a single core, while Crypto offload activity is on another core though not a bottleneck.

That's around the 244Mbit/s I got for one stream on ERX. The other direction is about 127Mbit/s. The detail of my tests are here done last year. I heard hEXr3 doesn't have this asymmetric phenomenon.

Digress a bit... I reported my observation to UBNT and heard no response. This and their marketing of ERX as "gigabit router" (which is not true as they crippled the HW from 1000/1000 to 500/500 in FW) leave me with a poor impression about this company.

 

[kvic]

Im gonna order up one of the rb750gr3 - Just got an ERX installed today, which provided even worse routing performance with even higher ping spikes and random issues..

Below is one spike event on my ERX captured by SmokePing (running on my RT-AC56U) where the green dots substantially deviated from its norm. When the spike happened, the WAN was about 80% saturated. The 2.5ms host is very close to me. The 35ms host is about 2500km away.

I think such spikes are not abnormal when WAN is moderately saturated. Keep us posted on how hEXr3 may have solved your spikes though.

 

[tehfish]

The thing is i am seeing the spikes even if the load is below 1% - I am fairly certain that this is ISP related though as I have actually gotten them to add a few more IPs so that im not limited to a single device, during that configuration directly on the fiber modem im seeing the same behaviour..

In either case, im looking forward to using the hEXr3, it appears to be a great little device - I just downgraded from 1000/1000 to 500/500 which it should handle without problems from what I can see, but I might need some help configuring it properly though

 

[System Error Message]

The thing is i am seeing the spikes even if the load is below 1% - I am fairly certain that this is ISP related though as I have actually gotten them to add a few more IPs so that im not limited to a single device, during that configuration directly on the fiber modem im seeing the same behaviour..

In either case, im looking forward to using the hEXr3, it appears to be a great little device - I just downgraded from 1000/1000 to 500/500 which it should handle without problems from what I can see, but I might need some help configuring it properly though

it will handle 1Gb/s just fine unless you use too much mangle.

 

[tehfish]

Im not sure how to set things up atm, right now I really only have the need to do port forwarding, nothing more fancy than that so im guessing i should be fine?

 

[System Error Message]

Im not sure how to set things up atm, right now I really only have the need to do port forwarding, nothing more fancy than that so im guessing i should be fine?

under NAT, dstNAT simply redirect the port and ip.

 

[tehfish]

under NAT, dstNAT simply redirect the port and ip.

Neat, thanks.

Is there no need to do firewall configurations? Is it automatically "hardware accelerated" in that case?

 

[System Error Message]

Neat, thanks.

Is there no need to do firewall configurations? Is it automatically "hardware accelerated" in that case?

there still is to filter traffic before you accelerate them. It will be hardware accelerated.

I find that only UDP and TCP are hardware accelerated though so dont expect to be able to forward 1Gb/s of pings.
Just 1 blanket rule in filter to accelerate established connections will do. You need a rule however to drop invalid connections and to deal with any traffic that doesnt get accelerated.

 

[sfx2000]

Below is one spike event on my ERX captured by SmokePing (running on my RT-AC56U) where the green dots substantially deviated from its norm. When the spike happened, the WAN was about 80% saturated. The 2.5ms host is very close to me. The 35ms host is about 2500km away.

I think such spikes are not abnormal when WAN is moderately saturated. Keep us posted on how hEXr3 may have solved your spikes though.

Normally when I see a spike like that, and it correlates across hosts - something upstream, not local generally...

 

[sfx2000]

For L2TP/IPSEC (aes-128cbc) with windows, ios and android devices; hitting 180Mbps as a limit. Most of all Ethernet load on a single core, while Crypto offload activity is on another core though not a bottleneck. Apparently fast-track does not work in this VPN scenario.
Tried other methods (direct IPSEC tunneling without L2TP) but was unable to get this to work with Windows clients, in combination with Shrew Soft. OpenVPN, cough, I have not tried, knowing that this would not hit more than 50mbps.

The L2TP/IPsec numbers aren't that bad, considering the resources at hand...

The OpenVPN numbers, not bad, but also reflects the context jumps that OVPN has to do going from Kernel/User/Kernel space...

 

[paraplu]

Can you share the content of "/proc/interrupts" from your hEX after up for a few days with moderate usage?
.

Unfortunately RouterOS does not provide access to the underlaying Linux filesystem. Not possible.

 

[kvic]

Unfortunately RouterOS does not provide access to the underlaying Linux filesystem. Not possible.

That's sad news. Not that it's "locked" but we can't get the info out of it. I'm sure @System Error Message has instructions on how to read /proc/interrupts..

 

[System Error Message]

That's sad news. Not that it's "locked" but we can't get the info out of it. I'm sure @System Error Message has instructions on how to read /proc/interrupts..

Normally you cant get into the underlaying linux file system but there is a way which is to boot the router using openwrt, this will expose the OS and let you tweak and get access to dev shell which will let you install things like busybox and other things. I've heard someone using the CCR1036 as some sort of media encoding server to do real time encoding.