Mikrotik CCR review?

as the modern small business is growing into Gbps speeds, with plenty of special requirements, a review of highend software-based routers, like mikrotik (CCR) offers, would be very much appreciated.

/opinion
small audience here for such an exotic beast.
That company (w/80 employees) is on my forever black list for nonchalantly shipping products with beau coup bugs, leaving me at a customer's field site in nowhere-land with no hope. Tinker-things, not products.
/endopinion

http://www.mikrotik.com/aboutus

Im working on making a CCR review but due to time and travel constraints would have to wait.
The CCR is quite a beast in throughput so if you really need performance it is a great option for the features. I have a CCR1036 with 2 SFP+ and is less buggy than ubiquiti's edgerouter. I may even get a the ubiquiti edgerouter with the fastest CPU to compare which i know will do 2Gb/s NAT.

Support from mikrotik is only via email because they rely on "certified" technicians to do it for them. What really annoys me is that many guides around arent practical for example a guide on defeating the NAT filter for mikrotik is to increase the TTL (or use TTL detection) but NAT detection is way more sophisticated than that which routerOS can beat but there isnt yet and example and i havent had time to figure out how myself. I know it requires a mixture of layer 2 and layer 3 stuff and knowing the sort of NAT detection used by the ISP.

Not trying to be racist or offensive but the guides from the south west part of ASIA and the west of it are unreliable. You need an actual expert to fully utilise mikrotik.

When it comes to choosing a router a lot of people actually choose a router that does half their throughput if it is symmetrical. A symmetrical gigabit internet requires 2Gb/s of NAT throughput since the total throughput needed is download bandwidth + upload bandwidth.

There is a test done on the mikrotik CCR that each core can do 2Gb/s of NAT on a port connected to the CPU.

I look at boxes like this, and it's similar to the Microtik/Routerboards...

http://store.pfsense.org/SG-8860/

These devices are probably at the very high end of the SNB universe, and the skills needs to fully exploit them without having to fall back to the boards - it's a big ask for what is for most folks a non-issue, as the general OEM's focused on the home/small biz market do a job that is - "good enough" - these devices go well beyond what folks might do with a *WRT distro on a consumer box.

sfx

Well compared to what a consumer box has if you need gigabit throughput there is no replacement for a proper router when in a multi user environment, whether its a tiny data center, an office, a business or even some similar environment. While it does cost a lot more than a consumer router you do get more than what a consumer router gives and more value for money in terms of price/performance.

For example the CCR1036 highest port capacity is 56Gb/s (all CPU connected, no switch) meaning it will need 28Gb/s forwarding capacity to max it out. Knowing that each TILE core does 2Gb/s of software NAT it has more than enough CPU power to max out the ports so it will do 28Gb/s. Taking the maximum price of $1200 will give you $42.857 per Gb/s not to mention that the it has extra CPU power so it will still do wirespeed after adding some configurations which gives you even better value.

Now compare this price/performance for software NAT to the cheapest consumer router's price/performance for hardware NAT and you will find none that can beat its price/performance even using software NAT. So even in an unfair comparison the enterprise router still wins out making it a much better choice for a larger multi user environment compared to a consumer box. Even the cisco RV is considered a consumer box and relies on hardware NAT.

The CCR series does have encryption acceleration so it will do VPN very fast. Each core does 300Mb/s of PPTP and L2TP/IPSEC. I actually measured the PPTP throughput myself. The bane of mikrotik is when it has to perform NAT when traffic is coming from a switch chip.

well, speeds and $850 price is n/a to most users of this forum!
And an MSEE/MSCS is needed to run one of these.

Sorry for the double posting but this isnt related.
So yes i agree we need a review on mikrotik products and there arent many here with access to mikrotik hardware. I myself have 3 of them which is a MIPS based router, a CRS switch and a CCR and im currently away for summer so it'll be another month before i manage to get back and finish the review. I also dont have enough working systems to do a proper speed test on CCR.

I dont have any wireless products from mikrotik. If you dont mind waiting a month you will find a review on the CCR1036 and CRS226.

But if you want a review on routerOS go to demo.mt.lv which is a routerOS on one of their low cost MIPS based routers. Except for hardware specifics like LCD and switch and metarouter the routerOS across all routerboards and x86 are the same so it will give you an insight into what it can do. Performance wise the CCR is more than capable to have multiple gigabit internet providers and the kind of control and features you need for a big network.

In essence the CCR is
Very fast for handling multiple gigabit ISPs
L6 routerOS license so theres no restrictions
full of enterprise features
hardware accelerated encryption for fast VPN (even faster than the cisco RV)
per core load balancing. Each connection/tunnel can only use 1 core which prevents 1 traffic from using all the capacity but also means you need to load balance over multiple tunnels for more throughput or that the max 1 user can get from VPN is 1 core's worth usage.
Very loud except for the CCR1009.....PC since the cooling fans are 1U deep.
Good power use for throughput (20W for CCR1009, 60W for CCR1036)
idle power is 2/3 of max
firmware has already matured that it has less bugs than a stable consumer router.
Will handle thousands of firewall and QoS entries, IP addresses and such with ease (look at a recent post of mine and a screenshot showing more than 6000 IP entries).

Just got myself the entry-level 8-port, 9-core
CCR1009-8G-1S-1S+
This to replace my ERL as it crashed again for the third consecutive time; asking again for a new internal usb storage. Gave up on ERL, after 3 years of usage, and 2 internal storage replacements. The ERL is definitely a good product for its price. Just doesn't fit my requirements on durability and QOS/VPN performance. For those that use this ERL device: get a ups battery upfront!

My experiences so far with the Mikrotik:
- 4 ports are connected to a switch cpu and therefore have a limited throughput of 1gbps combined. When these 4 ports are used as l2 switch (vlan supported) you get a wire speed switch. Think of the Mikrotik as a 4 port switch combined with 4 individual router ports, with l3 bridge option between these two parts. Or you can use all 8 ports through software individually keeping in mind that aggregated speed is limited to 1/1gbps for the first 4 ports.
- PPPoE is currently (v6.31) not supported for hardware offloading
- single PPPoE isp uplink port gave me 100% single core load under full 500/500 load. Got a bit scared about this one but read below.
- after adding some smart queues all load is now distributed between the 9 cores. Not more than max 10% usage per core when under full 500/500 load.
- perfect 500/500 flatline performance fully in software with dozens of firewall filters and mangle rules, and multiple queues in place. No fast offload used. Still haven't found the limits.
- ping/latency from other devices to wan remains below 20ms when under full load. Seems to work well against buffer bloat thanks to their queuing algorithms.
- power usage about 17w. Noisy fans, but this is a rackmount device not a desktop room thing. Otherwise get the +PC version.
- takes a while to get used to their configuration methods. The ERL is easier to setup. Not for the usual customer to configure. BUT it's all according standards and if you are a pro you would have no problems to set this thing up. Takes a couple of days though.
- not possible to get to the root OS. Therefore not possible to add third party products or to use simple Linux scripting.

Just got started with this thing and still have to check VPN, Adblock etc... So far; impressive device, but indeed not for the mainstream.

there isnt adblock in this device. To use adblock you will need to use its DNS static entry with DNS hijacking (add a dst-nat rule in NAT to redirect TCP and UDP of port 53 to the router). This will make it work with google chrome too because google chrome refuses to use the gateway supplied DNS. There might be a script to convert adblock plus list to DNS entries.

VPN is easy to set up PPPOE and PPTP but harder to set up the other types. However for best throughput for a more secure VPN use L2TP with IPSEC as it has hardware acceleration for that and will give you WAN speeds for your VPN.

Ive managed to get hardware acceleration working for PPPOE and VLANs but on the IP settings it says no packets have been considered for it which im not sure if it is a bug. In IP settings it shows hardware acceleration enabled and no packets gone through but in firewall a lot of packets went into the forwarding rule.

The main 2 things not present in routerOS is adblock (L7 http) and anti virus.

Well, there's some aspects of Microtik (and others).

Might want to reach out to thiggins directly, and contribute a review...

Consumer router are now lagging internet speeds for home use. If you want to fully utilize the fastest home internet connections we need faster routers. Right now you have few selections, mainly PC based.

[QUOTE="

The main 2 things not present in routerOS is adblock (L7 http) and anti virus.[/QUOTE]

If you want layer 7 tools you should be looking at UTM firewalls. They are much superior to routers for layer 7. Routers were really designed for layer 3. Some perform more.

coxhaus said:

If you want layer 7 tools you should be looking at UTM firewalls. They are much superior to routers for layer 7. Routers were really designed for layer 3. Some perform more.

Click to expand...

RouterOS does have a L7 firewall but what it can do is mainly what the firewall can do with something. To use L7 you must add the hash to it, it doesnt have pre-recognised L7 but you can for example add the skype hash and prevent skype file transfers but it would require scripts to check each http request from L7 to implement adblock.

I am contributing a review in a months time or 2, I just need to get back to my CCR and start making tutorials, pictures and such. I've already told thiggins that as well.

My review will cover the Mikrotik CCR1036, CRS226 and RB450G. The CCR is also faster than a PC based router. Comparing the CCR1036 and a computer at that speed, you would need expensive network cards just to match the port capacity of the CCR1036.

System Error Message said:

RouterOS does have a L7 firewall but what it can do is mainly what the firewall can do with something. To use L7 you must add the hash to it, it doesnt have pre-recognised L7 but you can for example add the skype hash and prevent skype file transfers but it would require scripts to check each http request from L7 to implement adblock.

I am contributing a review in a months time or 2, I just need to get back to my CCR and start making tutorials, pictures and such. I've already told thiggins that as well.

My review will cover the Mikrotik CCR1036, CRS226 and RB450G. The CCR is also faster than a PC based router. Comparing the CCR1036 and a computer at that speed, you would need expensive network cards just to match the port capacity of the CCR1036.

Click to expand...

Looking forward to your review! I now have Ad Blocking in place through simple webproxy/deny rules. Works, but still have to get used to their methods of scripting. VPN l2tp/IPSec also works fine and is very easy to setup. Didn't check on Vpn performance yet but seems to run smooth.
Btw the webproxy caching seems to work very nice as well; I will probably upgrade the internal 2GB to 16GB just for this purpose, as memory is cheap. Reminds me on squid; hoping this cache can be used for app version upgrades.
Regarding PPPoE fast path; its documented that this is not supported so you might want to reconsider your statement about this one.

stevech said:

/opinion
small audience here for such an exotic beast.
That company (w/80 employees) is on my forever black list for nonchalantly shipping products with beau coup bugs, leaving me at a customer's field site in nowhere-land with no hope. Tinker-things, not products.
/endopinion

http://www.mikrotik.com/aboutus

Click to expand...

While on holiday in Indonesia this summer, I noticed that each and every hotel was using ubiquiti or Mikrotik access points, all with Mikrotik routers. Performance was impressive, better than most hotels here in Europe.
As it seems, Indonesia is fully Mikrotik oriented. Hundreds of certified IT companies are utilizing this product all over this beautiful country. It's not limited to east Europe alone.

coxhaus said:

Consumer router are now lagging internet speeds for home use. If you want to fully utilize the fastest home internet connections we need faster routers. Right now you have few selections, mainly PC based.

Click to expand...

We're hitting design limitations of the current architectures - mostly inspired by the Linksys WRT54G and the GPL dump from it... more CPU/RAM and faster ports/WiFi, but generally not much different across any and all vendors...

paraplu said:

While on holiday in Indonesia this summer, I noticed that each and every hotel was using ubiquiti or Mikrotik access points, all with Mikrotik routers. Performance was impressive, better than most hotels here in Europe.

Click to expand...

I don't think it's vendor related - but mostly on design and integration - Last year I did a tour in Japan, and the hotel WiFi was amazing, and the wired Ethernet drops were even better... and that was Cisco and Aruba gear for the most part (and one place was Huawei based).

Not much free WiFi over there, but they've got insanely fast bandwidth throughout all the major cities (I was in Yokohama, Tokyo, Nagoya, Osaka, Kyoto, Miyazaki, and Kagoshima)..

Heck - Asia in general is well beyond what we expect in the US... residential or hotspots...

Worst hotel WiFi in the past year - Doubletree in Overland Park, KS (just outside of the Sprint headquarters) - 11g, but very unstable and almost useless - 5 Mbps at best - I was better off using the hotspot mode in my 4G smartphone... again, Cisco...

sfx2000 said:

We're hitting design limitations of the current architectures - mostly inspired by the Linksys WRT54G and the GPL dump from it... more CPU/RAM and faster ports/WiFi, but generally not much different across any and all vendors...

Click to expand...

Still waiting for a NUC with multiple Intel gb

sfx2000 said:

I don't think it's vendor related - but mostly on design and integration - Last year I did a tour in Japan, and the hotel WiFi was amazing, and the wired Ethernet drops were even better... and that was Cisco and Aruba gear for the most part (and one place was Huawei based).

Not much free WiFi over there, but they've got insanely fast bandwidth throughout all the major cities (I was in Yokohama, Tokyo, Nagoya, Osaka, Kyoto, Miyazaki, and Kagoshima)..

Heck - Asia in general is well beyond what we expect in the US... residential or hotspots...

Worst hotel WiFi in the past year - Doubletree in Overland Park, KS (just outside of the Sprint headquarters) - 11g, but very unstable and almost useless - 5 Mbps at best - I was better off using the hotspot mode in my 4G smartphone... again, Cisco...

Click to expand...

Yeah I can concur that in most Asian business hotels, in big cities, the big-name equipment is used, and works great. But I am referring to tourist quality hotels on small islands etc; I was very impressed that they got the same kind of quality in place using cheap Mikrotik gear.

The proxy server on routerOS is a bit buggy. When the cache fills up you cant empty it (really sucks if you used ram) which slows it down and apparently for it to function properly anyone can access it. For some reason it doesnt follow the access list and i had bots trying to use it as a way to get money by browsing ads so i defeated the bots using the tarpit rule and decided to use 3s whitelists only to IPs requested by the proxy/LAN clients. Doesnt help with proxy use because many websites that used redirects and such didnt load.

I really hope they fix it soon or this is going in the review.

System Error Message said:

The proxy server on routerOS is a bit buggy. When the cache fills up you cant empty it (really sucks if you used ram) which slows it down and apparently for it to function properly anyone can access it. For some reason it doesnt follow the access list and i had bots trying to use it as a way to get money by browsing ads so i defeated the bots using the tarpit rule and decided to use 3s whitelists only to IPs requested by the proxy/LAN clients. Doesnt help with proxy use because many websites that used redirects and such didnt load.

I really hope they fix it soon or this is going in the review.

Click to expand...

Thanks for the hints. Gave up on using webcache, although I did not encounter the kind of problems you mentioned. But having a http proxy gave me trouble with VPN clients. Instead implemented Adblocking through simple DNS static assignments. My DNS static list is currently having around 14k entries which works fine on the CCR without performance impact.
Btw I love their PCQ method of qos.