Confession of an IPFire Newbie ...

[Deepcuts]

During the last 3 days I have tested several alternatives to my "plastic" Asus RT-AC68U consumer router.
I think there might be some users out there, like myself, who would like to build their own x86 router, so I would like to describe my ramblings on the matter, in the hope it will save some of you some precious time.

Just to make one thing clear from the start: I do not think Asus RT-AC68U is bad per se.
It is just that I grew tired of getting home and wanting to watch something on my Samsung Smart TV via Plex only to find I have to "Please wait" while the darn thing updates itself way too often, many times actually getting stuck and requiring a cold restart.

If you can imagine, I actually started this little project because of Samsung. Thank you Samsung! No, I mean it. I have learned new things and there is a small chance I may actually be a bit smarter now, at least in regard to "do it yourself" routers.
Maybe you are wondering what Samsung has to do with all of this.
The reason is simple: most consumer routers don't have ACL support. Some of them have a crude support via scripts to interact with iptables, support usually added by 3rd parties and not the actual original vendor (see RMerlin). To be able to stop my TV from contacting Samsung servers for updates but still allowing it to access my Plex server, I needed ACL support in my firewall, which Asus does not have.
Of course there are also other features I am missing with a consumer router. On 1st place, as I have said, is the absence of ACL support. (at least an easy to use one)
On 2nd place is the absence of caching, which for most of you will not matter at all. For me although, it is of a big help, given that I have to install, almost daily, various operating systems which I also have to update. Using a caching solution like squid in transparent mode for example, will eliminate the need for an operating system to download it's updates directly from "the source", but instead will serve the updates from a cache stored on a fast SSD attached to the router.
On 3rd place is the performance of consumer routers. Even if RT-AC68U can sustain 1 Gbps traffic download, that does not mean it can do it symmetrical, or that it can sustain wire speed OpenVPN sessions. Given that most consumer routers use low powered processors, I can't really blame them for not delivering full gigabit symmetrical wire speed, although the prices on some models could be a bit lower because of this. Of course there are entry level enterprise routers that can do 1 Gbps up and down...for a price, but I came to the conclusion that building one myself is the better path for me.
Other notable features missing from everyday consumer routers in my opinion:
Lack of reporting tools.(who visited what and when last month?)
Lack of upgrade options. (need 10 Gbps? new card not a new router)
Few customization options. (need to host a small website? most linux router distros can do that)
No RTC battery on most consumer routers. (date has to be synced from ntp on every boot)
I will stop the list here.

So, I was saying I got upset with Samsung screwing with my free time and I decided to build myself a small mITX router.
Given that in the last few years I only used Asus routers at home (thank you RMerlin) and Mikrotik/Cisco at work, I wasn't very up to date with linux/BSD router distributions and their hardware compatibility list (HCL).
So I started thinking that during those years I haven't used one, all serious router distros have improved a lot. Who needs to check a hardware compatibility list? So I started building my dream home mini router. More on this later on.

 

[Deepcuts]

The hardware
I decided to stay on the safe side and over provision on hardware...for the future.

CPU: Intel i3 6100T
Decently powerful with only 35W consumption. 2 cores at 3.2 Ghz
Mainboard: Gigabyte GA-H170N-WiFi
Small mITX board with dual Intel Gb nics. The board also has an M.2 slot, just right for a fast SSD.
Dual Gb NICs Intel i211 and Intel i219-V
WiFi AC 2.4/5 GHz Dual-Band from Intel.
Storage: Samsung SM951 Series 128GB
Plenty of space for caching and very good R/W speeds. (650 MBps write and upto 2 GBps read)
Case: Inter-Tech E-W60 with 60 watt power supply included.
Small mITX case made with aluminium. (external PSU)
Cooling: To cool this setup, I chose a cooler from Noctua, NH-L9i.
Small profile and very silent. (if positioned correctly)
Additionally, two 50 mm silent Noctua NF-A6x25 fans for the summer heat.
Because the main board has only two RJ45, I purchased a Linksys Gigabit SE2800 switch too. Later on I realized I could just use the switch on the RT-AC68U.
All the parts=~500 EUR.

You might say: "hey, for ~500 EUR you could have bought a pretty decent router like Mikrotik CCR 1016-12G" and you might be right.
But then again, think about the noisy fan CCR has. Even under light load, you begin enjoying your noisy vacuum cleaner's company.
There is CCR1009-8G-1S-1S+PC with passive cooling indeed that removes the noise part from the equation. Corect.
But RouterOS is closed source, thus lacking any additional customization, and the caching engine is not something I would write home about (yet). Also, no VM, no IDP/IDS and maybe my biggest gripe for my use, comes only in 1 U form factor while I need a little mITX sized one.
Do not get me wrong. I love Mikrotik and I use them at work everyday. Just not for the project I have in mind.


Got all the toys, time to start tinkering.
I started thinking that this puppy has too much "horse power" under the hood (for my needs) so maybe a virtualization solution which would allow me to run additional VMs would be in order.

So I started with ESXi 6.0 update 1, the free version.
No such luck. For one, ESXi 6.0 does not recognize the Intel i219-V network card. Also, it cannot write the boot loader on /dev/nvme0n1 (the M.2 SSDs are seen as nvme0n1). Tough luck. Next!

I tried ESXi 5.5.0 update 3, after reading that somehow, an older version has support for my Intel i219-V network card.
Downloaded 5.5.0 update 3 version and wrote the ISO to a DVD.
It actually has support for it, but again, it cannot write the boot loader on the M.2 SSD. Next!

In the past I dabbled with XenServer so I thought: let's give it a try!
Downloaded the latest stable version and wrote the ISO to a DVD.
Latest stable version 6.5.0 does not support the Intel i219-V network card. Also, it cannot write the grub boot loader on the M.2 SSD. Next!

Let's try a beta version of XenServer. Loaded 6.6.91 and tried again. partial success. My Intel i219-V was detected but again, the grub boot loader could not be written to the M.2 SSD.
Already pissed off at the situation, I plugged in an additional Samsung EVO 840 120 GB SSD and, success!
I managed to install XenServer 6.6.91 beta with both network cards detected.
I created an ISO Repo and loaded an Ubuntu 15.10 Desktop and Windows 7 x64 just for testing the performance. Even with the tools installed, the performance was lackluster in both network and disk throughput. No matter what i tried I couldn't make any VM perform decently.
What is the point in virtualizing the router if it won't be able to perform?
Maybe in the future, more mature drivers will be available for the Skylake/H(Z)170 platform and the performance will be ok. But for now, a big no go.

Reading a bit about "Should I virtualize my router?" on the web, made me ditch the idea. At least for the moment.


So I got to the "Let's try a Linux Router Distro without a VM" part.
I made a small list of which distros I should try.
Distros that made the list: (order of testing)
Sophos UTM
pfsense
Untangle
Endian
ClearOS
IPFire

What I was looking for: a router distro with free/home offerings that doesn't have too many feature limitations and would work on my already purchased platform.
There are many other router distros out there, but given that I actually started this project with the intent of having more time to watch Plex on my TV as opposed to watching Samsung updating the darn Smart Hub (hint: not so smart), more than 6 distributions to test would have been pointless.

Sophos
Reading reviews about it made me curious. Never used it, so now it's just the perfect time to see how it performs.
One needs to register on their website to be able to download the ISO. All in all, an unnecessary lengthy process.
Got the ISO, wrote it on DVD, got a fresh mug of coffee and started the install.
Surprise, surprise. Intel i219-V network card is not recognized. It seems the chip is too new. Searched a bit on the web if somehow, somebody managed to get it working, but no luck. Next!

pfsense
Used it extensively in the past and even tho the interface is not the easiest one to use, it's features make up for it.
Got the ISO, wrote it on a DVD and started the install.(still had coffee left from Sophos)
Same old story. Installer detected only one network card, the Intel i211 one. i219-V support missing from pfsense, but it seems it could be possible in the near future to have support for it, given that FreeBSD will support it soon. Next!

Untangle
Never used it before so I was curious.
Got the ISO, wrote it on a DVD and by this time I needed a coffee refill.
Again, support for Intel i219-V is absent, as well as the old problem with grub boot loader and my new, shiny and all powerful M.2 SSD. So, I installed it again on my Samsung EVO 840 120 GB SSD, which worked.
Untangle is based on Debian (as far as I can tell) so i started thinking that maybe I can compile the latest 3.3.3 version from Intel website and somehow make it work.
Which I did! After having to install various prerequisites like linux headers and essentials.
Simply wget the tar.gz from Intel, decompress it and /make & make install.
Got the internet up and running. Felt good.
Accessing it's web management address I was greeted with "Install applications?" Sure. It's like a latte at Starbucks. "I just have to sit here while you make it and do nothing? Give me two!".
Once applications were installed, I could see that most of them are actually trial versions. I expected that much, but it seemed most of them were trials. Most of them I had no use for, but the one that made me say Next! was the proxy/caching application which of course, was a trial version (based on squid nevertheless). So, Next!

Endian
Used it in the past and actually still have one machine at work with a very old version installed.
Got the ISO...you know by now the rest.
Good news. It seems version 3.2.0-alpha1 supports Intel i219-V network card. The installer went smooth but again, could not write the grub boot loader to the M.2 SSD.
Given that support for my i219-V is a "rare sight for sour eyes", I installed it again on my Samsung EVO 840 120 GB SSD, which worked.
All went well and soon I got to the management web address. Did a basic setup and internet was up and running in no time.
Nevertheless, browsing experience from a machine behind the router was far from the expected stellar.
Downloads via sftp/http/https were not starting fast, requiring about 2-3 seconds to start.
Download speed was all over the place, mostly in the wrong one, the bottom.
I guess it is called alpha1 for a reason. Next!

ClearOS
Based on CentOS, I had high hopes. Reading web reviews only heightened my expectations.
Got the ClearOS Community version 7.
No warning during install. That is...until the grub boot loader had to be written. Old Samsung SSD to the rescue. By now it seemed like I could have just used the old SSD and not buy the M.2 one.
Rebooted and was greeted by a sleek interface. Nice.
Not so impressed when I was again presented with the "Only one network card" screen.
Too bad, so sad, love bread. Next!

 

[Deepcuts]

IPFire
To be honest, I never heard about it. Just stumbled upon the distro while reading reviews online.
The downside. Only 32 bit version. There goes my 8 GB RAM dream. Just think about caching in RAM. - edited below
Later edit: it seems that if you have a 64 bit system with more than 4 GB RAM, you can install a PAE enabled kernel via Pacfire and use all the available memory. Nice!
Mem: 8174180k total, 2472312k used, 5701868k free, 162744k buffers

So I decided to give it a shoot. I started with version 2.17 core update 98.
To my surprise, smooth install on the SATA SSD with both network cards detected. I guess I'll be using the new M.2 one in the future.
In no time, it was up and running and started testing the internet.
Best experience so far. Website were loading very fast and the distro came without any limitations.
All you would ever need and then some more is included. If not, you have 'Pacfire" from the web interface, through which you can install addons.
Web caching works like a dream and it has a neat feature called "Update Accelerator". Exactly what I was looking for. I might be wrong, but I think "Update Accelerator" is separate from the actual Squid cache.

As you can see from the cache statistics screen, after being downloaded once, almost all updates are delivered from cache. Lightning fast.
It might be a placebo, but even download speeds through ssh are more stable and fast then with my RT-AC68U.
The speed is now more like a flat line (on the upper side) then a 'It's alive! Nope, it's dead. It's a live again" sinusoidal curve, staying in the 100 MBps more than before, with less drops in speed.
OpenVPN is fast to setup and speed tests revealed very good throughput.
I needed a fast VPN for connecting via internet several Windows machines for a side project based on RipBot264. While working with 30+ GB video files is nice to have a fast VPN connection.

To make full use of IPFire, one needs to add some addons. For me, the required addons were:
apcupsd - for UPS support
clamav and squidclamav - for AV transparent proxy scanning.

miniupnpd - for upnp (opening ports in firewall automatically for certain applications)

As far as I can tell, transparent proxy only works with certain extra steps.
Add the following two option to the DHCP server:

Change the IP to suit your setup.
Browsers behind the router must have "Automatically detect settings" for proxy enabled and it only works for DHCP clients (as far as I can tell).
Maybe there is a better way to do it, but I haven't discovered it yet.

 

[Deepcuts]

SSH to the router and I could fdisk the M.2 SSD. Now I just need to find a way to make caching store the files on this SSD for even faster file delivery.

Testing with hdparm:

Half of the 2 GBps but still faster than the EVO 840

So far, IPFire seems to be the match for me. Too bad I didn't knew about it sooner.
Until VM solutions catch up with my Skylake/H170 platform and have mature drivers for it, I will use IPFire as a dedicated router. If I get lucky soon and VM solutions will provide adequate support for my platform, I intend to switch to a virtualized router.
Will keep my RT-AC68U as an Access Point only, given that the WiFi module on the mITX board is not supported by any router distribution I have tested (for the moment). To be honest, even if the module was supported, I do not think the performance would have been on par with an actual Access Point. Then again, without testing I cannot say for sure.


So the moral of the long story is:
if you plan on deploying a custom x86/x64 router, always check the HCL for your desired distribution.
That is, if you are lucky enough to find it and it is updated/maintained.

Buying first and thinking last can be a painful and time consuming experience.
Of course, I learned some new stuff, managed to compile my first driver in linux and feel good about it, got to see what "free" actually means for some router distributions, learned that not all Intel nics are created equal (slap it in and it works) and at the end of the day(s), I have a router pretty close to what I intended when I started this little project.

Whatever you chose for a router distribution, a good place to start might be checking if the project is actively maintained. Having an insecure platform as a door to the big bad internet is a bad idea.
A bonus is also an active community/forums.

Thank you for reading this far.
English is not my native language so please do excuse the eventual grammar slaughtering.

With the hope this might help someone with a similar setup, good luck.

later that day....

So...I started by saying Samsung "made me do it".
Having finished my brand new router and feeling good about myself I decided to call it a day and go watch a movie while trying to get some rest.

Of course I forgot to block the darn Samsung
In case the image is too fuzzy, it says: "Updating Smart Hub now..."

So, open the management interface.


Step 1:
Go to Firewall>Firewall Groups
Click the "Hosts" button
Add your TV by IP or MAC and give it a name. Click "Save".
You might want to assign a static IP from the DHCP server page first.

Step 2:
On the same page, click on "Service Groups"
Group name: DNS TCP and UDP
Click "Add"
On the same page, edit the newly created group and add port 53 TCP and UDP (just to cover all the bases)
Save by clicking on "Modify"

Step 3:
Go to Firewall>Firewall Groups
Click on "New Rule" button and add the rule:
Source - Hosts (Your TV)
Destination - Standard networks (Any)
Protocol - Preset (and select the service group DNS TCP and UDP)
Set the Action to "Allow"
Rule position - 1

Step 4:
Click on "New Rule" button and add the rule:
Source - Hosts (Your TV)
Destination - Your Plex Server IP
Protocol - All
Set Action to "Allow"
Rule position - 2

Step 5:
Click on "New Rule" button and add the rule:
Source - Hosts (Your TV)
Destination - Standard networks (Any)
Protocol - All
Set Action to "Drop"
Rule position - 3

Make sure all rules are activated.
Not exactly a clever method, but with this setup, all connections from my TV to any other host BUT my Plex server IP are dropped. I have to mention that my Plex server is not on the same network, but hosted in a DC.
I allowed DNS ports to trick the TV into believing that remote sites (the ones it is connecting to) are resolving but at least for the moment, are down.
Tested this setup and could only observe a small delay in starting up the Smart Hub and Plex App.
Also, all my Plex libraries are doubled for some reason in the menu.
Plex streaming is working as intended.

I guess if one wants to get scientific with this, he/she could just create a rule to drop all traffic from the smart TV to "any", but also log the rule.
This way, all outbound connections will be logged and can be observed in "Firewall logs"
Create a group with the logged destination IPs and drop the list outbound. ( hint: remove your Plex server IP. Just saying )
I haven't tried this one yet, but for the moment, the "not so clever method" works for me.
I only use Plex on my Smart TV, so blocking everything else does not hurt the 'Smart" experience for me.

Other tips and tricks

VoIP/SIP phones behind IPFire (SIP/NAT Helper)
If you have a VoIP phone connecting to a PBX outside your network, the default setup will not work, given that nf_conntrack_sip & nf_nat_sip modules are disabled by default.
This means that your phone will register to the PBX with the local IP instead of the Public IP, resulting in one way audio or the phone not ringing at all.
To fix this, ssh into the router and run the following two commands:
modprobe nf_nat_sip
modprobe nf_conntrack_sip
Restart/Re-register your Phone and should be good to go.
To make changes permanent, edit /etc/sysconfig/rc.local and add at the end of file:
modprobe nf_nat_sip
modprobe nf_conntrack_sip
exit 0
Save the file and reboot.

Alternative: backup and delete the file /var/ipfire/main/disable_nf_sip
Reboot

GA-H170N-WiFi - No post without a monitor
Do not disable the 'Full Screen Logo" in BIOS or the board will not post without a monitor attached.

Hardware errata
Noctua NF-A6x25 is 25 mm thick.
The case and mITX board only allow 10 mm thick fans. Do not buy them for this build.

 

[Nullity]

What method did you finally choose to block the samsung smarttv servers?

For a similar purpose, I have used the gateway DNS forwarder to block requests for certain domains. dnsmasq supports blocking entire domains like "samsungupdate.com" (includes all sub-domains) or wildcards like "update#tv.samsung.com" (# is wildcard character).

 

[sfx2000]

Great writeup - nicely done!

 

[Deepcuts]

What method did you finally choose to block the samsung smarttv servers?

For a similar purpose, I have used the gateway DNS forwarder to block requests for certain domains. dnsmasq supports blocking entire domains like "samsungupdate.com" (includes all sub-domains) or wildcards like "update#tv.samsung.com" (# is wildcard character).

I went with the "let's just take away all candies from the baby" method.
Please check the end of the updated post.

 

[YeOldeStonecat]

Always fun stuff to play with.

The various *nix firewall distros tend to often not support the latest "motherboard of the month club" chipsets and NICs....but you'll generally have a smoother experience if you use some slightly older, business grade hardware. Typically better support from the native OS of the distro.

Years ago I played with TONS of *nix firewall distros...each month I'd download something new and load it...and played with it for about a month. IPCop was a big one back in the old days, and the add-on which made it sorta a UTM...called Copfilter. I remember trying PFSense many times since the year they started...great "Ferrari" distro....very fast. Sophos is a very matured UTM distro. m0n0wall, Smoothwall, Simplewall is another fairly new UTM distro, Endian is another decent UTM. ClearOS is a good one, a sort of combination with an SMB Server. (like an open source version of Microsoft Small Business Server Premium).

I'm big on Untangle now...we sell and manage lots of those for business networks. It's a bit overkill for the home user.

 

[Deepcuts]

Some more info regarding the Intel WiFi PCIe cards like 8260.
Suport for the WiFi card has been added in Core 100.
For some reason, Intel locked them on 5 Ghz, so using one with hostapd to make a WiFi Hotspot will result in very low speeds (~50 Mbps up and down).
More info: https://communities.intel.com/thread/60677?start=0&tstart=0
jfyi

 

[sfx2000]

Some more info regarding the Intel WiFi PCIe cards like 8260.
Suport for the WiFi card has been added in Core 100.
For some reason, Intel locked them on 5 Ghz, so using one with hostapd to make a WiFi Hotspot will result in very low speeds (~50 Mpps up and down).

Intel makes great client adapters, but I wouldn't consider them suitable for hostapd services - they're pretty low power (16-18 dBm) compared to other solutions - and as noted above, they're intended more for AdHoc/IBSS services...