pfSense (or other dedicated router) questions

[garyd9]

There are AP's out there that can bind a VLAN to a specific SSID - so one can have two (or more) SSID's, each bound to a specific VLAN - and then you can apply routing choices (and policies even) on each one of the VLAN's

I'm doing it in tomato (based on one of the WRT variants... and oddly AsusWRT is based on tomato) on a netgear R7000. It's cumbersome as hell, but still doable.

First I create "virtual" wireless adapters for the alternate SSID's. Then I have to create new bridge interfaces, and assign different virtual wireless adapters to different bridge interfaces. Finally, I assign a vlan and VID to each bridge. (It appears I can only assign a single vlan to each bridge.) Finally, I can assign one or more bridge interfaces to a physical ethernet jack on the AP. If I assign more than one bridge, the firmware requires that the vlans are tagged (which makes sense.)

One very large shortcoming, in my opinion, is that the firmware only seems to support 16 sequential vlan ID's. 0-15 or 16-31, or 32-47, etc. (Actually, it only supports 0-15, but then lets me add a multiple of 16 to all of them at once.) On the other hand, I'm not going to complain loudly about this... I'm just happy that I can get the thing to do vlans this flexible to begin with. (This router isn't designed for me. It's designed for someone who plays games, has only a few wired devices, has no clue what a vlan is, and can't imagine ever having to buy a switch with more than 4 ports.) An alternative to that would be to NOT use tagged vlans at all, and just assign each bridge to a different ethernet jack (I have 4) and then set them as untagged vlans in my switch. However, I don't want to run 3 or 4 separate wires to the AP so I'll just live with the limitation on vlan ID's.

Eventually, I'll buy a "real" AP (or two), and I'm sure it'll have much more flexibility on the configuration of vlans. (The UAP-AC-PRO sounds very appealing...)

It doesn't have the bling that pfSense has, but there are OpenWRT builds for x86 architecture. Before I bought my Ubiquiti, I ran OpenWRT in a VM on my ESXi server. I never got the hang of managing it via the CLI, doing upgrades was cumbersome, and the GUI was not superclean. But probably had most solid IPv6 implementation of any of the tools at the time

Yeah, it really amazes me how incredibly poor the IPv6 support is in all these products. IPFire? No built in IPv6 (can be added by adding modules, editing files, recompiling nettools, and managing only via CLI?) Untangle? Sophos XG Firewall? All barely have any IPv6 support at all. In each case, it seems like something they just added support for in ONE module just so they can say they support it... It's crap. I haven't tried the older Sophos UTM yet, but my hopes aren't high.

In that regard, pfsense is better than most of the others. I can request a /60 from comcast on my WAN interface, and it'll not only properly request it, but then lets me delegate the last 4 bits to each of my other interfaces (so if I get A:B:CEF0::/60 from comcast, I can tell pfense that "LAN" tracks the WAN ipv6 and gets a prefix ID of "2", and LAN will do assignments from A:B:CEF2::/64.

Outside of TWO things (both resulting from the whole L3 only mentality), pfsense is pretty awesome. The first is the inability to do MAC based filtering. The second is that all reporting is also L3 based, which is f'ing annoying as hell. Instead of seeing what traffic machine XYZ is doing, I have to manually figure out which IPv4 it had assigned, and all the different IPv6 addresses it might have used, and then manually put the results together.

Oh, well... I'll keep looking... (and trying to bug the pfsense folks to see things slightly differently.)

 

[sfx2000]

I'm doing it in tomato (based on one of the WRT variants... and oddly AsusWRT is based on tomato) on a netgear R7000. It's cumbersome as hell, but still doable.

First I create "virtual" wireless adapters for the alternate SSID's. Then I have to create new bridge interfaces, and assign different virtual wireless adapters to different bridge interfaces. Finally, I assign a vlan and VID to each bridge. (It appears I can only assign a single vlan to each bridge.) Finally, I can assign one or more bridge interfaces to a physical ethernet jack on the AP. If I assign more than one bridge, the firmware requires that the vlans are tagged (which makes sense.)

One very large shortcoming, in my opinion, is that the firmware only seems to support 16 sequential vlan ID's. 0-15 or 16-31, or 32-47, etc. (Actually, it only supports 0-15, but then lets me add a multiple of 16 to all of them at once.) On the other hand, I'm not going to complain loudly about this... I'm just happy that I can get the thing to do vlans this flexible to begin with. (This router isn't designed for me. It's designed for someone who plays games, has only a few wired devices, has no clue what a vlan is, and can't imagine ever having to buy a switch with more than 4 ports.) An alternative to that would be to NOT use tagged vlans at all, and just assign each bridge to a different ethernet jack (I have 4) and then set them as untagged vlans in my switch. However, I don't want to run 3 or 4 separate wires to the AP so I'll just live with the limitation on vlan ID's.

Eventually, I'll buy a "real" AP (or two), and I'm sure it'll have much more flexibility on the configuration of vlans. (The UAP-AC-PRO sounds very appealing...)

Yeah - been there - got the scars and the project t-shirts, lol...

When building out VLAN's - just as a tip - try to stay above 100, and perhaps below 1000, and that's a fairly decent range to work with for VLAN ID's

Below 100 - believe it or not, some of these might already be in place internally within a device, esp a customer router/AP - it's not exposed, but one must be wary of that.

The UAP-AC-PRO is a nice AP - if I were in the market for an AP, I'd consider it as it is a good performer, and a decent value -

At the moment, my Airports in AP mode are just fine - they're paid for, which is an important consideration, and they do GuestSSID over a known VLAN ID, which makes things a bit easier - there were some performance issues with that VLAN, but solved with their recent FW update last month (the 7.7.7/7.6.7 builds for 11ac/11n Airports).

With DDWRT/OpenWRT on your wireless AP's, you might get what you're looking for - and the R7000 is pretty DDWRT friendly - not sure about it's OpenWRT status - but there are a couple of AsusWRT forks that supposedly support the R7000, but I'd stay clear of those builds (mainly because, IMHO, it's pretty sketchy, and disrespectful of ASUS intellectual property, along with RMerlin's contributions, but that's me..)

 

[sfx2000]

So in many ways - you've got a good foundation - esp with the SuperMicro board you've selected...

Can go to many places with it - pfSense, OpenWRT, SophosUTM, Microtik's RouterOS (with a license fee there), VyOS, build your own with Linux or *BSD, or let Windows do it (Windows Server can do this as well, acting as a router with policy managment and the whole nine-yards, under appreciated by many).

You're on the right path - it's not just the Router, it's the switching and the AP's, and how to manage it - make it complicated, then it's more work, make it simple, it'll be solid and reliable - but every network is a snowflake, and most have a mix of needs/wants that make it this way.

And to that end - sometimes it pays to step back - and really ask - is this the right way?

Home networks can be really simple and flat and fast, or they can be complicated and fragile and a lot of time to manage - and common sense is usually the best approach.

Reason why I asked for a document earlier in the thread - it's to get something on paper/document/whatever - as this turns wants/needs into requirements on something tangible - and by doing this step, it turns those into requirements that one can design a network around.

This is not just for your benefit - but for the benefit of those following this thread...

By doing that tangible effort - it gets things out of that short-term scratchpad called memory, which will fail you when you least expect it. And the upside is that you have a document to refer to for debug and expansion/upgrades as needed in the future.

The rest is all implementation - and there, as I mentioned earlier, you're on the right path, and there's not much more I can contribute perhaps...

 

[garyd9]

There is one MAJOR issue with the R7000 (and the reason I tried to replace it with the asus 3200 to begin with): The current drivers used in all WRT based firmware versions BREAK WPA2-Enterprise (EAP/PEAP) radius authentication against my windows server. If I use any non-netgear firmware with drivers newer than about 18 months ago, BOOM... suddenly nothing can authenticate to my windows server.

I've never been able to figure it out, but it prevents me from updating to something that closes a couple of security issues that have been discovered over the past year. That's not really acceptable for a router that stands between my home network and the Great Wild Internet. (Most home users probably don't realize that they are exposed to all kind of vulnerabilities. I do.) As a plain AP, I don't mind the security flaws so much. Even if it's wide open wireless, you'd still need physical proximity to get in.

Unrelated to the AP, but back to the firewall/router: Sophos UTM looks like it'd be a viable alternative to pfsense. The problem is that the "free" version is limited to 50 users. No, there's not 50 people in my house, but I do peak around 35 to 40 devices with an IP address (includes IoT devices, etc.) So, if my kids have friends over, it's very possible for me to go over that 50 number...

Why isn't there a "perfect" product for me?

 

[mtganzer]

Can go to many places with it - pfSense, OpenWRT, SophosUTM, Microtik's RouterOS (with a license fee there), VyOS, build your own with Linux or *BSD, or let Windows do it (Windows Server can do this as well, acting as a router with policy managment and the whole nine-yards, under appreciated by many).

Just FYI VyOS does not yet support DHCPv6-PD (been there...done that). They are using the ISC DHCPv6 client, which doesn't seem to support Prefix Deligation yet. Ubiquiti swapped the ISC client out for the WIDE DHCPv6 client so they can support DHCPv6-PD in EdgeOS. I have seen suggestions on the VyOS board to do the same, but they have not done so. If I had the cycles I would give it a shot, but I am a network engineer, not a developer.

Oh the joys of supporting IPv6 - my organization has been on the bleeding edge of IPv6 since early 2000's (part of the original 6Bone), and I was compiling USAGI stack into Fedora kernels. Yet 15 years later we still we have issues with Windows not supporting RDNSS, and Android not supporting DHCPv6 - yikes!!

 

[sfx2000]

Just FYI VyOS does not yet support DHCPv6-PD (been there...done that). They are using the ISC DHCPv6 client, which doesn't seem to support Prefix Deligation yet. Ubiquiti swapped the ISC client out for the WIDE DHCPv6 client so they can support DHCPv6-PD in EdgeOS. I have seen suggestions on the VyOS board to do the same, but they have not done so. If I had the cycles I would give it a shot, but I am a network engineer, not a developer.

Gah - totally get it... and part of the problem is the operators themselves there...

The Mobile operators seem to be doing better at it - mostly because 3GPP has limited the options, which makes v6 a bit easier to work with...

 

[garyd9]

I've been trying out a bunch of different free x68 based router/firewall software distros/packages (that include some type of web management UI.) In particular, I've been looking at their IPv6 support.

My minimum ipv6 expectation is that the router IPv6 DHCP client requests (AT LEAST) a /64 PD, and uses that to assign IP addresses to clients of the router. I'm okay with a bit of manual configuration as long as I can expect the router software to change the prefix used for clients if the prefix on the WAN side changes. Here's what I've found so far:

• pfsense: good (to excellent) ipv6 support. For example, can force a /60 request, and then give out the 16 subnets of that /60 to different subnets - all dynamically tracked from the /60 retrieved from the WAN side.
• IPFire: poor ipv6 (can only be done via manually loading modules, editing files, recompiling tools, and using the CLI)
• Sophos UTM: 50 IP address limit. That limits to < 25 devices if they all get 1 ipv6 address and 1 ipv6 address (Otherwise really, really nice.) So, it meets the ipv6 requirement, but is fairly useless in even a small home due to each ipv6 reassignment causing another of the "50 addresses" limit to be used. (The cost of moving past the 50 address limit is unreasonable for even a rich home user.)
• Sophos XG firewall: only supports getting a single IPv6 addr. (/128) from dhcp6 client
• Untangle: poor ipv6 support (static only.) (Reading their forums, the approach to PR is "if you don't like it, you're banned from the forum.)
• IPCop: no ipv6
• Endian: no sign of ipv6 support
• Smoothwall: no ipv6
• DDWRT: doesn't seem to be free for x86
• RouterOS: not free
• VyOS: no PD support (might get just a single /128)

Have I missed anything?

 

[sfx2000]

Had to head over to Fry's this morning to pick up a couple of SDCards (they had a local special on them) and saw something on the networking shelf that reminded me about this thread...

Engenius Managed WLAN in a box - aka "EWS Starter Kit for Small Business" - lot of good stuff for $380 bucks...

Two N300 dual-band AP's with POE and centralized management
8 port Managed POE switch with Wireless Management - including two additional SFP ports

https://www.engeniustech.com/products/controllers-switches/wireless-management/ews2910p-kit-300.html

Amazon has it for the same price..

Would perhaps be a bit more attractive if they put two AC1200 AP's in that box rather than two single band N300's, but the real value is in the bundled switch and wireless LAN controller...

 

[sfx2000]

I've been trying out a bunch of different free x68 based router/firewall software distros/packages (that include some type of web management UI.) In particular, I've been looking at their IPv6 support.

pfSense usually has good IPv6 support, but much depends on your ISP - I've had ongoing issues with my ISP (Cox HSI) - they do DCHP6-PD, but their network tends to lose track of me - can refresh the WAN interface, and things come back, but that's a pain - esp. since my work VPN will use v6 if available, and then all of a sudden, when Cox gets forgetful, I lose connectivity there

I've escalated this internally within Cox, and actually had a good teleconference with one of their network engineers, and pointed out what's going on, but at the end of the day, he was a bit resistant to implement, or even test the proposed change, as this would cascade into a lot of testing across their equipment... end of the conversation, he blamed the premises equipment, e.g. my router, and said the problem is there - not because it was, but because their rental CPE, it works, so it makes it a de facto spec to interop with their network.

Puts me in a bit of a bind, as I can either go DSL (45/2MB at my location and get very good ipv6 support) or stay with cable (150/10mbit) and live with ipv4 at the same out of pocket cost...

And the DSL is capped at 350GB a month vs. my cable co at the moment doesn't have a hard cap, but they're looking in to it, and there the proposed cap is 1TB/month...

BTW - you missed OpenWRT in your eval - it does support x86, at least they claim to, but I haven't really looked too closely into it.

 

[garyd9]

@sfx2000

I'm too lazy to quote and clean up the quotes right now...

That's a nice little wlan kit! While I agree AC1900's would be nice, I think it'd drive the price too high. Perhaps something with at least 2 stream 'ac' would be good. That would work at the highest speed 90% of today's notebooks work at.

As for COX, you have my sympathies. I just discovered today that, unlike /64 delegations, when comcast gives me a /60 PD, it changes fairly frequently. After being up for < 2 days with pfsense, I wanted to reboot it because I reconfigured a bunch of interface stuff (a reboot will ensure all the arp caches are cleaned.) Suddenly, I realized that the prefix on the router changed. I rebooted pfSense again, and AGAIN the prefix changed. (This is after only 5 minutes or so!!!!) All the IPv6 addresses assigned to every single one of my devices is suddenly invalid. Thankfully, most will check in with DHCPv6 within 2 hours and get updated. It's still annoying. I realize that as a "home" user, I don't get a static IP, but my ipv4 usually NEVER changes unless they do head-end work, and my /64 prefixes never changed unless I changed equipment.

...You're right - I skipped openWRT. I looked at it very quickly, and decided that getting it up and running isn't something I could do very quickly, so moved passed it. I really should revisit it again, though, now that I've eliminated everything else.

As far as pfsense...

I have vlan's working, mostly, with pfSense and my AP... Keep in mind that I've never had an excuse to use vlans before. I've known the theory, etc - but never actually implemented with them.

First discovery: a default rule isn't set up for newly activated interfaces, making it completely useless until that rule is set up. That only took 15 minutes to figure out. (Thankfully, with 4 LAN ports on my pfsense box, the LAGG'd ports I'm setting up vlans for isn't the port I was using for configuring.)

Second discovery: pfSense, by default, happily routes ALL traffic between all vlans with no restriction. I'm not entirely sure why it doesn't default to being more restrictive, instead of more permissive, but that's something I can easily fix with rules.

Third discovery (and I should have been prepared for this): Windows network discovery doesn't work across vlans/subnets even when they are routed together. At the moment, I'm having to manually type all my server names instead of finding them in explorer. I'm sure there's a way to get windows to work better across subnets - I just need to figure out what it is. Thank God for google...

Fourth discovery: If I set up my windows ADS as the DNS server, and then isolate that same ADS server from a vlan, name resolution doesn't work really well. (duh.. not really a discovery.. just something I overlooked when configuring things.)

I'm still not 100% sure that pfSense is the best tool for me, but right this minute, it's the only option.

If that "sophos xg firewall" did ipv6 as well as "sophos sg utm", I could probably get very excited about that. They are from the same company.. the "firewall" product is supposedly the next generation of their "utm" product... but I guess they left out a few things. In terms of "free" licensing, the "firewall" product drops the 50 address limitation and changes it to a restriction that it only uses 4 cores and 6 GB of RAM. I think I could live with that. (I wonder if I could convert the other 10 GB into a ram disk.)

 

[sfx2000]

I'm too lazy to quote and clean up the quotes right now...

That's a nice little wlan kit! While I agree AC1900's would be nice, I think it'd drive the price too high. Perhaps something with at least 2 stream 'ac' would be good. That would work at the highest speed 90% of today's notebooks work at.

That's ok - AC1200 class AP's would make that kit a terrific value for anyone, and esp. here on SNB - so seeing the N300's was a bit of surprise, as the costs there are pretty minimal, and would be, for most that have outgrown a single AP solution, the right choice...

As for COX, you have my sympathies. I just discovered today that, unlike /64 delegations, when comcast gives me a /60 PD, it changes fairly frequently. After being up for < 2 days with pfsense, I wanted to reboot it because I reconfigured a bunch of interface stuff (a reboot will ensure all the arp caches are cleaned.) Suddenly, I realized that the prefix on the router changed. I rebooted pfSense again, and AGAIN the prefix changed. (This is after only 5 minutes or so!!!!) All the IPv6 addresses assigned to every single one of my devices is suddenly invalid. Thankfully, most will check in with DHCPv6 within 2 hours and get updated. It's still annoying. I realize that as a "home" user, I don't get a static IP, but my ipv4 usually NEVER changes unless they do head-end work, and my /64 prefixes never changed unless I changed equipment.

Don't get me started here - there's months of frustration built up - and Cox is a fast follower of what Comcast is doing - for a while, Cox was doing the right thing, but they changed a few months back with their peering - moving from Level3 to Comcast for most of their backbone, and it's been hella mess since then.

I've basically disabled IPv6 on the WAN side, as Cox is unreliable, and that's that... outside of that, they've been pretty good...

 

[garyd9]

I think I figured out the problem with the ipv6 PD changing every reboot (and submitted a bug for it): https://redmine.pfsense.org/issues/6667

 

[sfx2000]

I think I figured out the problem with the ipv6 PD changing every reboot (and submitted a bug for it): https://redmine.pfsense.org/issues/6667

only if you reboot the pfSense router... there's a network side that is a factor...

 

[garyd9]

only if you reboot the pfSense router... there's a network side that is a factor...

I don't understand your statement. The DUID gets recreated on each reboot, and the comcast DHCPv6 server identifies my router only by the DUID... so it thinks I keep changing routers each time a reboot.

 

[sfx2000]

I don't understand your statement. The DUID gets recreated on each reboot, and the comcast DHCPv6 server identifies my router only by the DUID... so it thinks I keep changing routers each time a reboot.

Understood - and agree, the client/prem side could do a better job there.. but this is more that that..

Give it a couple of days without a reboot... and you'll find that the network just seems to forget what you are... and there is a reason - and an implementation detail which is a should, not a shall, so some don't do it...

 

[garyd9]

I've read on some (other) forums about that happening with comcast customers with /60 PD's. The comcast side just forgets the routing for the prefix... However, I thought I also read that comcast fixed that issue back in May or so... (I hope they did!)

 

[mtganzer]

Have I missed anything?

Open WRT current release "Chaos Calmer". I don't remember what packages were available, but I remember seeing for example a Snort package compiled for it, as well as people running proxies on it and such. But on your hardware, it screams to be virtualized (which was what I did). And gives you options for example to run Security Onion in another VM (though you;d want a probably want a bigger disk for your Snort logs).

 

[mtganzer]

I don't understand your statement. The DUID gets recreated on each reboot, and the comcast DHCPv6 server identifies my router only by the DUID... so it thinks I keep changing routers each time a reboot.

Yup that is definitely a bug!! There is no reason the DUID should not stick between reboots. In fact on my Edgerouter, I had to force a prefix change on TWC the other day when my PD block lost IPv6 connectivity, and I had to force the DUID change:
release dhcpv6-pd interface eth0
delete dhcpv6-pd duid
renew dhcpv6-pd interface eth0

 

[garyd9]

Open WRT current release "Chaos Calmer". I don't remember what packages were available, but I remember seeing for example a Snort package compiled for it, as well as people running proxies on it and such. But on your hardware, it screams to be virtualized (which was what I did). And gives you options for example to run Security Onion in another VM (though you;d want a probably want a bigger disk for your Snort logs).

I'll have to take a look. 100 GB isn't enough for snort logs? wow.. I have a 500GB spinner I could use in there, but the speed of an SSD (as long as it's not abused with writes) would be hard to give up. I reboot in just seconds...

Worked around the pfsense DUID bug by following the instructions in this post: https://forum.pfsense.org/index.php?topic=114390.msg640854#msg640854

It's a work-around, and if /conf (a symbolic link to /cf/conf) ever gets wiped, I'll lose the config, but it'll do for now. I also considered making changes to a couple of /etc/rc.* files to preserve it on shutdown and restore it on bootup (and perhaps even a cron job to keep it backed up), but those things wouldn't survive a firmware up either.

(Actually... now that I'm typing this.. if I use the cron package to back up the duid file to /conf, and the shellcmd package to restore it... that SHOULD survive a firmware update... hmm)

 

[Nullity]

Of course. If I wasn't open to alternatives, I'd have already reformatted the pfSense box and put something else on instead of engaging in this thread...

As for being the first person with the problem, I happen to know I'm not as I see many other people asking similar questions. The problem is that I can't find where any of those people are getting reasonable answers or solutions. I'm starting to wonder if some of them have just given up on pfSense and moved on to other software. (I don't know.)


Your implying that I'm not using common knowledge. How is that so? It is common knowledge that ipv6 can (and will) assign multiple IP addresses to a single interface. It is common knowledge that ipv4 only assigns a single one. It's also somewhat common knowledge that modern OS's will generate their own IPv6 addresses (and frequently change them!) for internet access traffic.

The post you quoted was me trying to approach the problem from a different angle. The problem is that I keep coming back to the same question: How can I block an interface?

What about this isn't using common knowledge?

Oh, and quite often it's the complete novice who breaks new ground - not the PhD student or professor. The educated ones have been trained to think inside the box and are afraid to ask questions outside of it. The novice isn't aware that the box exists, so is able to see outside of it. To use an old fable: it's the young ignorant child who points out that the emperor is, infact, naked.

Finally, this whole mysterious "if you aren't finding your answer, you're asking the wrong question" thing is really getting annoying. Seriously, this isn't some metaphysical thing. Redefining a problem in order to find an easier answer is NOT solving the original problem. I realize and understand that there might not be a good solution - but I'm not one of those people who will ignore a problem just because I can't find an easy solution.

The networking stack supports what I'm trying to do in the form of MAC filtering. I realize that pfSense can't do MAC filtering, and I'm willing to learn of reasonable alternatives. No one seems to be able to offer any (reasonable) alternatives.

This reminds me of the Apple iphone... some people had a problem with the early generations because they couldn't get a list of previous notifications. Those users would point out that a "lesser" product, Android, could do this with their notification shade. Apple told those users that this wasn't a problem. They said that the iphone was a much more advanced product and those users were asking the wrong questions. Then, out of the blue, Apple "invented" a notification shade.

Being that everyone tells me that I'm "asking the wrong question", I won't use a question. I'll use a statement.

The problem, in it's most basic form, is that I need a mechanism to block an interface from the LAN to WAN gateway in non-permanent ways on a fully dual-stack network. A similar problem would be that I need a mechanism to block specific interfaces from other specific interfaces - and that doing so with vlans/subnets is unreasonable due to the volume and mix of interface combinations.

My quoted response gave you less respect than you deserve. Sincere apologies.

The simplest thing I can think to say is that perhaps your text is not saying what you mean to say. On this and pfSense's forum, more than a few people feel like you are being (initially) confrontational.

I know I am confrontational... (conversation is a battle, right? ). It's perhaps my worst habit.

This post is off-topic so no response is needed. I just wanted to voice my apology along with some sort of observation or explanation. Regardless of perceptions, this thread has a useful goal and some good responses.