Home network configuration (PFsense)

[john9527]

A little late....but this guide helped me out...
https://nguvu.org/pfsense/pfsense-baseline-setup/

 

[L&LD]

@Val D. see post 8. If you can't keep track of how the conversation is going, stop asking questions you don't want to hear the answer to.

I know the answer. The question was rhetorical, because the goal wasn't achieved back then.

I asked for a tutorial if he gets it going. I was on topic. And still trying to be.

Don't throw red herrings about me supposedly trying to convince others about things. Nobody stated that here. Not interested in debating these facts again and my supposed inadequacy.

You're the one derailing the thread, once again.

 

[L&LD]

A little late....but this guide helped me out...
https://nguvu.org/pfsense/pfsense-baseline-setup/

That guide looks great! And updated just a few days ago too. Thank you for the link @john9527.

 

[Val D.]

You're the one derailing the thread, once again.

No problem, I'm not going to write here anymore. Sorry, @Discy for derailing your thread.

 

[L&LD]

@Val D. No need to leave.

Just try and stay focused and helpful.

I did 'like' your posts where you give helpful advice after all.

 

[Val D.]

Just try and stay focused and helpful.

No, thanks. Feel free to offer support to pfSense users as well, or explain how bad it is... whatever you decide.

 

[L&LD]

Taking your ball and going home huh?

 

[Discy]

Thanks guys! Let's keep it civilized .

@L&LD I actually don't mind getting myself a networking degree by online learning .
Networking is part of my dayjob when working with legacy applications that are being pulled to Azure so this is helpful on many levels!

My setup is stable enough for continuous use and streaming in past three days and I'm dealing with some latency. Let me be clear about where I stand now:

@MichaelCG

1. Get basic interfaces and routing functional
   Done
   - determine if you will need more than one subnet
   Not sure? Currently one.
   - determine if your guest will be done via pfSense or via your WiFi
   No guest network right now.
2. Get your DHCP scopes defined
   Done
3. Tweak your DHCP static reservations for your specific important devices
   Done

@Val D.

• run your modem in Bridge mode to avoid double NAT, if possible
  Done
• run pfSense as main router with DHCP server, default NAT/Routing/Gateways will auto-configure
  Done
• if you don't want to deal with manual port forwarding, enable UPnP in Services -> UPnP
• Done
• connect your TV to the main switch, you can't use your modem as extra switch
  Done (via WiFi)
• assign static IPs to your devices, x.x.x.10-90, automatic assignment IPs x.x.x.100-200, for example**
  Done
• Issue 1: you can do selective routing per device or alias, an example how here:
  https://support.nordvpn.com/Connectivity/Router/1136266682/pfSense-2-4-4-selective-Routing.htm
  [UPDATE: Pending - Will retest this after other changes I did today]
  Members of my VPN Gateway group go offline as soon as I enable "Don't pull routes".
  I've manually assigned gateway ip's to my OpenVPN gateways. They should be in the same subnet as my OpenVPN clients else I cannot save in PfSense (attached below).
  I get a different IP's (with overlapping subnets) each time the OpenVPN clients refresh. So I started to refresh the clients until subnets were unique and aligned the gateway IP accordingly and press save . Seems like a bad work-around......
  If I have the gateway ip's "dynamically" assigned as is default I run into some gateways not coming online - while OpenVPN clients are fine. I might need to fix something that I'm unaware about..
  Wanted to look at this https://nguvu.org/pfsense/pfsense-multi-vpn-wan/ but it's a followup on the guide also suggested by @john9527 so can't use it immediately as he uses VLANS for a lot of stuff.
  Planned to go through it today and try to pinpoint the issue I have with selective routing.
  [Update] One difference I notice is that the nguvu guide doesn't use "Don't pull routes" while NordVPNs does. In NordVPNs basic setup guide they leave it disabled - so they add it when dealing with selective routing.
  
• I don't know what is your NAS doing on Internet, but you can limit bandwidth with Limiters
  Done - It's for cloud backup (push / in»out)
• I would run a Limiter in order to minimize bufferbloat, example video in the link below*
  Done

---
Issue 2: Latency [UPDATE] Retesting - See my post below
So everything basically is in place and connection is fine. However, I'm dealing with some flutations in latency.
Websites load pretty slow and most of the time when I ping 1.1.1.1 the first request times out. I figured this might be because of some configuration problem - if it's not due to the VPN link.
Overall throughput is fine (100mbps - max of my line)

Pinging 1.1.1.1 with 32 bytes of data:
Request timed out.
Reply from 1.1.1.1: bytes=32 time=12ms TTL=59
Reply from 1.1.1.1: bytes=32 time=13ms TTL=59
Reply from 1.1.1.1: bytes=32 time=18ms TTL=59

Quality graph seems OK? All VPN connections are basically like this.


A wireshark-log of pinging 1.1.1.1 (pcapng). You can see the first ping times out. Is it useful for diagnostics?
Filter: ip.dst == 1.1.1.1


---
Issue 3: Nguvu guide

A little late....but this guide helped me out...
https://nguvu.org/pfsense/pfsense-baseline-setup/

Yes found it yesterday myself and thought about starting with it. But how does he assign IPs to devices? I emailed him about it . He said:
"Clients are either assigned through the switch port they connect through, or in the case of wireless devices, which SSID they connect to. Google policy routing to understand how to route specific devices or ASN's through specific gateways."

In my case, I have multiple devices connected to the same SSID, only the tv should go to directly to WAN without VPN. Do you know if/how this is possible? I would expect a rule with something like a Mac address?

--
Three issues mentioned in this post:

1. (Selective routing) Don't pull routes breaks VPN gateway connections [Update] Pending for now
2. Latency [Update] Re-testing
3. If I should go for the nguvu guide. Basically starting with a clean Pfsense. If so:
   1. How to route a single WiFi client directly to WAN (without VPN)
   2. He doesn't use UPNP because he doesn't trust it. Can it still be used in his setup?
   3. As I understand from him, he routes based on switch ports. Is it possible on mac-address for when I'd accidentally switch the ports?

Again - a lot of information at once. I feel it's better to put it all out in case things are related.
Feel free to choose and answer only where to start.

---
Issue 4: Debugging
Apart from looking for solutions I want to get my head around how I can debug these things.

Right now one of my Gateway connection has completly dropped while the OpenVPN clients are fine. Maybe the cause of this is causing more issues? Where should I look to debug this?

• Gateway log (csv - Excel)
• Routing log (csv - Excel)

----
[Update] Possible fixed: post below
Lastly..
Someone adviced me to move the LAN NordVPNGateway rule on top as the current prioritization wouldn't make sure that all traffic goes to VPN. Makes sense I guess, because "first matching rule wins". However, as soon as I do my Internet becomes very unstable:

• Pinging to 1.1.1.1 mostly times-out and in rare occassions doesn't
• OpenVPN clients are correctly initiated and connected + all gateways are online
• Internet does work with some website I visited rearlier (CTRL+F5), so I suspected DNS.
• DNS Lookup from PFsense works fine.
• Tried to solve it by limiting the Rule to LAN net
• Tried to add Floating rule from Firewall and * to WAN on the DNS servers
• Reset states and even restarted PfSense in between the attempts
• Tried Log packets that are handled by this rule and find them in System Logs > Firewall but they don't show up/rules are not hit.

All in all, no luck so moved the rule back down by reverting all changes. It is another good example of the instability I'm dealing with and my blanks on how to debug properly.
I assumed the Block all WAN rule would mitigate this problem anyway. Tested this by disabling all OpenVPN clients and trying to reach internet from my PC.

 

[coxhaus]

Interesting. I had a an issue where I picked up latency on DNS with my layer 3 switch. You have to setup routed gateways to use a layer 3 switch. It was not there when I first installed pfsense but somewhere after 1 or 2 versions of upgrades I picked up this latency. I never solved it as I went back to a Cisco router. This was several years ago running high end hardware. I was running a real Intel server motherboard with Xeons. Utilization was like 1% or 2%.

 

[Discy]

Interesting. I had a an issue where I picked up latency on DNS with my layer 3 switch. You have to setup routed gateways to use a layer 3 switch.

I thought it might have something to do with DNS and/or the configuration of Pfsense DNS resolver, but it seems fine looking at the screenshot below? Assuming I'm looking at the right place. What do you mean by "Setup routed gateways to use layer 3 switch"? My switch (TL-SG108PE) is a layer 2 switch, if that matters.

I was running a real Intel server motherboard with Xeons. Utilization was like 1% or 2%.

On 80mbps load via speedtest.net top -ash shows this output. Doesn't seem to be a CPU bottleneck in my case as well.

last pid: 42656; load averages: 0.35, 0.22, 0.16 up 1+10:21:47 12:47:11
462 processes: 7 running, 428 sleeping, 27 waiting

Mem: 14M Active, 153M Inact, 359M Wired, 172K Buf, 3211M Free
ARC: 146M Total, 29M MFU, 109M MRU, 4937K Anon, 529K Header, 2617K Other
45M Compressed, 112M Uncompressed, 2.49:1 Ratio
Swap: 2048M Total, 2048M Free

PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
11 root 155 ki31 0K 64K CPU3 3 33.8H 98.00% [idle{idle: cpu3}]
11 root 155 ki31 0K 64K RUN 0 33.5H 97.46% [idle{idle: cpu0}]
11 root 155 ki31 0K 64K RUN 2 33.8H 96.29% [idle{idle: cpu2}]
11 root 155 ki31 0K 64K RUN 1 33.7H 93.80% [idle{idle: cpu1}]

 

[coxhaus]

What do I mean by setup a routed interface for pfsense to work with a layer 3 switch. A layer 3 switch handles routing by itself so there is no layer 3 routing needed from pfsense. pfsense does not route the networks like it would for a layer 2 switch. All pfsense knows is for certain networks it needs to forward packets to the layer 3 switch. The way this setup works on pfsense is you create routed gateways with the forwarding IP to the layer 3 switch. So all my layer 3 switch traffic runs through the routed gateway on pfsense.

I hope this helps for you to understand. A layer 3 switch works the same as if you had a second router on your network. And I mean router not firewall. There should be less overhead for forwarding packets than routing packets.

 

[L&LD]

@Discy thanks for the concise and detailed overview of your pfSense experience up to now. I never managed to get that far because I was stuck with the issue you're still facing too (slower and slower networking 'latency') over a year later.

@coxhaus you also basically verified this latency issue with pfSense.

I know I wasn't imagining it. If a resolution can be found for this issue, that is when I may be tempted to try pfSense once more.

Looking forward to the next few posts and hopefully a quick resolution too (are the defaults not correct, is there something I/we missed?)!

@Discy I wouldn't mind (trying to) get that online networking degree either with your help and the civilized participation of others too!

 

[coxhaus]

[USER=5292]@coxhaus you also basically verified this latency issue with pfSense.

I know I wasn't imagining it. If a resolution can be found for this issue, that is when I may be tempted to try pfSense once more.

Looking forward to the next few posts and hopefully a quick resolution too (are the defaults not correct, is there something I/we missed?)!


[/USER]

[USER=68359]
[/user]

There is no way you had the problem I am talking about so yes you are imagining it. The issue I reported had no problem with moving lots of data using speedtest. Yes I reported it pfsense forums.

 

[L&LD]

@coxhaus, okay, if you say so.

 

[Discy]

I know I wasn't imagining it. If a resolution can be found for this issue, that is when I may be tempted to try pfSense once more.
Looking forward to the next few posts and hopefully a quick resolution too (are the defaults not correct, is there something I/we missed?)!

Not sure if my issue will be the same for all of you. There are so many possible settings and setups that may come into play.
When I allow direct WAN for all devices to circumvent my custom stuff (other interfaces/gateways/rules/VPN) there are no latency issues for me.

Let's wait and see

 

[coxhaus]

I agree. My issue was related to using a layer 3 switch. What was interesting is you are seeing latency on a routed gateway which is the way you interface a layer 3 switch. I don't know whether my issue was related to DNS or the routed gateway because if you don't use a routed gateway there was no problem. And it did not happen under 2.1 version I think. It was a while back. I think it started with version 2.2.

PS
The more I think about my pfsense issue it may have been version 2.0 to 2.1. And I think I may have ruled out DNS because caching did not fix the issue. But It has been a few years.

 

[Discy]

I created seperate NAT rules for each OpenVPN interface instead of just one for all.
Am now able to move the LAN to VPN rule up and latency seems to be greatly improved.
No more ping issues as well.

Don't want to judge too quickly - have to test for at least a day to be sure. Did reboot PFsense.

Would also still be very interessted to know how I could have found it in the logs.

Let me re-test some of my mentioned issues like selective routing. As this might have impacted their results as well, unless there is already something to add.

 

[Discy]

Hi guys,

All before mentioned issues are solved . Main culprits were:

• NAT mapping should be done for each OpenVPN interface instead of using the "OpenVPN" option. This solved latency and instability.
• Getting FW rules in correct order
• Use "Tracking ID" to debug rules in Firewall logs

Also:

• My floating rules made things rather complicated. To make sure traffic never goes over WAN:
  • I replaced them by some rules on LAN including a default deny.
  • Enabled System > Advanced > Miscellaneous > Gateway Monitoring > "Skip rules when gateway is down" as explained here
• Did keep floating rule for Bufferbloat

Have a good sunday!



----
[04/12/2020]

• QoS per application is available as well, in case you want to play with it
  Done
• pfBlockerNG for IP/DNS-based blocking
  Done - Some NordVPN IP's were in the list (getting better at debugging )
• ntopng for network stats
  Done
• Snort/Suricata - ISP/IDS
  Skipping this for a while as adviced by @MichaelCG
  Also disabled and blocked IPv6.

Seems we've got it all covered for now! Thanks!

[04/13/2020]
Had to add a CoDel limiter on the VPN gatewaygroup as well see this post.

 

[coxhaus]

Good.

 

[Discy]

Final setup looks like this. Tried the AP outside the closet but it didn't improve the range for all rooms on this floor. The closet is surrounded by wood (no concrete) on all sides and located exactly in the middle of the floor.