What's new

Confessions of a pfSense Newbie ...

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

Dennis Wood

Senior Member
EDIT: (Nov 18) With our draytek 2950 routers locking up our network the last few days, the decision to insert the new pfsense 2.0 routers was made. They are fast, comprehensive and incorporate antivirus, proxy, content filtering, SNORT (instrusion protection) and even VPN. So far, the performance improvement over the draytek 2950 routers is impressive.

Some of you may have recalled a whole lot of testing and tweaking to get the two Draytek 2950G routers (see the sticky posts in this section!) configured and working with two WAN connections, VPN, RSYNC between several NAS units, and Iphone VPN. So things have been stable/good but there were a few things that bothered me about the Draytek units:

1. The Smartmonitor software (been working very well) must be run on a seperate Windows XP workstation. Well, it's time to retire that box.

2. I could never properly prioritize SKYPE packets which we use a lot.

3. At times I had some questions about how well the QOS was working with our VOIP systems. For sure, the faster speeds now provided by our ISPs are pushing the total routing performance of the Draytek units.

4. The Draytek boxes are unable to provide proxy/caching features..an easy performance boost on any network.

So after much research, PfSense (open source distribution) was an obvious path to a very high performance router that does everything the Draytek did (and a lot, lot more)but for about $350. The need for the external XP workstation for web monitoring/restricted access is gone, and we've gained proxy caching performance for the network. This all ends up in a box simiarly sized to the Draytek 2950 router box...but much quieter. So here's the chosen path, presented here so it would save a few folks a lot of time!

This article here is an excellent read before you go on: http://www.smallnetbuilder.com/secu...1406-build-your-own-ids-firewall-with-pfsense

The Chosen Hardware

Criteria: Low power, low cost, quiet operation, and compact footprint, but with minimum of 1 LAN Gigabit port, and 3 WAN Gigabit ports. Target performance is in the 1Gbps for firewall throughput, 70Mbps (Similar hardware, iperf measurments: http://www.hacom.net/catalog/mars-openbrick-m-pfsense-appliance )

M350 Mini-iTX Case with silent power supply: http://www.mini-box.com/M350-enclosure-with-picoPSU-80-and-60W-adapter ($69)

Intel Atom Dual Core 1.6 GHz Mainboard: http://www.mini-box.com/NC92-Intel-N330 ($109)

Daughterboard (adds 3 more Gigabit ports): http://www.mini-box.com/VERSA-3-x-Gigabit-LAN-Port-Daughterboard ($48)

1GB DDR2 memory: (old stuff I had on hand..usually $20-30 new)

Intel 40GB SSD drive: http://www.ncix.com/products/?sku=59822&vpn=SSDSA2CT040G310&manufacture=Intel ($100)

So purchased in bits and assembled yourself (add in some shipping), and we're looking at about $400. We won't talk about the time it took to sort out a few issues...several of them classic bone-head moves on my part. Here's the basic skinny on getting it working.

Installing Software/Hardware

1. Assemble the motherboard, plug in the daughterboard, install ram, install the HD, make your connections and fire it up. This was the simple part...many online tutorials on mini-itx builds you can refer to.

2. Attach a monitor and keyboard to the box and boot er up. Set up the BIOS to look to the hard drive first. Assuming everything boots (no OS yet..don't worry) then onward.

3. Prepare a USB memory stick to boot pfsense and install it to the SSD drive. I had a USB 8GB stick...refused to boot. An older Kingston 4GB Datatraveler..yes.
a) Download the "pfSense-memstick...amd64...img.gz" you will find on one of the mirrors here: http://www.pfsense.org/mirror.php?section=downloads
b) Use 7zip to decompress the .gz file to a .img file.
c) Use Win32DiskImager.exe to write .img file to your USB stick. You can download the utility here: https://launchpad.net/win32-image-writer/+download

4. During boot, hit DEL key and enter BIOS. Set box to boot from USB key (in BIOS set hard drive order so USB stick is #1 priority, make sure 1st boot device is hard drive)

5. You'll see some choices come up during the boot. It's important that you choose option 3 "Boot pfsense from USB"

6. From there, the wizard will ask you questions. You can use this as a guide: http://doc.pfsense.org/index.php/Installing_pfSense

7. Once you're done choosing LAN and WAN interfaces (hint re0, re1, re2, re3 are the port names looking from left to right, then choose 99 to install pfsense to the SSD hard drive in the box.

8. Choose easy install, and when asked which kernel to install, choose default multi-kernel.

9. When done, the machine will reboot and stop at a console. Here you can change the LAN IP address (hint: most times your bitmask will be "24") to something other than the default of 192.168.1.1 if required. Once this is done, you can disconnect the keyboard/monitor etc. as everything else can be done from the web interface.

Use this to set up load balancing:
http://forum.pfsense.org/index.php/topic,28121.0.html

Problem: IPSEC Passthru did not work. VPN connections outbound were established but nothing was routed over them once the load balancing (two WAN connections) was set up. This one was frustrating!!
Solution: System, Advanced, Miscellaneous, Enable "Use sticky Connections"

Port Forwarding, for RSYNC Use this:
http://doc.pfsense.org/index.php/How_can_I_forward_ports_with_pfSense?
In troubleshooting remote rsync inbound (requires rsync port and encryption port forwarded to host NAS) I wasted several hours because the default gateway on the NAS was set to the old router address!! Bonehead mistake!

Dynamic DNS on interfaces, set up under SERVICEs, Dynamic DNS

Packages

This is the cool stuff. Under the System menu on the pfSense router, you can very quickly install available packages from the web with one click. My picks:

EDIT: As at Jun 21 SNORT is not working with RC3 release :-(

a) Improved security provided by SNORT which adds literally hundreds of rules that are updated daily via subscription from snort.org
b) A high performance proxy system using SQUID which caches content on the SSD drive so frequently requested data, windows updates etc. come from the cache..not the web)
c) Advanced content filtering via SQUIDGUARD allows you to filter/restrict web content based on a ton of criteria
d) Finally LIGHTSQUID provides a reporting facility to check on web usage/bandwidth use etc.

Once you install the packages they show up as items in the SERVICES and STATUS menus. Each package requires configuration/setup but they are relatively easy to figure out using guides provided in the pfsense forums...in the RC2 versions of pfsense much of the context help is not finished yet so you're left to the forums for information. More on this later as we figure out VPN server setups to replace the Draytek 2950 IPSEC server which allows both 64 bit windows (using Shrewsoft VPN) as well as iPhone VPN. Pfsense provides similar functionality.

My overall thoughts after messing around for 3-4 days is that pfSense is incredibly well featured. The traffic shaper wizards worked great so I was able to set up VOIP priorities etc. with a few clicks. Once set up, the ability to look at queues as well as pile of graphs etc. pretty much tells you exactly how traffic is being routed and load balanced. Going forward, the ability to filter packets at Layer 7 means the router can inspect the contents of packets not just the headers which gives you all kinds of power to block, or prioritize just about anything. This includes SKYPE which is almost impossible to deal with due to the constant changes in ports. Routers like the Draytek 2950 do not have the horsepower to do this...but our $400 pfSense router does. More to come.

Whew...enough for tonight.
 
Last edited:
Long time, no hear. Welcome back and thanks for the post.
 
My pleasure sir :) I still manage to sneak some time to play with the network.
 
Greg, yes very helpful. I added links in my orlginal post as those great articles was part of the reason I decided to build the routers.

The one problem with pfSense is the lack of documentation like the series linked on this site. Looking at the pfsense forums can be daunting as there are so many threads, and version 2 is new...therefore one can spend a lot of time digging. Just getting the boot working from USB was a bit of a chore as again, there is no "idiot's guide" to creating a bootable USB for pfSense. Having a bootable USB stick would be important in case you have a hard drive crash or similar. One thing I learned very well over the last few years is that having system backups combined with a proven disaster recovery is very important these days. I always build/buy in pairs so actually will have two pfSense routers going at two sites, just in case one fails.
 
The one problem with pfSense is the lack of documentation like the series linked on this site.

I saw you were using RC2, have they resolved the issues with Snort? the SO rules were failing, invalidating a bunch of dependent rules.

I like the new multi-wan setup, and the ability to do fail-over, but without Snort for security (a compelling reason to go with pfsense) I'll wait until it is solid.

I use the old equipment that was replaced as a redundancy, tho with PFSense and Cerberus they have been absolutely rock solid ( knock on wood )

I agree on the doc, but I've been writing this series on openfiler where the only free doc is the install map. And unlike Openfiler, there appears to be an active support forum. If you look at the Openfiler forums the only responses you get to having a problem are guys who say "me too"

PFsense is outstanding in comparison to so many other products, just wish they develped a little faster :)
 
Yes, RC3 now and yes, I discovered today the Snort issues :-( Snort is not working for many folks (including my installs) currently, however the next version of snort is apparently right around the corner.

So a few points to add:

1. Multi-wan setup has changed which was quite confusing. The procedure in:
http://www.smallnetbuilder.com/security/security-howto/31451-build-your-own-utm-with-pfsense-part-2 applies to the older versions of pfsense.

For the new version RC1 (now RC3), the suggested method is using the link I posted: http://forum.pfsense.org/index.php/topic,28121.0.html

This itself is a bit confusing as RC3 (as of yesterday) has the load balancing menu options still available, it's just that this method of multi-wan is no longer recommended. So in terms of the PITA factor, pfsense is certainly nowhere close to the slick Draytek 2950 setup. That said, I'm not turning back as with a bit more time version 2 of pfsense and Snort will get along just fine I'm sure. The whole reason for going with version 2 (not an official release yet) was the promise of multi-wan support for snort, squid etc. No question though that getting a proxy server, Antivirus, content restrictions, IP blocking and reporting all on one little box for $400 is pretty amazing.

For now, the pfsense boxes will not go live until Snort works as it should. The Shields up site returned a perfect "score" scanning my current pfsense setup (which is nothing too amazing) but still it's all good. More tweaking to come. I'm learning a lot.

Greg, are you author of the UTM series? If so, the information was excellent and unlike the pfsense forum...serial, therefore made sense to follow along. It's the best resource I've found to date.

Cheers,
Dennis.
 
Yes, RC3 now and yes, I discovered today the Snort issues :-( Snort is not working for many folks (including my installs) currently, however the next version of snort is apparently right around the corner.

So a few points to add:

1. Multi-wan setup has changed which was quite confusing. The procedure in:
http://www.smallnetbuilder.com/security/security-howto/31451-build-your-own-utm-with-pfsense-part-2 applies to the older versions of pfsense.

For the new version RC1 (now RC3), the suggested method is using the link I posted: http://forum.pfsense.org/index.php/topic,28121.0.html

This itself is a bit confusing as RC3 (as of yesterday) has the load balancing menu options still available, it's just that this method of multi-wan is no longer recommended. So in terms of the PITA factor, pfsense is certainly nowhere close to the slick Draytek 2950 setup. That said, I'm not turning back as with a bit more time version 2 of pfsense and Snort will get along just fine I'm sure. The whole reason for going with version 2 (not an official release yet) was the promise of multi-wan support for snort, squid etc. No question though that getting a proxy server, Antivirus, content restrictions, IP blocking and reporting all on one little box for $400 is pretty amazing.

For now, the pfsense boxes will not go live until Snort works as it should. The Shields up site returned a perfect "score" scanning my current pfsense setup (which is nothing too amazing) but still it's all good. More tweaking to come. I'm learning a lot.

Guilty as accused.

Your reasoning follows mine exactly - except RC1 came out just as I was writing the second series, which was a bit of a bummer for me, all that work on a soon to be past version - I want multi-wan that works with Snort/Squid&Guard/HAVP, but without Snort working I'm going to wait. Its that line from Dr. Strangelove, "What use is a doomsday device if you don't tell anyone about it?" What use is a security appliance without IDS?

Yes, PFSense is amazing for the cost - just those last few things are needed (isn't that always the way...) I'm not familiar with the Draytek, expensive?

I think in the summary of the last part of the UTM series I point out RC1 was not ready for prime time, but did add a different, supposedly better approach to multi-wan, one that worked with packages.

As part of the UTM series I did a whole section on getting SpamD working, which also shows you how to bring up other BSD packages that are not in PFSense, like LaBrea and HoneyD, exposing some internals. It also worked with RC1. Since RC1 came out with a (older) version of SpamD included, and it was quite dense, we bypassed publishing it. If interested PM me with an e-mail address.

Once 2.0 goes into general release, I am hoping to do a series on Honeypots, like HoneyD and LaBrea. Making PFSense more passively offensive to scanning, I think would be quite cool. I'm sure your multi-wan write will be useful then.
 
Last edited:
Greg, there's a few sticky posts in this forum area about the Draytek 2950 units that I've been playing with for some time now (2 of them). They offer some of what pfsense does, namely dual wan load balancing, IPSEC/SSL VPN, content filtering, and if you're willing to hook up an external XP box, comprehensive monitoring reports etc. using an Apache web server/packet sniffer etc.

They were $450ish if I recall correctly. Feature for feature though, pfsense pretty much beats the 2950s up (and adds many features like caching, all in one little box)..providing you're willing to endure the learning curve.

What I like about your article series is that for many newbies, it takes days (weeks for some) out of the learning curve. I have to admit it's kind of cool putting together UTM ish routers that match one's needs pretty much exactly. Time permitting we should work up some cool bits incorporting video etc.
 
After a week or so of messing around with pfSense, I'll admit I'm torn.
Pfsense RC3 load balances with multiwan just great...until you install packages like SQUID, HAVP, or SNORT. There are pfsense forum posts suggesting work-arounds but right now but there appears to be very little success...I can't recommend using pfsense other than as a load balancing router yet. Using two boxes would be the only way to do this where box 1 is doing the load balancing on your WAN connections, and box2 (LAN side) is doing SNORT (stateful packet inspection, SQUID (proxy server) and HAVP (antivirus). So we're back to two boxes, and therefore, the Draytek 2950's are still in the running as a good pick. The reporting on WAN use in Draytek's Smartmonitor is much better than Squidlight as it covers more than just http (POP, IMAP, VOIP etc.) traffic. It also does this whilst load balancing.

So perhaps two pfsense boxes at this point (we're up to $700) would be a lot easier to work with. One dedicated to load balancing, the other to SPI...not ideal.
 
2.0

This was my conclusion on Version 1.23, not ready for the enterprise yet - I still have hopes for the final release of 2.0, everything is there just not yet gelled.

Snort gotten better? I looked and them seems to be some momentum....

Thanks for the update.
 
Last edited:
Snort...nope.

I did have pfSense running on two boxes as a test. The first doing SQUID, AV, and SQUIDGUARD, the 2nd only load balancing. The setup was LAN -> pfSense box 1(proxy, AV) -> pfSense box 2 -> WAN1 +WAN2.

For whatever reason, having SQUID on box 1 seemed to kill load balancing on box 2...like it was forcing the same outbound WAN connection. Given I was using utorrent to test load balancing on the 2 box setup (doesn't use port 80) I was a bit puzzled as to what was going on. Hmm.

A big surprise though was testing load balancing (the single box setup, only ntop installed) using utorrent. I don't really care so much about torrent performance however it's a good test as it grabs a lot of connections and seems to stress out routers with the number of connections. After pumping up a few settings in utorrent so Global Max number of connections increased to 1500 and connected peers/torrent to 300, I was literally stunned to see speeds of 3.3 MB/s ((~26 000 Kbps) which meant a 700MB file was downloaded in 10 minutes! This was coming from 0.8MB/s WAN1 and 2.5 MB/s WAN2 connections. Any router I've tested before like this simply choked on the connections.
 
Last edited:
2.o etc

Weird, Box 1 shouldn't affect upstream load balancing.

Do you have "Smart" connections turn set on box two? It is possible box two sees box one as monolithic connection ( routing of the same IP...), and hence routes through the same ISP. Try turning it off (may cause other problems....). Have to think of a way to have BOX 1 transparent, hmmm.

I presume you are running Squid transparent?

On Utorrent. Yeppers, the state table is not not limited like it is on regular routers, you have more memory, you can have a bigger state table. Why cheap routers suck, fixed or limited state tables. Why it is a good idea not to run Verizon's FIOS router, known real limited connection limit.

I saw the same thing with threading and JPerf against PFSense, you don't cap out the number of connections and start dropping them like a regular router.

Does 2.0 seem snappier than 1.23?

I really want to switch over, darn.
 
Squid transparent yes. Sticky connections..yes it's turned on as otherwise my VPN IPSEC connections don't work. I get VPN connections, but the IPSEC Security associations fail. It took a bit to figure that out as the Shrew VPN software indicates a connection..it's just that nothing is routed over the connection. That said, load balancing works just fine without SQUID if I remove it (from box1) in the two box setup. Greg, I've never run anything previous to RC2. What I can say is that a clean RC3 install with just ntop running shows very low latency. So yep, snappy.

One thing that I didn't spend enough time on is port forwarding from a two box setup. Here's the setup:
The internet facing (load balancing) box has a "LAN" ID of 192.168.10.1
The LAN facing box (SQUID) has a WAN ID of 192.168.10.2, with a its "WAN" gateway set to 192.168.10.1. It's LAN IP is 192.168.0.1.

So in a two box setup, the LAN subnet of the web facing box no longer matches my internal network, therefore port forwards on the load balancing box don't work. Other than another port forward on the LAN side box, suggestions? Ideally I'd only have one set of firewall rules on the WAN side box..less to debug that way. I'm not an expert in subnetting, but perhaps I can get everything working on 192.168.0.1 using bit masks.
 
Well a few more updates on the experiments here.

1. I've been building, tweaking and playing with two different boxes (as per my first post). pfSense is very quick to install on the SSD drives via a USB bootable flash drive, so it's been very easy to wipe out, install, and start again. Using Pfsense 2.0 RC3 I've settled back in on a one box solution, and quite frankly the SQUID load balance issues aren't as bad as I thought. Yes, all port traffic ends up on your first WAN connection so port 80 traffic is not load balanced. However everything else is (so Torrents for example still zing on both connections). That said, you'll likely find in a real network that you need to direct traffic a lot more directly particularly if you're using web based apps. Web servers drop connections after their set limits (usually pretty short) and if you bounce back on a different WAN IP (load balancing), your session is generally dead..so the web server "insists" on a new session, even with pfSense sticky connections enabled). Posting here for example doesn't work as the forum software on Tim's web host sees a new IP and requests you log in again..if you're working on a long winded post like this, you lose it.

So one might think this is a bad thing. Looking over the Draytek load balancing set of rules we're using "live", it became apparent that I was more or less directing IP's to WAN1 or WAN2 anyway...as load balanced connections irritated the daylights out of everyone.

2. I picked up a $25 inline power meter to check on power use. The Draytek 2950G uses 36 watts..the pfSense box I built, 30 watts. We're working on a new net-zero builing for Cinevate, meaning by using pfsense on one box we're not only saving the six watts, but also another 60 or so by not having an Windows XP box running Draytek's Smartmonitor software. That's a fair bit of power considering the boxes are on 24-7.

3. I'm officially a big fan of the pfSense HAVP antivirus install sitting at the network gateway. Why? It immediately identified a worm that was masquerading as an Adobe or Java update. This one is very clever (Trojan.Fakealert.Sesh) as it appears to the user as a normal application update. Because it entirely replaces the .exe files in question, it does not trigger an alert from programs like MS Security Essentials. To test HAVP by the way, this EICAR virus test link works great: http://www.eicar.org/85-0-Download.html Try grabbing a file and HAVP returns a message via your browser. Very cool. Given what I can only guess is a staggering number of infected computer's worldwide, gateway AV makes a lot of sense.

4. The proper load balancing setup for pfSense is much simpler than the forum link I had used previously and many forum posters are posting complex setups that are not required. Basically, via System/Routing you just edit the default gateway check box (so it's not default anymore). If one of your WAN connections is faster than the other, you can weight them here too in the ADVANCED section. Then, add one group via the Group tab ... I called my group LOADBALANCE. Both WAN gateway priorities get set up in this group as Tier 1, and you're almost done. Under Firewall/Rules, you select the LAN tab and edit the rule there (by default) that allows all outbound LAN traffic. Set the gateway to LOADBALANCE (or whatever you called your group) in this rule. Done. Both load balance and failover then work as advertised.
 
Last edited:
I've been a big fan of PFSense since its early days. Very fast, and great QoS features.

I've not been a big fan of the clamav plugin....I've seen it let too many things through. Clam does OK for legacy viruses on SMTP mail scanners....but it's horribly weak for Windows threats. The Eicar test...every antivirus vendor out there is aware of that test and ensures their product will catch it. I don't think any AV product has ever failed that test...unless it's install is somehow corrupted on your system.

I am a HUGE fan of antivirus at the gateway though, as an additional layer of protection. Have found other products more effective at this...true UTM products.
 
What's your favorite?

I'm not hedging any bets on the gateway AV, particularly as it's useless at IMAP encrypted email etc. However, if something nasty ends up on a workstation, then we've got:

1. The workstation's AV which is updated daily/scanned nightly.
2. SNORT
3. HAVP

Currently really we only have one of the three ... until I bring the pfSense boxes live at all locations. The Eicar test link I posted is just a simple way to make sure the package is working...not that it is effective. You're 100% correct in that even crappy AV engines should catch the file as it's a standard test. It's amazing how many threads there are with the question, "How do I know if ____ package is working?" With AV it's as easy as visiting the EICAR site. With SNORT, doesn't seem as clear.
 
Last edited:
Agreed, EICAR resolves the question, is it working, not much more. Clamav does fail the double embedded test, but what do you expect for a non-blocking scanner?

I found that shields up does a pretty good job of answering the same question for SNORT.

I've yet to have a problem with HAVP on my front door, but think individual AV on each node is also required. In this config, HAVP on the perimeter and AV inside on each node you get active/active scanning, without any of the conflicts of doubling up on AV on each node.

You can have all the tools in the world though, if you are not paying attention, things will ultimately get by you.
 
Hi,

is possibile using pfsense to share an ethernet printer (linked to the pfsense box) in a network where there are multiple vlans created using a layer 2 swich?

regards

Ugo
 
Hi,

is possibile using pfsense to share an ethernet printer (linked to the pfsense box) in a network where there are multiple vlans created using a layer 2 swich?

regards

Ugo

If you can write the routing rules, pfsense can do it....
 

Similar threads

Latest threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top