Stories wanted about working routers in virtual machines

[AdvHomeServer]

I'm getting ready to create a pfSense router in a bare metal Hyper-V server. Any reports from anyone who has a working router in a VM would be appreciated. Good or bad.

I had wanted to use esxi, but the J1900 bay trail processor in my pc is not compatible with it. Hyper-V is said to work both on the processor and with pfSense.

If successful, another VM will hold Win 7 pro or Win 8.1 Pro, accessible by TeamViewer run as a service. Common programs that can't run on all devices in the home will be accessible on the VM. I had planned on a Linux Mint VM, but instead the Linux Mint laptop will access the common programs, as needed. Linux Mint makes an excellent Win10 substitute for oldsters like me. It put new life in a laptop that was sluggish under Win 8.1 pro. Fedora and OpenSuse were tried and set aside ... too many odd nuances to be family friendly (as in "Why can't I do this simple ordinary little thing anymore?")

 

[sfx2000]

It's not recommended to run Routers in VM's... mostly due to performance issues along with security concerns.

If I recall correctly, pfSense is based on BSD, so any recipes on getting BSD running on HyperV should be equally applicable.

 

[AdvHomeServer]

It's not recommended to run Routers in VM's... mostly due to performance issues along with security concerns.

If I recall correctly, pfSense is based on BSD, so any recipes on getting BSD running on HyperV should be equally applicable.

Thanks.

I've read the same about 'not recommended' but anecdotally there's quite a few who do this. Hyper-v is bare metal with no gui and has a low exposure risk. Chicken and egg problems come to mind. Any specifics?

 

[sfx2000]

I think much of it has to do with the shared/virtual interfaces and shared memory from a security perspective - so if one is running pfSense for example in one domain, and another VM in another, someone could break one, and get into the memory stack/heap there..

From a performance perspective - the kernel is tightly optimized to talk to the network interfaces at a very low level, running on top of a hypervisor (Xen, HyperV, or higher level hypervisors like VirtualBox, VMWare Workstation, Parallels, KVM/QEMU) - virtual IO does have some overhead compared to metal.

Last comment - pfSense is not only acting as your router, but it's also the firewall - which is why I think many recommendations are "not recommended".

 

[AdvHomeServer]

I think much of it has to do with the shared/virtual interfaces and shared memory from a security perspective - so if one is running pfSense for example in one domain, and another VM in another, someone could break one, and get into the memory stack/heap there..

From a performance perspective - the kernel is tightly optimized to talk to the network interfaces at a very low level, running on top of a hypervisor (Xen, HyperV, or higher level hypervisors like VirtualBox, VMWare Workstation, Parallels, KVM/QEMU) - virtual IO does have some overhead compared to metal.

Last comment - pfSense is not only acting as your router, but it's also the firewall - which is why I think many recommendations are "not recommended".

Performance, yes but I have about 95% available capacity relative to it being a dedicated router.

Re security: pf sense is watching the door for the hypervisor. Yes, a dedicated intruder who knew hyper-v bare metal version was also at the door with pfSense might break through the crack, but is it realistic?

Re: getting past pfSense: Isn't getting past pfSense getting past pfSense whether or not it's a router or a VM? SPI and NAT don't go away if it's a VM. Does Hyper-V have security risks? What about esxi?

Binding the physical nics to the pfSense VM, would this help? Does Hyper-V override this? In this case, Hyper-V and all vms are subordinate to pfSense.

 

[sfx2000]

I think it would be fine as a science project perhaps, and some folks have done it with Xen - not sure if I would run it as "production" however...

 

[AdvHomeServer]

I think it would be fine as a science project perhaps, and some folks have done it with Xen - not sure if I would run it as "production" however...

Still wonder about binding the nics to the VM. Does that make everything subordinate to pfSense with respect to network communications. Especially compared to a $35 router with nat and spi.

By this, and still in theory only since I haven't done it yet, if the physical network is bound to the pfSense VM via Hyper-V and all other network communications is using virtual adapters, wouldn't pfSense be in control, subject to Hyper V not messing up?

For those with actual experience ... where does the hypervisor network connection come into play relative to what can be done in the VM? Can the VM take control and make the hypervisor subordinate or will the hypervisor always see the nics and ignore the configuration when it wants to?

This might answer it ... if pfSense has control of the physical nics, can Hyper-V update without regard to the pfSense vm? If 'No, because pfSense VM is down' then it looks safe.

 

[sfx2000]

I'm not sure with HyperV and how it would behave with pfSense with regards to binding a specific NIC, but it seems reasonable that one could bind a specific NIC...

I'm just wondering if instead of running pfSense on HyperV, perhaps run it inside KVM/QEMU, have you considered this?

https://doc.pfsense.org/index.php/VirtIO_Driver_Support

 

[AdvHomeServer]

I'm not sure with HyperV and how it would behave with pfSense with regards to binding a specific NIC, but it seems reasonable that one could bind a specific NIC...

I'm just wondering if instead of running pfSense on HyperV, perhaps run it inside KVM/QEMU, have you considered this?

https://doc.pfsense.org/index.php/VirtIO_Driver_Support

No, I don't really know what they are. I want to stay 'traditional'.

Basically, this is learn by doing. VMs are the coming thing even though they are still a little out there. Others have built VM routers, but security is not addressed clearly in the stories.

My plan is

1) take out the existing hard drive, in case it needs to be put back later
2) install a 2.5" 500gb drive not in current use
3) install bare metal hyper v and update, install client apps to manage hyper v from other devices
4) load pfSense VM and get it working with nics configured to be 'owned' by pfSense, if possible
5) test to see if hyper-v can access the network or the internet if pfSense VM is down
6) if pfSense is in charge and performance is acceptable (gigabit, not fast ethernet) load windows pc vm
7) if hyper v is independent, then put old hard drive back in and move on.

logically, if router is down, no aspect of the network could function, regardless of if hyper v is independent or not. But, I'll try to find the flaws if there are any. hyper v is hypervisor, not router. In theory, without a router, there's nothing to hack from the internet even if hyper is directly attached with a physical nic. hyper v is not on the osi model. In some respects, it's like plugging the cable modem into an alarm clock.

 

[sfx2000]

Basically, this is learn by doing. VMs are the coming thing even though they are still a little out there. Others have built VM routers, but security is not addressed clearly in the stories.

My plan is

1) take out the existing hard drive, in case it needs to be put back later
2) install a 2.5" 500gb drive not in current use
3) install bare metal hyper v and update, install client apps to manage hyper v from other devices
4) load pfSense VM and get it working with nics configured to be 'owned' by pfSense, if possible
5) test to see if hyper-v can access the network or the internet if pfSense VM is down
6) if pfSense is in charge and performance is acceptable (gigabit, not fast ethernet) load windows pc vm
7) if hyper v is independent, then put old hard drive back in and move on.

I'd give it a spin and see...

Remember with Security issues/concerns - unless public, most folks don't say much, and this is the danger... every hypervisor has been exploited at one time or another, and generally those bugs that are published are fixed... if you're running an OS on top of a hypervisor, now one has two levels of security concerns, and that is a 10Log increase in concern...

 

[AdvHomeServer]

I'd give it a spin and see...

Remember with Security issues/concerns - unless public, most folks don't say much, and this is the danger... every hypervisor has been exploited at one time or another, and generally those bugs that are published are fixed... if you're running an OS on top of a hypervisor, now one has two levels of security concerns, and that is a 10Log increase in concern...

I'll be running Hypervisor-v 2012 R2, which is the bare metal subset of Windows Server 2012 R2. It's character based, not gui. The attack surface is smaller.

I will be facing the same risks as anyone who runs this or any other hypervisor, regardless of whether it is or is not hosting a VM router. The main risk involved attack from a compromised client. Hyper-V role on Windows server notes several risks due to attack via a client system getting malware designed to attack hypervisior. Windows clients are the primary mode of attack / entry. The fact a client is a router is not material. A BSD based client is less likely to attack the hypervisor. The Windows VM is also a low risk since it will run only limited software and run an anti virus.

The issue of a virtual router is just a new and scary idea, the fear of the non traditional. Your QNAP home VM hypervisor carries the same risks, although linux altogether faces fewer risks than Windows based software / servers. Exposure of the hypervisor to the internet should be a small issue since the router VM is / should be managing the connection. Attacks, historically, do not do straight for the hypervisor. They don't get infected just by being there. They get to the hypervisor via an infected client VM.

The hypervisor without a router is like attaching a toaster to the internet ... unless I find out it has it's own router characteristics .. which is doubtful..

 

[sfx2000]

The issue of a virtual router is just a new and scary idea, the fear of the non traditional. Your QNAP home VM hypervisor carries the same risks, although linux altogether faces fewer risks than Windows based software / servers. Exposure of the hypervisor to the internet should be a small issue since the router VM is / should be managing the connection. Attacks, historically, do not do straight for the hypervisor. They don't get infected just by being there. They get to the hypervisor via an infected client VM.

No doubt - the security folks have been beating hard on KVM and esp. QEMU, and there has been a number of critical vulns found there, not specific to QNAP or QTS or the client OS. Xen is under similar attention as well.

Hypervisors are a target - as many cloud service providers use them for hosting on VPS's... so if they can get to the hypervisor, they can then attack the hosted VM's.

My QNAP has nothing facing the public, no services whatsoever... which is probably common practice with any NAS box. The VM is bound to it's own private interface, and only has two strong authenticated services to the outside world. I'm still taking a risk though, as the host is completely aware of that ethernet port, and of course, it hosts the virtual disk that the VM runs on. It's those common interfaces, and the fact that KVM/QEMU do run in the kernel, that is a security risk...

That being said - HyperV runs below the OS as a Level 1 (sitting between bare metal and the client OS) which can be more secure than the Level 2 provided by KVM/QEMU.

pfSense is very secure itself, not only with their code, BSD adds a level of security on top of that...

I said it before - give it a try.... would be interesting to see how it works.

 

[sfx2000]

'll be running Hypervisor-v 2012 R2, which is the bare metal subset of Windows Server 2012 R2. It's character based, not gui. The attack surface is smaller.

Just want to add a quick note - Hyper-V is still Windows with all the benefits (and risks) of Windows running on bare metal.

 

[Deepcuts]

Mikrotik has support for virtual routers
http://wiki.mikrotik.com/wiki/Manual:Metarouter

 

[YeOldeStonecat]

For the reason of "security risk'...I will never have an edge appliance/firewall on the same physical hypervisor host as my clients production servers. There have been exploits which hop across the hypervisor host...and if the edge NIC is exposed...the exploit can rip right through the host, and have full access to the inside...the guest instances...including production servers which you thought were protected. IMO..the role of the edge appliance/firewall is to stand between the product network..and the outside world. Different physical hardware...separate NICs...nothing in between but the firewall operating system itself.

Adding a hyper-visor host introduces a whole second potential gaping hole...if it has a NIC set to be the WAN NIC of the firewall. There have been exploits for VMWare..and that is, in theory, quite hardened. To stick a Microsoft Hyper-V host NIC out on the wild side..just shoot me now! No way I'm supporting that.