Correct way to set up multiple Internet connections on 2 routers with managed switches

[TanyaC]

I have a 100/40 fttp connection with ISP A This is on an ASUS RT-AC88u (We'll call this Router A).
The switching infrastructure comprises 2xMS510TC 10G multigig switches and a bunch of Netgear GS108TV2
I have a PC setup dedicated for testing various things.
Right now, I'm testing a connection with ISP B. This is using an ASUS RT-AC68u (Router B).
The IP address of Router A is 192.168.x.254.
The IP address of Router B is 192.168.x.154
The test PC is set up with a static address to point it to Router B (Gateway=154).
The Test PC connects to an MS510TX switch, which has a gateway of 254 configured.
This switch is then connected to the downstairs switch which is a GS108Tv2 which is likewise configured with a gateway of 254.
A cable comes from that downstairs switch and goes to Router B.
The idea is that all other people can continue to use ISP A whilst I test the performance of ISP B.
But I am getting upload speeds of 0.01Mbps all of the time.

Should I just be running a cable down the stairs to Router B from the test PC, or is there another way to set this up?

I'm assuming what I have set up is not correct.

I've attached an image of the network looks like, sorry if it's a little busy.

EDIT: I just plugged cable from Test-PC to router B and disconnected the router from downstairs switch. Upload speed still 0.01 Mbps. ISP B problem I suspect.

 

[dosborne]

Although it would seem that you have isolated the issue, I must ask if you need the different subnets, particularly in the testing phase. You could simplify things easily as unless you have some other reason, what you have and are testing, based on what you've written, by using a single subnet. Just change the default GW to switch between ISPs.

Not that it would necessarily resolve anything directly, but would simplify things potentially for testing.

 

[TanyaC]

The problem turned out it wasn't an ISP issue, but Receive Segment Coalescing. When enabled I couldn't upload at all. Disabling the setting on the 10G adapter resolved that issue. I figured it out when I connected a 1G NIC and it was fine, then the 10G NIC direct cabled to NTD wouldn't even connect to ISP at all.

But I am still unsure of my configuration.

With two routers plugged in I will get multiple connections on PCs - a connection to both routers. For example, my Win 2012 R2 server picked up a connection to both router, which broke IIS and MySQL (The server has 3 NICs. The 10G NIC is the primary. Others are disabled, but I still got a connection to both routers).

The idea is that all PCs use DHCP and that I do not have to manually configure any IP details manually, except on the Test PC.

I've disabled DHCP on the second router. (.154, Router B).

But the switches still point to .254 (Router A). I don't understand the impact of this.

So, to test ISP B I've manually configured my IP settings on the test PC. And I've plugged it back into my network so I can access my server.

I'm concerned I have a big bloody mess.

I also seem to have a DNS leak, which I will create a separate post for in the Merlin sub forum.

 

[dosborne]

With two routers plugged in I will get multiple connections on PCs - a connection to both routers. For example, my Win 2012 R2 server picked up a connection to both router,

Not sure what you are saying here.

Computers don't "pick up" connections. They are all distinct items on your internal network(s). Being able to access all the systems, including multiple routers, on a subnet is the whole point and is my point in making the setup and testing easier.

If you assign addresses within a single subnet, then they can communicate directly with each other. The only difference with multiple routers is picking (or assigning) which one will serve as the default gateway.

As for dhcp servers, it is best to have only one act as a dhcp server, but technically there is nothing stopping you from running multiple. You don't want the assignment ranges to overlap or you may get duplicate ip address issues? You may have to assign the default GW manually, etc.

But, if a dhcp enabled client +your pc in the example) requests an ip address, then whichever ONE dns server responds first, wins. The pc won't be assigned morec than one address.

Simplest scenario: Router A is 192.168.0.1 (or whatever subnet you choose). It is your primary and uses whatever dhcp server range you want, let's say 192.168.0.25 to 192.168.0.225. All your systems use dhcp and are assigned addresses from this pool. In this respect, this is your "old" working environment. Wan connected to isp a.

Now, you add a new router to "test" with. Assign an ip, 192.168.0.2 (or anything that is unused and outside the dhcp pool of your primary router). You connect the wan to isp b. No impact on your main environment.

Done. Just set the default gateway of your test pc to either router a, 192.168.0.1 to use that router and isp a, or set the the default gateway to router b, 192.168.0.2 to use isp b.

 

[TanyaC]

All PCs get their IP addresses from reservations set in Router A. I do this to guarantee each PC is always allocated the same address.
Router A had DHCP enabled, as did Router B, which I've since disabled.
Default GW and DNS are not specified on any PC. It's all automatic.
I gather the Default GW is allocated from the Router that offers the IP address.
And I gather that the switches play no part in which router the data goes to regardless of what they are configure with.
So if the IP address is offered from Router A but I manually configure the GW on a PC the data will go out router B and DNS servers would be determined by the DNS entries defined in router A (Since I've not manually overridden the DNS setting on the PC)?
And since this is all on 192.168.1 is doesn't matter.

I will see if I can replicate the issue I had with the server and post a pic.

 

[Trip]

@TanyaC - Simplest way to set this up is to just isolate the two networks from one another at Layer 1 (physically). If you can, hard-wire from the AC68U (your test router) directly to your test PC; otherwise, if you kept the current connections as-is, you're going to have to segment your network at Layer 2 with VLANs, all the way to the Test PC, by setting the downlink port on the Asus-connected GS108Tv2 and uplink port the MS510TP (connected to the GS108Tv2) as trunk ports, both set as tagged members of each VLAN (one VLAN for your main network leading to/from the AC88U, and the other VLAN for your test network to/from the 68U (test router).

If you'd like to keep your cabling as-is, then VLANs will have to come into play. The good news is, you have the managed switches in place to do it. If my description read like a foreign language, then you're going to have to do some learning up on VLANs and subnetting, and how they work.

 

[TanyaC]

VLANs is a better solution than running cables. Though, setting up a couple of VLANs for one PC for the occasional time I do this seems more work than is needed, though I guess I only have to set it up once.

The routers are downstairs. There are no spare ports on the walls so I have to run a cable down the stairs which people keep tripping over. However, I only do this stuff once in a blue moon. If I'm doing it at all it means I'm having problems with my current ISP and I'm testing a second one. If we had good ISPs here then I wouldn't have to keep looking around. However, I've exhausted all the options so if the new one doesn't work out there is nowhere else to go.

The problems I was getting (of course), are PCs getting an IP address from each router where multiple NICs exist (which is half of the PCs here). The test-PC actually has 4. Using DHCP reservations just confused things because the MAC address for the test-PC was configured in both routers. So I have removed the DNS reservation from both routers and configured the NIC with the specific IPv4 details. It's not ideal, but at least it seems all traffic from the Test-PC is routed via router B (154), and I can still access my local network, including the server,

 

[Trip]

Setting up VLANs is really not that much work. All the config would be done on the switches (no endpoint config necessary), and since they're all Netgear, a duplicate-able process. Provided you configure the VLANs properly, you will have proper Layer 2 segmentation, which will keep your main network and test network separated into completely different broadcast domains, all the way to each Asus router, and back.

To properly isolate each network, you should assign a unique VLAN to each network, and a corresponding Layer 3 subnet to each VLAN -- example: VLAN 10 for your main network, with a subnet of 192.168.10.0/24 (this is CIDR notation for a subnet mask of 255.255.255.0 and usable IP space of 192.168.1.1 to 192.168.1.254), and VLAN 20 for your test network, with a subnet 192.168.20.0/24. Note: the third octet of each network doesn't have to match the VLAN ID; I just did that to keep the VLAN/subnet pairs more easily identifiable.

Then, each separate network will have its own distinct gateway (using the example from above, the AC88U would be 192.168.10.1, and the AC68U would be 192.168.20.1), and each router (gateway) would then offer a completely separate set of IP services (DHCP, DNS, etc.) for each subnet. At that point, each network should be completely isolated from one another at layer 2, each with a unique IP space at layer 3 and you should get zero duplicate assignments or mixed traffic at the endpoints. Also, you should be able to leave all endpoints NICs set on DHCP, with no static IP or VLAN designations done on the endpoints themselves, because your switches should be properly segmenting traffic upstream, with your routers controlling the IP addressing (via DHCP or static reservations). The point there is that you retain centralized control of addressing by keeping it on the routers, versus having to jump into the endpoints to make changes (which can get messy and unsustainable).

So that's the right way to do this. To keep yourself dialed in, I would stay focused on the network config alone, and ignore all the peripherals for now. Change one switch at a time, starting with your core-most switch (the GS108Tv2 connected to the Asus routers), then test with an open port on that switch for correct VLAN-to-subnet mapping and behavior, and moving on from there, downstream to the next switch, to propagate changes, then re-test and continue. Once you have the whole LAN properly configured, then you can focus on evaluating internet connections and/or the details of other non-network-related components.

 

[TanyaC]

Thank you for the very detailed explanation. very grateful.

Just to make sure you understand my network - my test PC is used primarily to test new versions of software, windows patches, game updates, changes I make to security (like hosts file entries, firewall rules etc). These sorts of things are generally not network-configuration sensitive. Just a way for me to test stuff without affecting production PCs. Only once every couple of years to I test a new ISP and look to configure only that PC to direct it's internet traffic to the appropriate router. So for 23 of 24 months the PC is typically connected to the production router.

In a couple of days I will finish my testing and the test router will be disconnected and packed away for hopefully at least a couple more years.

I've saved the instructions you've provided and when I can get some alone time with the equipment I'll give it a go.

thanks again.

 

[Trip]

@TanyaC - Very welcome, and understood on the test PC role. Honestly, the simplest way to switch the Test PC back to your production network, using my above example, is to change its access port VLAN assignment (on the core GS108Tv2) from VLAN 20 (test network) to VLAN 10 (production). The switch will then send the Test PC's traffic bound for the internet upstream to the AC88U and out your production internet connection. So, one simple change of an access port is all you need to do; nothing on the endpoint itself.

Hope that helps for whenever you do revisit this.