Setting up a layer 3 switch to do IP VLAN Routing

[coxhaus]

I want to respond to clear up some confusion which seems come up often with using a layer3 switch. Most people seem to setup the layer 3 switch the same as a layer2 switch where the router is handling the VLANs. This is not the best way for a layer3 switch. Yes it will work but you are not using the layer 3 switch as a layer3 switch. You need to let the layer3 switch do the routing. It has the bigger backplane over the router and can free up the load on the router.
You do this by the following:
Setup the router off an access port on a VLAN. Run DHCP from the switch or a separate sever using DHCP RElAY. Add routing statements for the other VLANs to the router pointing to the switch. Run the router without VLANs defined and no DHCP, set the router IP address to fit the access port network on the switch. The switch is the default gateway for the clients and the router is the default route for the switch.

This is the way I run my setup. The switch is faster at routing VLAN traffic than the router.

PS
If the DHCP server in consumer routers was more sophisticated you could use DHCP RELAY with a router instead of a PC based DHCP server so the router could perform DHCP but I have not seen it in the routers I own. The routers are just too low tech when it comes to DHCP.

 

[ekhoo]

Thanks coxhaus. I just did that today on the production environment. All the DHCP and VLAN routing is on the switches now. 5 VLANs, 5 scopes.

I have 5 SG300-28Ps and I've set one switch to be DHCP and main route to the Internet Router. This switch is the one with one port to the router. All switches are linked via 2 x 1G LAG links set up as Trunks, 1UP, 10T, 20T, 30T and 40T (one switch is linked via Fiber SFP). Is there any benefit to spreading the DHCP around? Management wise, it's easier to track what/where is where if it's just on one switch.

 

[System Error Message]

not all layer 3 switches are faster than routers. Theres the CCR1072 with 8 SFP+ ports which will do filtering much faster than a layer 3 switch.
Using a DHCP relay allows you to control DHCP servers on your network much easier such as for blocking other DHCP servers on your network so the switch knows which one is the correct DHCP server so your clients dont get tricked by some rogue person attached to your network. Some of cisco tutorials on security use layer 2 which have to be applied to each individual switch.

 

[coxhaus]

Yes I was thinking average joe router. I guess you can find a router faster than some of the lowest level switches which this forum is based on but there are faster layer3 switches than a CCR1072 just not on this forum. I think if you look at the fastest routers and switches the switches will be faster. At least that is the way it was when I worked with Cisco gear years ago. I assume we are talking layer3 since this what this thread is about.


Actually DHCP RELAY redirects DHCP requests from a client to a specified point into some network. Normally DHCP requests are a non-directed traffic which is only broadcast within one network so a DHCP server in a different network would not see the requests from these clients. DHCP RELAY allows the DHCP server to process these requests from a different network. So the main idea of DHCP RELAY was not to control servers on your network but to allow one DHCP server to function for multiple networks. Security is a side effect.

 

[coxhaus]

Thanks coxhaus. I just did that today on the production environment. All the DHCP and VLAN routing is on the switches now. 5 VLANs, 5 scopes.

I have 5 SG300-28Ps and I've set one switch to be DHCP and main route to the Internet Router. This switch is the one with one port to the router. All switches are linked via 2 x 1G LAG links set up as Trunks, 1UP, 10T, 20T, 30T and 40T (one switch is linked via Fiber SFP). Is there any benefit to spreading the DHCP around? Management wise, it's easier to track what/where is where if it's just on one switch.

One DHCP server is much preferred. You want all your DHCP in one place to see the whole picture of your networks.

 

[coxhaus]

ekhoo, I assume your routing structure is such that all DHCP clients are using the switch as there default gateway. For each DHCP scope the IP address would be the IP address for the VLAN IP switch address. If you followed my example the default gateway IP address would be 192.168.0.254 for first VLAN1 scope. The second default gateway IP address for the second VLAN10 would be 192.168.10.254. The third default gateway IP address for the third VLAN20 scope would be 192.168.20.254. And so on for each VLAN setup. So for each DHCP scope there will be a different default gateway IP address but all the IP addresses will be the same layer3 switch. You want to do this so all the local traffic is routed by the switch and not the router.

Using my example the default gateway for the static IP address of the switch not any of the DHCP scopes would be 192.168.0.254 with the default gateway being the router IP address so all traffic going outside the switch will be routed to the router.

I hope this makes sense and this the way you setup your system.

 

[System Error Message]

in terms of layer 3 router vs switch speeds, for high speed routers like the CCR they perform filtering much faster than switches do in their bandwidth range. Many switches use the same chips that scale for wirespeed switching up to a certain point i.e. 40Gb/s but when given even a single rule will not be able to maintain 40Gb/s of forwarding capacity. The CCR1036 requires only 28Gb/s so you would need to compare it to a switch around that capacity whereas the CCR1072 needs 80Gb/s of forwarding capacity to max out the ports so you would compare it to an 80Gb/s layer 3 switch. Interestingly mikrotik already has given layer 3 throughputs for those routers with the number of filter rules and with the standard 1500 byte packets they maintain wirespeed with lots of filters whereas layer 3 switches like the CCR226 cannot maintain wirespeed when given filter rules.

 

[coxhaus]

I was looking at the specs on Cisco's SG500X-24 layer 3 switch. It has 128 Gbps switching capacity. This is what I want next if I can find one at a good price.

The SG500X switch has protocol routing which I want. I want to create a VLAN on the switch to connect to my router. I will implement protocol routing on this VLAN. The switch and the router will broadcast routing info so I will not need to create static route statements on the router to the switch. All routing info will be carried in the protocol routing. This will allow me to add a second layer 3 switch with routing information. Routing between the 2 L3 switches will happen without using the front door router. The switch will be able to find networks on this protocol routing VLAN and route directly to the other L3 switch.

The SG500X supports 10 gig. I can also implement 10 gig to the front door router and other layer 3 switches when routers start having 10 gig.

 

[System Error Message]

The higher end CCR series do have a lot of switching capacity for layer 3 too but they are only limited by their port capacity. If you look at the solutions Tilera release they have a lot more port capacity and have no issues providing wirespeed layer 3 switching. One other impressive thing that Tilera has which mikrotik never uses is they have 288 core routers which they achieve by combining 8 36 core CPUs. They do have the 72 core ones but i think for them 36 core is cheaper and easier to make. Even so the limit of a 36 core TILE CPU is 70Gb/s limited by the CPU grid if you have a really bad setup or coding so it is faster than a switch but it just doesnt have the port capacity to reach it. It means that the maximum it can handle per interface is 70Gb/s using standard size packets. I got the numbers by stress testing my CCR in ways that it would never be used such as generating packets and sending it to a virtual interface only for it to be processed and than dropped and it wasnt using all its CPU power. Switch CPUs lose a lot of capacity when you have add firewall filters to them while the CCR only loses less performance when using firewall filters.

So if you need wirespeed switching + firewall and other configs than you want something like a CCR or TILERA's evaluation server or even the Cisco edgerouter series.

But if you're just doing vlans with ip segments and some configs that are part of switching logic (like using ipv6) than you would gain no benefit using a very fast router instead of a layer 3 managed switch.

 

[coxhaus]

My main reason for an upgrade is to gain a routing protocol. I like using routing protocols. There is a certain amount of overhead using them but I think they are fun to use. For my small home network my current SG300-28 and my SG200-8 is plenty fast enough for what I do. I have never maxed my switches and never will. If I ever get a gig connection I will LAG 2 ports together for my backbone connections. Of course this does not mean I won't upgrade sometime in the future just because I want to play with something new.

 

[System Error Message]

using routing protocols wont slow down the switch. Its only if you add filter rules that it starts to slow down and when you need a core router instead.

 

[coxhaus]

Yes I agree routing protocols will not slow down the switch. The routing protocols take up bandwidth on the gig connections. It is a very small percentage but it is there. Overall I like it. You make a routing change or add a new network and it is dynamically propagated out so all layer 3 devices know. No more adding static routes or forgetting to add static routes so part of the network does not work.

 

[hobara]

When you begin to network lessons, it is known that switching is layer - 2 ! Omg it is something like that 11 is not divided to 4. But after the pre school you learn it that it is divided to 4. So today we are gonna learn that Layer-3 Switching, what is inter vlan routing ! Inter vlan routing is made with two methods : Router on a stick and Multi-Layer switching.

ROUTER ON A STICK



As we talked in VLAN subject, different VLANs can't communicate each other without a Layer -3 device ! As you can see here , we have a router to establish communication with Vlan 10 and Vlan 20. If VLAN 10 wants to communicate with VLAN 20 , packet goes to router first and routed to VLAN 20 via sub - interfaces. That is a slow operation because of routing is slow.

ROUTER ON A STICK CONFIGURATION
Router(config)# int f0/0.10 (May be other than 10 too, but mostly used like this.)
Encapsulation dot1Q 10
IP add 10.1.10.1 255.255.255.0

MULTI-LAYER SWITCHING

As you can see on the figure above, there is just one multi layer switch and no any other router or something like that ! There are two vlans, vlan 10 and 20 and they can communicate because this is a Layer - 3 switch ! It makes routing on cable-speed and has a great backplane bandwith so that it's much faster than inter vlan routing via routing on a stick.

MULTI-LAYER SWITCHING CONFIGURATION
SW(A)(config) # ip routing
interface VLAN 10
ip add 10.1.10.1 255.255.255.0
interface VLAN 20
ip add 10.1.20.1 255.255.255.0


Source : http://www.networkel.com/2015/10/layer-3-switching-inter-vlan-routing.html

 

[System Error Message]

Just to make things clear,
Layer 2 switching would be sending one packet from one mac address to another. It only reads the layer 2 section of the packet.
Layer 3 switching would be sending one packet from one IP address to another but unlike a router it doesnt translate, only forwards and reads more headers of the packet.
Layer 2 bridging would be connecting 2 layer 2 networks together just like a router for layer 3 which can also forward without translating but it can perform layer 2 NAT

Its very easy to get confused. A layer 3 switch will also perform layer 2 because during the switching of packets in layer 3 it will refer to layer 2 as well but in the layer 3 switch layer 2 switching is part of it. By using vlans and communicating between them via layer 3 you would be blocking layer 2 traffic from going about. Everytime a packet is switched on layer 3 from one layer 2 network to another the layer 2 header is changed.

Routers actually have a lot of backplane bandwidth but the CPU is more of a general purpose one whereas the CPU of a switch is a massively parallel special purpose CPU. Thats is why you can get a dumb 10Gb/s switch with 5 gigabit ethernet ports really really cheap now because unlike other CPUs they are tiny and perform few functions, even a lot less than a GPU's shader and are also clocked very low.