Slow performance in home studio with multiple LAG's

[NoxtheNox]

Hello all,

I have a small boutique computer animation studio with some underperforming networking. I have 6 workstations PC & Mac and a rack system with both a PC and Mac file servers and a small 8 node render farm. I recently trunked all the connections and configured everything for jumbo frames but all of my connections are capping out around 116 MB/s. I've tried every conceivable file copy setup mac to mac, pc to pc, pc to mac etc. I know that I'll lose some performance between AFP and SMB.

I have everything connecting to 2 Cisco Smart switches. One smart switch sits in the rack with the render farm and file servers and the other lives in the work area and has dual gigabit trunks running to all the workstations. My Mac based server has a 12 disk raid 6 that does around 500-600 r/w, it has a 6 gigabit trunk to the switch. The PC server has a similar raid setup and has a dual gigabit trunk. Everything is running over gigabit cables.

The actual gear is as follows:

Rack Smart switch is a Cisco SG500-52
My Mac Pro server has a 6 port smalltree P2EG-6-T
My PC file server has an onboard dual gigabit NIC (it's a dual Xeon), it's also handling the DHCP
The workstation smart switch is a Cisco SG308-28.
I have a smaller Netgear SG108 8 port for 2 workstations. This connects to the Cisco with a 2 gigabit trunk.
Router and wireless comes from an Asus rt-ac66.
There is a 4 gigabit trunk connecting the 2 Cisco smart switched

I had someone design and build the system but they haven't been available to help me get it optimized. Something in the ecosystem is not allowing me to get any of the trunked performance I was hoping to get. It is a bit frustrating as I've invested some decent money to try to get some enterprise level gear. I'm also not an IT professional, I know just enough to get myself in trouble.

Any help diagnosing the bottleneck(s) would be super appreciated!

Thanks

N

 

[Terry Kennedy]

I recently trunked all the connections and configured everything for jumbo frames but all of my connections are capping out around 116 MB/s. I've tried every conceivable file copy setup mac to mac, pc to pc, pc to mac etc. I know that I'll lose some performance between AFP and SMB.

Any help diagnosing the bottleneck(s) would be super appreciated!

116MB/sec is 928Mbit/sec which is pretty close to the theoretical capacity of a single Gigabit Ethernet link.

When you say you trunked the connections, it seems you mean multiple physical connections? At least for many versions of SMB, a single transfer won't use multiple connections - the link aggregation method will classify the traffic onto a specific cable. SMB multichannel is needed to utilize more than one physical link, and is relatively new (Windows Server 2012) and seems to still be a work in progress on Samba.

 

[NoxtheNox]

116MB/sec is 928Mbit/sec which is pretty close to the theoretical capacity of a single Gigabit Ethernet link.

When you say you trunked the connections, it seems you mean multiple physical connections? At least for many versions of SMB, a single transfer won't use multiple connections - the link aggregation method will classify the traffic onto a specific cable. SMB multichannel is needed to utilize more than one physical link, and is relatively new (Windows Server 2012) and seems to still be a work in progress on Samba.

Terry,

Thanks for reply. I guess I should have been a bit more specific with my language. I have created teamed NIC's on all the workstations. I have a Mac Pro that has a dual onboard gigabit NIC, I have a Mac Mini with a thunderbolt chassis that has a dual port gigabit NIC, I have a PC with a quad port gigabit NIC etc etc. They all have LACP and jumbo frames enabled. I was under the impression that LAG's would combine the speeds of each connections and get me better file transfer and access speeds.

Thanks

 

[Terry Kennedy]

I was under the impression that LAG's would combine the speeds of each connections and get me better file transfer and access speeds.

No, that would require a true random selection algorithm for each packet, to determine which LACP member port would be used. There is no such algorithm in the protocol, because it has the potential to produce out-of-sequence packets on the receiving side.

There are a number of standard algorithms to assign traffic to a particular member. I believe this is the full set, but note that not all devices support all algorithms:

• Source MAC, VLAN, EtherType, and incoming port associated with the packet.
• Source IP and Source TCP/UDP fields of the packet.
• Destination MAC, VLAN, EtherType, and incoming port associated with the packet.
• Source MAC, Destination MAC, VLAN, EtherType, and incoming port associated with the packet.
• Destination IP and Destination TCP/UDP Port fields of the packet.
• Source/Destination MAC, VLAN, EtherType, and incoming port associated with the packet.
• Source/Destination IP and source/destination TCP/UDP Port fields of the packet.

Some devices have additional algorithms, but both sides need to support them.

Note that this limits the bandwidth to 1 * the port speed for a single logical connection (such as to a network share), but leaves additional bandwidth for another logical connection between those two systems such as a FTP file transfer, as long as the algorithm chosen is one of the ones that takes TCP/UDP port numbers into consideration when queueing the traffic for a LACP link.

 

[System Error Message]

make sure that LACP is used on both the server and the switch. Make sure that that the bandwidth between switches isnt the bottleneck too. If you draw your network and list the link bandwidths that may help too.

You may need to test using multiple clients or multiple things at the same time.

 

[thiggins]

make sure that LACP is used on both the server and the switch. Make sure that that the bandwidth between switches isnt the bottleneck too. If you draw your network and list the link bandwidths that may help too.

Terry has provided the proper advice. LACP does not provide higher bandwidth for a single logical connection

 

[azazel1024]

Terry has provided the proper advice. LACP does not provide higher bandwidth for a single logical connection

Yup. LAG will only increase the AGGREGATE bandwidth. It will not increase single connection bandwidth any.

SMB multichannel WILL increase aggregate and single connection bandwidth, but it requires Windows 8, 8.1, 10 or server 2012 to do it with (must be on both sides of the connection). With SMB multichannel you can do LACP/LAG on the infrastructure between the computers, but the connections of the computers themselves to the switches must NOT be in a LAG configuration (IE interface 1 and 2 straight to switch 1, switch 1 to switch 2 can have a 2 port LAG and then switch 2 to interface 1 and 2 on server, no LAG there will work. However if you do a LAG for interface 1 and 2 to Switch 1, SMB multichannel breaks).

So if you want this to work, you'll have to ditch the Mac server and run all Windows based systems and only Windows 8 or newer. Otherwise all you are going to get is ~116MB/sec. However, you can do 116MB/sec x 6 connections to that Mac server because of the 6 port LAG group.

Reason number 3 I prefer windows based servers (in my home environment) over Linux or Mac.