1 Server 3 Switches, MAXIMUM Bandwidth

[cdusseau]

This thread branches from this one: Connect FC to Existing Ethernet LAN

I now have 3x 48-Port managed switches and 1x PE2900 Server

I want to feed the 3 switches form one server, but maintain the MAXIMUM bandwidth possible for all clients connected to any switch. I cannot afford FCoE so I am looking at Link Aggregation (LA) instead.

The plan:

1) Install 3x Quad Port NICs in the server.
2) Use LA to team each Quad into a single "link".
3) Connect one "link" to each switch, using trunking on the switch to Aggregate the 4 ports on the switch side.
4) Attach clients and enjoy 4Gbps spread among all clients on each switch.

To note:
A) The switches will not be linked to each other, their only connection will be through the server itself. This should avoid creating a loop (I hope).

I'm looking for comments or suggestions on this setup. Any ways I could improve it without increasing cost significantly.

Thanks!

 

[devnull]

There is a no-cost experiment that you can run on this environment before you purchase new hardware.

You have two 1Gbps ports on that server.

Attach one to a single switch.

Attach another to the second switch and daisy chain that one to the third switch.

Configure the server for this setup then conduct function and bandwidth testing from ports on all three switches.

Another test you could run is link aggregation with two ports on all three switches daisy-chained.

Those two tests should be enough proof-of-concept to purchase hardware for your proposed setup. If those two experiments work, then you have confirmed that you can operate your environment over multiple NICs, and that you can successfully use link aggregation.

For about $2000, you could get a fourth switch with 4x 10Gbit SFP+ ports and 24x 1Gbit ports, a NIC with dual 10Gbit SFP+, and two SFP+ Direct Attach Cables. Then you configure that switch with link aggregation to the three 3Com switches. You could achieve up to (theoretical) 8Gbits per switch with this setup.

Note that your server has a single PCIE 8x slot that's good for about 16Gbps (for the dual SFP+ NIC). The PCIE x4 slots would be good for the quad gigabit NICS, though.

 

[cdusseau]

There is a no-cost experiment that you can run on this environment before you purchase new hardware.

You have two 1Gbps ports on that server.

Attach one to a single switch.

Attach another to the second switch and daisy chain that one to the third switch.

Configure the server for this setup then conduct function and bandwidth testing from ports on all three switches.

Another test you could run is link aggregation with two ports on all three switches daisy-chained.

Those two tests should be enough proof-of-concept to purchase hardware for your proposed setup. If those two experiments work, then you have confirmed that you can operate your environment over multiple NICs, and that you can successfully use link aggregation.

For about $2000, you could get a fourth switch with 4x 10Gbit SFP+ ports and 24x 1Gbit ports, a NIC with dual 10Gbit SFP+, and two SFP+ Direct Attach Cables. Then you configure that switch with link aggregation to the three 3Com switches. You could achieve up to (theoretical) 8Gbits per switch with this setup.

I actually love that idea, but my budget will probably even cut that off at the pass. I should be able to get away with spending around $800 and getting the hardware I require with my plan in the OP.

I will take your advice and do a proof of concept first, but an engineer acquaintance of mine assured me that it "should" work.

Thanks!

 

[devnull]

You're welcome.

I added a note about the expansion slots in that server.

The PCIE x8 slot is enough for 16Gbps, and the PCIE x4 slots are sufficient for those quad gig NICs.