Mac OS SMB max NVME throughput on 10GbE network?

[sharkey]

Not sure if there's any fine tuning that can be done here to optimise R/W performance but in a nutshell I’m not seeing the R/W speeds on NVME drives I would expect on a 10GbE network (maxing out around 1100mb/s)

I have 2 Mac Computers, both with TP Link TX401 10GbE NICs, connecting to an Asus ax89x via 5m Cat7 cables, one connected to an SFP to Rj45 converter to utilise both 10GB ports.

Mac 1 is a computer video editing workstation.
Mac 2 is acting as a server with a 60TB Softraid 5 array, and a 6tb NVME Softraid 0 array (3x2tb PNY nvme’s in an Asus HyperX m.2x16 v2 in PCI slot 1)
Both Mac’s running BigSur 11.7.3


I’ve run iperf3 both ways on both computers and getting max link speeds of 9.8 Gbits/s

AJA System test results:

• NVME Raid Array locally gets 3500+/5000+ mb/s W/R speeds
• 60TB Raid Array locally gets 600/700 mb/s W/R speeds

However through the network SMB protocol R/W speed performance from Mac 1 to Mac 2 is:

• NVME Raid Array network gets 1100/1100 mb/s W/R speeds
• 60TB Raid Array network actually performs better at 660/720 mb/s W/R speeds

I’m beginning to wonder if the Mac OS SMB protocol is becoming the bottleneck around 1200-1500mb/s with throughput (I’ve seen R/W pulses at around 1200-500 but never higher). Does anyone else have opinion?


Settings I’ve Tested:

AX89X:

• I've enabled Jumbo Frame in the AX89X settings, set negotiating speeds to the max for both the 10GB and SFP port. After caparison testing setting the SFP port to Auto/ and SFP TX clock to off, resulted in better iperf3 link speeds.

MACOS 10GbE NIC Manually Configured Network Settings:

• I've set all the settings to the max
• Disabled AVB/EAV Mode
• Jumbo Frame 9000. I’ve tested different settings (can go up to max 16000 on the TPLINK NIC) but 9000 default jumbo frame seems to perform best with W/R speeds.

MACOS .conf files

• I’ve updated the etc/nsmb.conf on both mac’s to include the line: signing_required=no
  • This did result in a boost to the current speed performance
  • source https://support.apple.com/en-us/HT212277
  • source https://kb.promise.com/thread/disable-smb-signing-on-os-x-10-11-5-10-12-x/
• I’ve learned more about fine-tuning the sysctl.conf file, which seems to be depreciated in BigSur+, and it looks as if creating and writing to the file, and creating a custom .plist to load on boot is required to make the settings stick upon reboot. But regardless the below settings didn’t improve performance
  • sudo sysctl -w kern.ipc.maxsockbuf=4194304
  • sudo sysctl -w net.inet.tcp.sendspace=2097152
  • sudo sysctl -w net.inet.tcp.recvspace=2097152
  • sudo sysctl -w net.inet.tcp.delayed_ack=0
  • sudo sysctl -w net.inet.raw.maxdgram=16384

The last maxdgram value is an interesting one because it allowed me to do some testing on max MTU on Mac OS and the AX89X

Mac OS does not actually by default allow you to ping 9000 bytes, even if jumbo frame 9000 is enabled.

When running in terminal
sysctl -a | grep maxdgram
Default values are:
net.local.dgram.maxdgram: 2048
net.inet.udp.maxdgram: 9216
net.inet.raw.maxdgram: 8192

Which allows a max ping of 8184 bytes
ping -D -s 8184 192.168.50.1

Anything higher by default results in a timeout, or message to long

Once the net.inet.raw.maxdgram is manually increased to 16384 (using the above terminal code sudo sysctl -w net.inet.raw.maxdgram=16384) I found these results:

When Keeping the Jumbo Frame Hardware settings in the NIC at default 9000 MTU:
The max byte size ping is 8972 bytes before timing out to the other Mac or the Router
ping -D -s 8972 192.168.50.1

If I increase the Jumbo Frame Hardware settings to 16000 MTU:
The max byte size ping is 12242 bytes before timing out on the Router
ping -D -s 12242 192.168.50.1
The max byte size ping is 8990 bytes before timing out to the other Mac
ping -D -s 8990 192.168.50.101

The above tells me that Mac OS limits the jumbo frame to 9000 and the Asus AX89X is at 12250. Any thoughts?

sources:

Tuning TCP/IP Performance for MassTransit | Knowledge Base

Summary: This article describes how to tune TCP/IP for maximum performance with MassTransit. Description: Explanation & Overview TCP/IP Settings As network speeds have increased some of the default configuration settings for TCP/IP no longer result in maximum performance. In particular, networks...

kb.acronis.com

source https://www.gdt.id.au/~gdt/presenta...et-tcp/reference-materials/web/psc-tuning.pdf

How to test if 9000 MTU/Jumbo Frames are working

An overview on how to test if jumbo frames/MTU 9000 is working on mac, Linux, and Windows

blah.cloud

 

[Tech Junky]

Well, it's simple math. 10gbps is 1250MB/s and take into account overhead of protocols and it's performing as expected.

 

[sharkey]

Well, it's simple math. 10gbps is 1250MB/s and take into account overhead of protocols and it's performing as expected.

dang math, bits and bytes, yikes

 

[sfx2000]

Not sure if there's any fine tuning that can be done here to optimise R/W performance but in a nutshell I’m not seeing the R/W speeds on NVME drives I would expect on a 10GbE network (maxing out around 1100mb/s)

I have 2 Mac Computers, both with TP Link TX401 10GbE NICs, connecting to an Asus ax89x via 5m Cat7 cables, one connected to an SFP to Rj45 converter to utilise both 10GB ports.

Mac 1 is a computer video editing workstation.
Mac 2 is acting as a server with a 60TB Softraid 5 array, and a 6tb NVME Softraid 0 array (3x2tb PNY nvme’s in an Asus HyperX m.2x16 v2 in PCI slot 1)
Both Mac’s running BigSur 11.7.3

Looks fine by me...

About the only thing I would change is to insert a 10Gbe switch between the two Macs, taking the router out of the middle of things, but I don't think that would improve thruput...

 

[Tech Junky]

dang math, bits and bytes, yikes

No prob. We all have senior moments and forget to divide or multiply by 8.