Disappointed in iSCSI speed

[usao]

I bought 8 of the Samsung 850 evo 1Tb SSD drives and configured Raid-10 on an HP server. Im using this as an iSCSI server for a database application.
When I try to map luns using iSCSI, im getting an average throughput of about 50-60 MB/sec.
Each SSD is capable of almost 500 MB/s streaming speed according to the docs, and in Raid-10 I should be able to do better than that.

• Max Sequential Read: Up to 550 MBps
• Max Sequential Write: Up to 520 MBps
• 4KB Random Read: Up to 100,000 IOPS

My thought is it's being limited by the speed of the gigabit switch, and am looking for a 10Gb switch to get my speeds closer to the expected value of 500 MB/s for streaming access.
What im seeing is different physical layers (10GBase-T, SFP+, XFP). What are most people using, and/or what is the most cost effective on a per port basis? I don't need many ports, probably an 8-12 port switch would be good enough. The only switch I have found is a Netgear XS series.
So, im looking for suggestions on 10 GbE switches. Mainly wondering what other folks are using.

 

[System Error Message]

with SSDs you do not need to perform RAID 1 since they will all fail at the same time once their write endurance is reached and they are more reliable than hard drives until that happens. You will get a smart warning before this happens and when an SSD reaches its write endurance some of them turn into read only mode and bricks itself on the next power cycle.

50-60MB/s is half of gigabit ethernet for one direction. If you do a proper benchmark of SSDs you will find that the stream speed means sequential but depending on how you configure your database it can end up being random reads. random reads can slow an SSD to a few MB/s especially on a RAID array since it puts load on the controller. I suggest changing your RAID cache mode and setting up your database to be more sequential.

SFP is just a modular port which you can plug different modules in so you can use different mediums. It lets you plug a lot of different mediums such as various fibre optics modules, cable, direct and even ethernet except for DSL. SFP is rated at 1.25Gb/s speeds while SFP+ is rated at 10Gb/s. 10GBase-T is the term given for a link speed of 10Gb/s. XFP is a bit like SFP but uses a different shape and electrical compatibility.

There isnt really the most common or the most cost effective, it depends on what you're doing. If you dont need 10Gb/s you can use port teaming on gigabit ethernet ports to achieve more bandwidth which would save a lot of money but requires a lot of ports and cabling. SFP or XFP modules are usually expensive but you can use SFP direct which is a copper SFP to SFP port cable that is many times cheaper than the other modules. The reason people use SFP is because you dont need to produce so many different products to support a particular medium, you can just get the module for the medium you want for example using fibre optics with SFP on a switch/router and laying a few kilometers of fibre optic cable since ethernet has a limitation of 100 meters. Although the cost of doing that is high it uses less power and costs less than using ethernet cabling with many switches as repeaters along the way.

 

[Terry Kennedy]

My thought is it's being limited by the speed of the gigabit switch, and am looking for a 10Gb switch to get my speeds closer to the expected value of 500 MB/s for streaming access.

You should be getting a lot closer to the theoretical 125MB/sec on Gigabit Ethernet. I would investigate that problem before making the jump to 10GbE. If you are running on a Unix-like system or Linux, you can use something like iperf to see if you have a network problem. Also, the quality of iSCSI implementations seems to vary quite a bit. You might want to try testing with either NFS or SMB to see if that could be affecting you.

What im seeing is different physical layers (10GBase-T, SFP+, XFP). What are most people using, and/or what is the most cost effective on a per port basis? I don't need many ports, probably an 8-12 port switch would be good enough. The only switch I have found is a Netgear XS series.
So, im looking for suggestions on 10 GbE switches. Mainly wondering what other folks are using.

"Cost effective" and "10GbE" don't belong in the same sentence at this point. The Netgear is likely the least expensive switch you will find, and has the advantage of lots of copper ports. Switches with lots of SFP+ ports are much more common, but if your systems are all within reach of 10GbE copper cabling you can avoid the hassle of fiber.

I had a Netgear XS708E and it seemed to work fine. I just needed more ports and a better management interface, so I switched to a Dell PowerConnect 8024 and sold the Netgear. A user on this forum purchased it and wrote up an excellent series, Confessions Of A 10 GbE Network Newbie which is well worth a read.

If you go with fiber, you want to avoid the "legacy" transceiver types - XENPAK, X2, XFP as they are larger and often more expensive than SFP+. There doesn't seem to be a SFP+ RJ45 transceiver (there is a GigE RJ45 SFP), so your choice of switch will dictate whether you use copper or fiber. For host NICs, I like the Intel X540 series.

You should be able to get "wire speed" on 10GbE. Here's an iperf test between 2 systems here:

(0:1) test2:/sysprog/terry# iperf -c test1
------------------------------------------------------------
Client connecting to test1, TCP port 5001
TCP window size: 35.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.20.30.9 port 64500 connected with 10.20.30.8 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 11.5 GBytes 9.87 Gbits/sec

 

[chadster766]

I recommend first running Microsoft SQLIO on your drive array to find out if you are getting the most IOPS out of the array. I did something similar with an HP Gen8 all SAS SSD but went Raid 5 for redundancy (less costly because less disks required). I ran SQLIO and discovered the array was performing like a standard or less SAS spinning disk array.

This was done a few years ago. HP tech support was lost because it was too new even for them especially since I licensed HP SmartPath firmware for the Raid Controller.

I discovered I wasted some money on additional Raid Controller cache memory because the cache is unused\unnecessary when SSD's are involved.

With the clues HP Support provide and my own research I found that a specific HP Array driver was required to get SmartPath operational. This was key but then I discovered that it was only available for Windows Server 2008 R2 and my server was Windows Server 2012 I though I had wasted money on that too!

The trick was to manually install the Windows Server 2008 R2 driver (the driver install program fails) through the device manager onto the Windows Server 2012.

Below is the SQLIO final results:

Raid 5:
*** Start of Test for DATA Disk ***
******************************************************************************
Sequential Read 8K - TARGET better than 7000 IOs/sec
sqlio v1.5.SG
using system counter for latency timings, 1948427 counts per second
parameter file used: paramDATA.txt
file c:\temp\datafile.dat with 4 threads (0-3) using mask 0x0 (0)
4 threads reading for 30 secs from file c:\temp\datafile.dat
using 8KB sequential IOs
enabling multiple I/Os per thread with 8 outstanding
size of file c:\temp\datafile.dat needs to be: 4194304000 bytes
current file size: 0 bytes
need to expand by: 4194304000 bytes
expanding c:\temp\datafile.dat ... done.
using specified size: 4000 MB for file: c:\temp\datafile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 40795.70
MBs/sec: 318.71
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 0
Max_Latency(ms): 14
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 80 20 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
******************************************************************************
Sequential Write 8K - TARGET better than 7000 IOs/sec
sqlio v1.5.SG
using system counter for latency timings, 1948427 counts per second
parameter file used: paramDATA.txt
file c:\temp\datafile.dat with 4 threads (0-3) using mask 0x0 (0)
4 threads writing for 30 secs to file c:\temp\datafile.dat
using 8KB sequential IOs
enabling multiple I/Os per thread with 8 outstanding
using specified size: 4000 MB for file: c:\temp\datafile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 3240.20
MBs/sec: 25.31
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 9
Max_Latency(ms): 39
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 1 3 2 1 2 3 6 15 10 6 6 20 13 4 2 1 1 1 1 1 0 0 0 0 0
******************************************************************************
Random Read 8K - TARGET better than 700 IOs/sec
sqlio v1.5.SG
using system counter for latency timings, 1948427 counts per second
parameter file used: paramDATA.txt
file c:\temp\datafile.dat with 4 threads (0-3) using mask 0x0 (0)
4 threads reading for 30 secs from file c:\temp\datafile.dat
using 8KB random IOs
enabling multiple I/Os per thread with 8 outstanding
using specified size: 4000 MB for file: c:\temp\datafile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 62614.70
MBs/sec: 489.17
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 0
Max_Latency(ms): 22
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 98 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
******************************************************************************
Random Write 8K - TARGET better than 500 IOs/sec
sqlio v1.5.SG
using system counter for latency timings, 1948427 counts per second
parameter file used: paramDATA.txt
file c:\temp\datafile.dat with 4 threads (0-3) using mask 0x0 (0)
4 threads writing for 30 secs to file c:\temp\datafile.dat
using 8KB random IOs
enabling multiple I/Os per thread with 8 outstanding
using specified size: 4000 MB for file: c:\temp\datafile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 15909.63
MBs/sec: 124.29
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 1
Max_Latency(ms): 29
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 34 31 15 9 5 3 2 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
******************************************************************************
*** Start of Test for LOG Disk ***
******************************************************************************
Sequential Read 8K - TARGET better than 7000 IOs/sec
sqlio v1.5.SG
using system counter for latency timings, 1948427 counts per second
parameter file used: paramLOG.txt
file c:\temp\logfile.dat with 4 threads (0-3) using mask 0x0 (0)
4 threads reading for 30 secs from file c:\temp\logfile.dat
using 8KB sequential IOs
enabling multiple I/Os per thread with 8 outstanding
size of file c:\temp\logfile.dat needs to be: 4194304000 bytes
current file size: 0 bytes
need to expand by: 4194304000 bytes
expanding c:\temp\logfile.dat ... done.
using specified size: 4000 MB for file: c:\temp\logfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 40853.16
MBs/sec: 319.16
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 0
Max_Latency(ms): 11
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 80 20 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
******************************************************************************
Sequential Write 8K - TARGET better than 7000 IOs/sec
sqlio v1.5.SG
using system counter for latency timings, 1948427 counts per second
parameter file used: paramLOG.txt
file c:\temp\logfile.dat with 4 threads (0-3) using mask 0x0 (0)
4 threads writing for 30 secs to file c:\temp\logfile.dat
using 8KB sequential IOs
enabling multiple I/Os per thread with 8 outstanding
using specified size: 4000 MB for file: c:\temp\logfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 3259.03
MBs/sec: 25.46
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 9
Max_Latency(ms): 42
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 1 3 2 1 2 3 6 16 10 6 7 21 13 3 2 1 1 1 1 1 1 0 0 0 0
******************************************************************************
Random Read 8K - TARGET better than 700 IOs/sec
sqlio v1.5.SG
using system counter for latency timings, 1948427 counts per second
parameter file used: paramLOG.txt
file c:\temp\logfile.dat with 4 threads (0-3) using mask 0x0 (0)
4 threads reading for 30 secs from file c:\temp\logfile.dat
using 8KB random IOs
enabling multiple I/Os per thread with 8 outstanding
using specified size: 4000 MB for file: c:\temp\logfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 62389.60
MBs/sec: 487.41
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 0
Max_Latency(ms): 21
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 98 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
******************************************************************************
Random Write 8K - TARGET better than 500 IOs/sec
sqlio v1.5.SG
using system counter for latency timings, 1948427 counts per second
parameter file used: paramLOG.txt
file c:\temp\logfile.dat with 4 threads (0-3) using mask 0x0 (0)
4 threads writing for 30 secs to file c:\temp\logfile.dat
using 8KB random IOs
enabling multiple I/Os per thread with 8 outstanding
using specified size: 4000 MB for file: c:\temp\logfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 15947.06
MBs/sec: 124.58
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 1
Max_Latency(ms): 28
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 34 31 15 9 5 3 2 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
*** Test Done ***

 

[awstey]

I once used Netgeat Prosafe 8-port 10 Gigabit Unmanaged Plus Switch, consists of eight 10GE copper ports and one combo 10GE Fiber SFP+ port. Go to fiber or copper as you like.

 

[azazel1024]

Yeah, you have something else going on. What kind of "HP server"?

I don't run iSCSI, but I have played with it on my network. In most cases, I can get slightly better IOPS on a single 1GbE link than I can with SMB/CIFS, but I get slightly lower streaming speeds.

At ~128KB file sizes, I can get around 100MB/sec with iSCSI and about 70MB/secs with SMB/CIFS. With 10MB file sizes I get about 115MB/sec with iSCSI and 117.5MB/sec with SMB/CIFS (3.01 if it matters to anyone).

Since 99% of my stuff is big files and I don't need some of the other features of iSCSI, I only run SMB.

That and since I run two GbE links between my server and desktop, with SMB3.01, I can get 235MB/sec with large files, which I can't do with iSCSI (unless it has been updated recently, I don't see support for Multichannel, like MS added in Windows 8+ for SMB).

Anyway, I would suspect something else. Possibly a bad/misconfigured switch, possibly bad wiring, probably more likely a server misconfiguration or some bit of hardware not being able to handle it (not enough RAM or slow processor).