breaking the magic 100MB/s

[nox]

Hi,

Been following your articles on really fast NAS, and sorry, I think I may of just done it

I have two p5w64's linked via a dell poweredge 2716 switch. using the onboard pcie laned nics, the data xfer in windows reports 100-124MB/s down speed, for a 7GB file. I do have an areca 1210 in there, OS disk (XP) is a 74 gb raptor, and 3x samsung F1's in raid 5 on the areca. the other machine just has two raptor 150's on the onboard controller.

so... i reckon your hardware raid card will do the job for you.

Nox

 

[nox]

um, any ideas of free software i can use to test it out? have been using iperf for tweaking.

Nox

 

[thiggins]

Thanks for the report. You can use iozone, as I do for all NAS testing.

 

[nox]

ah I did look briefly at that, then got scared away when I didn't understand the graphs anyway, and realised i'd have to compile it myself.

Nox

 

[nox]

i take back the having to compile it yourself part! I decided to read the instructions before trying it out for once! so unlike me...

Nox

 

[nox]

thats a lot of info... the write value seems higher than the read?? that seems a bit odd, i'm not sure how to import it into excel either, i could probably could the number of times i've opened that program on my fingers... I *think* i have it on my work pc, put it that way... still does this look about right? maybe it's not getting what windows is reporting and just faking it

C:\Program Files (x86)\Benchmarks\Iozone 3.311>iozone -Rab test.txt -i 0 -i 1 -+u -f p:\test\test -q 64k -n 32M -g 1G -z
Iozone: Performance Test of File I/O
Version $Revision: 3.311 $
Compiled for 32 bit mode.
Build: Windows

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.

Run began: Sun Oct 12 19:35:31 2008

Excel chart generation enabled
Auto Mode
CPU utilization Resolution = -0.000 seconds.
CPU utilization Excel chart enabled
Using Maximum Record Size 64 KB
Using minimum file size of 32768 kilobytes.
Using maximum file size of 1048576 kilobytes.
Cross over of record size disabled.
Command line used: iozone -Rab test.txt -i 0 -i 1 -+u -f p:\test\test -q 64k -n 32M -g 1G -z
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random bkwd record stride
KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
32768 4 45671 170272 39750 39557
32768 8 77990 298255 39647 39647
32768 16 144939 484311 47232 47123
32768 32 273550 707896 59000 58960
32768 64 440824 905724 46498 46787
65536 4 44867 161692 39584 39599
65536 8 78177 291308 39451 39436
65536 16 155507 476327 46479 44894
65536 32 275871 700072 53217 52477
65536 64 72952 900634 50307 51805
131072 4 15329 28081 39590 39583
131072 8 77707 285670 32060 36802
131072 16 138104 456731 46816 46989
131072 32 267459 685649 56509 56721
131072 64 451515 892407 47767 49628
262144 4 15932 45897 39462 39530
262144 8 19804 51364 39216 34232
262144 16 157558 442318 48142 49835
262144 32 267730 640962 53196 52050
262144 64 426999 877829 49759 51978
524288 4 15776 72604 36301 39434
524288 8 21066 274006 38307 37191
524288 16 29489 29197 50513 46909
524288 32 309844 652728 60183 54981
524288 64 449095 51294 48711 50345
1048576 4 15127 33456 37699 37678
1048576 8 19502 36567 37717 37641
1048576 16 22891 139268 48737 48586
1048576 32 22551 96050 59250 59895
1048576 64 522943 793339 49445 49751

iozone test complete.
Excel output is below:

"Writer report"
"4" "8" "16" "32" "64"
"32768" 45671 77990 144939 273550 440824
"65536" 44867 78177 155507 275871 72952
"131072" 15329 77707 138104 267459 451515
"262144" 15932 19804 157558 267730 426999
"524288" 15776 21066 29489 309844 449095
"1048576" 15127 19502 22891 22551 522943

"Re-writer report"
"4" "8" "16" "32" "64"
"32768" 170272 298255 484311 707896 905724
"65536" 161692 291308 476327 700072 900634
"131072" 28081 285670 456731 685649 892407
"262144" 45897 51364 442318 640962 877829
"524288" 72604 274006 29197 652728 51294
"1048576" 33456 36567 139268 96050 793339

"Reader report"
"4" "8" "16" "32" "64"
"32768" 39750 39647 47232 59000 46498
"65536" 39584 39451 46479 53217 50307
"131072" 39590 32060 46816 56509 47767
"262144" 39462 39216 48142 53196 49759
"524288" 36301 38307 50513 60183 48711
"1048576" 37699 37717 48737 59250 49445

"Re-Reader report"
"4" "8" "16" "32" "64"
"32768" 39557 39647 47123 58960 46787
"65536" 39599 39436 44894 52477 51805
"131072" 39583 36802 46989 56721 49628
"262144" 39530 34232 49835 52050 51978
"524288" 39434 37191 46909 54981 50345
"1048576" 37678 37641 48586 59895 49751

"Writer CPU utilization report (Zero values should be ignored)"
"4" "8" "16" "32" "64"
"32768" 10.85 6.02 7.08 2.61 2.33
"65536" 9.74 6.07 4.62 3.07 2.90
"131072" 11.54 7.37 5.66 3.90 2.93
"262144" 10.74 6.50 4.76 2.88 3.34
"524288" 11.35 7.22 4.82 3.85 2.59
"1048576" 10.87 7.79 5.83 3.91 2.64

"Re-writer CPU utilization report (Zero values should be ignored)"
"4" "8" "16" "32" "64"
"32768" 9.33 9.79 7.42 3.76 3.86
"65536" 18.66 12.54 6.05 4.38 3.03
"131072" 15.64 10.20 6.48 5.01 3.31
"262144" 15.73 7.40 6.17 4.11 3.23
"524288" 17.78 10.59 5.73 4.51 3.55
"1048576" 15.35 9.61 6.11 4.00 3.50

"Reader CPU utilization report (Zero values should be ignored)"
"4" "8" "16" "32" "64"
"32768" 31.95 16.82 15.59 19.60 17.59
"65536" 27.24 14.09 12.13 7.51 5.96
"131072" 28.16 15.58 13.87 8.02 3.95
"262144" 24.58 14.90 12.86 9.72 3.25
"524288" 24.24 14.55 11.84 10.71 5.92
"1048576" 22.77 14.22 12.80 10.10 6.74

"Re-Reader CPU utilization report (Zero values should be ignored)"
"4" "8" "16" "32" "64"
"32768" 33.58 13.10 13.41 2.67 2.12
"65536" 28.15 13.06 8.45 7.48 7.39
"131072" 24.88 19.25 11.14 12.79 6.48
"262144" 22.52 16.45 11.21 10.18 6.48
"524288" 24.69 14.79 12.37 7.98 7.16
"1048576" 24.10 13.85 10.30 8.43 6.20

C:\Program Files (x86)\Benchmarks\Iozone 3.311>

 

[thiggins]

That iozone output looks reasonable. Looks like you are seeing cached write performance. You need to run iozone to a filesize that is larger than the memory on both the NAS under test and the machine running iozone to get a measure of the actual non-cached write performance.

Read performance with 64KByte record size is around 50 MB/s, which is typical of what I have seen with higher-performance NASes.

 

[00Roush]

Hi,

Been following your articles on really fast NAS, and sorry, I think I may of just done it

I have two p5w64's linked via a dell poweredge 2716 switch. using the onboard pcie laned nics, the data xfer in windows reports 100-124MB/s down speed, for a 7GB file. I do have an areca 1210 in there, OS disk (XP) is a 74 gb raptor, and 3x samsung F1's in raid 5 on the areca. the other machine just has two raptor 150's on the onboard controller.

so... i reckon your hardware raid card will do the job for you.

Nox

What OS was on the client and what OS was on the server?

Also you could try out this program... http://nodesoft.com/DiskBench/ I have been using the copy file function and it seems to be inline with what Windows Vista shows.


00Roush

 

[nox]

I actually have everything the wrong way round! the server is running windows XP and the client server 2008! using the microsoft trial to see if it's worth upgrading or not at the moment.

I'll give diskbench a go tonight, looks much easier to interpret than iozone!

I did have some sort of error last night whilst testing things out, was a one off copying a file up to it, and it just hung. so it's not 100% at the moment, which is slightly worrying. not been able to replicate it since either

Nox

 

[nox]

gave it a go, seems to only utilise about 40% of the network speed? using regular windows file copy from the same file went up to 95+%

how odd... oh well, not really too woried about speed now, just want to know if that miss copy is going to rear it's ugly head again, or if it was a one off.

Nox

 

[HelpMoi]

Could someone tell me what the purpose of record length is? I have no idea what it means exactly and how it relates to the benchamrk, any help would be greatly appreciated.. I have been looking it up for ages now and cannot find a simple definiton or anything

 

[thiggins]

When a file is transferred or written to disk, it is broken up into smaller pieces. The length of these pieces is the record size or length.

We show data for the 64KB record length because it is the typical size that Windows uses when applications try to transfer blocks of data that are bigger than 64 K.

 

[HelpMoi]

Thanks for your help, I really appreciate it. I have done a couple of mock tests on an NTFS partition seperate from the Windows one and have noticed that there is a serious drop in performance after filesizes of 16mb +
I'm also not sure why it skipped the 4kb record sizes for the larger files..

I did a write and read test, this is the write test;

For linux does it have a typical record size that it would normally use?

I am going to be doing a few tests for NTFS,FAT,ext2,ext3,jfs,xfs,raiserfs on a single disk and different RIAD setups using an external array.

 

[thiggins]

I'm also not sure why it skipped the 4kb record sizes for the larger files..

I think that's just something that iozone does because the results wouldn't be that informative.

For linux does it have a typical record size that it would normally use?

I don't know, sorry.

 

[HelpMoi]

Thanks again

 

[nox]

wow, bit of a difference!

changed the server OS from XP to 2008 on monday, wasn't until last night I could have any sort of play though, managed to 'lose' my raid 5 array, but it's back now! learnt quite a lot!

Anyways, there is a HUGE difference now, alteast on write speed - read speed seems slightly slower, but negligible. The reliability seems to be up to 100% now, whereas before I was getting a few errors and files were not copying up to it, or down from it fine - also used to get a 'pause' occasionally when watching a film streamed from it. Annoying. Seems to have gone now, and the write speed has doubled too - it's now reporting (in windows) to be over 100MB/s, which i'm VERY happy with!!! I'm guessing it enabled jumbo frames by default though, it's not something i've been in and specifically done yet on the server. Next test... ubuntu, plan was eventually to have this machine as a myth back-end too.

Nox