WHS Performance

[wathman]

You can always throw money at the problem! A Intel PCIe NIC is only $30, and Intel NICs are supported by nearly everything.

Yeah, I second this idea. I ended up doing this with a previous build and I was trying to get Untangle Server up as a firewall. Was having issues with my built in NIC, so I got impatient and bought a Dynex gigabit PCIe adapter from the local Best Buy. Worked perfectly out of the box.

 

[mtber]

I can hook up a monitor and keyboard to the WHS if I get stuck.

I tried some transfers with my current power scheme on Eric and in Performance mode. There wasn't much of a difference between the two. It was interesting that successive runs got faster then hit a plateau.

Performance Power Mode 20, 25, 31, 30 MB/S
Typical Power Saving Mode 18, 22, 25, 28 MB/S

I’ll check out the Realtek drivers link.

I don’t really have the cash to get new NICs for all of my systems at this time, so I hope that something can be sorted out.

I got JPerf running and the results for Eric as client to WHS as server are below with the default parameters.

Anything I should set differently to trouble shoot?

bin/iperf.exe -c 192.168.1.101 -P 1 -i 1 -p 5001 -f M -t 10
------------------------------------------------------------
Client connecting to 192.168.1.101, TCP port 5001
TCP window size: 0.01 MByte (default)
------------------------------------------------------------
[128] local 192.168.1.105 port 49393 connected with 192.168.1.101 port 5001
[ ID] Interval Transfer Bandwidth
[128] 0.0- 1.0 sec 34.7 MBytes 34.7 MBytes/sec
[128] 1.0- 2.0 sec 31.7 MBytes 31.7 MBytes/sec
[128] 2.0- 3.0 sec 32.0 MBytes 32.0 MBytes/sec
[128] 3.0- 4.0 sec 32.0 MBytes 32.0 MBytes/sec
[128] 4.0- 5.0 sec 34.0 MBytes 34.0 MBytes/sec
[128] 5.0- 6.0 sec 33.5 MBytes 33.5 MBytes/sec
[128] 6.0- 7.0 sec 33.9 MBytes 33.9 MBytes/sec
[128] 7.0- 8.0 sec 30.6 MBytes 30.6 MBytes/sec
[128] 8.0- 9.0 sec 32.4 MBytes 32.4 MBytes/sec
[128] 9.0-10.0 sec 32.9 MBytes 32.9 MBytes/sec
[128] 0.0-10.0 sec 328 MBytes 32.8 MBytes/sec
Done.

 

[00Roush]

I usually use this command line for Iperf as it gives results in Mbps... iperf -w 64k -c 192.168.0.2. As Jay_S said, if everything is working correctly you should be in the 900 Mbps range.

Your network setting looked fine except for the different driver versions. I always go directly to the manufacturer for network drivers. They have the most up to date drivers and in many cases extra drivers for different OSes. I would think that both the Vista and WHS (Win XP/Server 2003 just like Jay_S mentioned) drivers should have nearly the same list of options. Makes me wonder if something might be a little off there.

So far I have not read anything in this thread about what exact version of Vista (on Eric) or WHS you are running. Could you let us know which service packs you are running?

00Roush

 

[00Roush]

Almost forgot... if possible try turning off I IPv6 in Vista and WHS. Should be able to do this in the properties for a particular network connection.

00Roush

 

[mtber]

The Os's and versions are summarized below.

WHS Windows Home Server ver 5.2.3790 Service Pack 2 Build 3790
Eric Vista™ Home Premium (32 bit) ver 6.0.6002 Service Pack 2 Build 6002
John XP Media Center 2004 ver 5.1.2600 Service Pack 3 Build 2600
Dorene XP Media Center 2004 ver 5.1.2600 Service Pack 3 Build 2600
Kelsey Vista™ Home Premium (64 bit) ver 6.0.6002 Service Pack 2 Build 6002

I've also attached an Excel file summarizing the NIC configurations on each machine.

I did find some mismatches between duplex and a couple of other settings as noted in the Excel file. Have harminized them as noted, but again no really noticable differences in JPerf results.

I haven't had a chance to update the NIC drivers on Eric and the WHS yet.

I have been looking for drivers for the other machines (Realtek 8211BL & Marvell PHY/nVidia Network controllers) and it's driving me nuts. I can't seem to find anything for them . (I thought that I had one from D-Link's site for the D-link DGE-660TD PCMCIA card in John, but when I try to insall it I get a message that it is the same as alredy installed - even though the version numbers are different)

I turned off IPv6 on Eric, didn't seem to make much difference. Wasn't installed on WHS or either of the XP machines.

I understand the wisdom of focusing on one problem at a time (Eric to WHS), but am a little baffled as to why it seems to be very sub-par performance no matter which machine combos the transfer is tested between. Maybe I'll try another direct connect cross over cable test between two of the systems - if no significant difference, then that eliminates network switch and cables and points to NIC to NIC issue (or software/driver) right?

Maybe I'll have to break down and get a couple of PCI-E cards to try as suggested - just hate to keep throwing money at the problem. (I used to have all of our shared files on the Kelsey Vista 64 machine and built the WHS in hopes of better access performance.)

Thanks for all of the help,

John

 

[Jay_S]

John,

I just noticed, in your jperf results it looks like you forgot to set TCP Window @ 64k.

I was sorta joking about throwing money at more nics. I don't think you should need to do that. I have the same 8111C in three computers, including my unRAID server. Unfortunately, unRAID doesn't ship with iperf and I don't know how to add it. But jperf between my gaming PC and HTPC looks like this:



A bit under 800MB/s. The only thing I tweaked was TCP Window Size - to 64k from whatever XP defaults to (I think 8k). I'm losing a little speed right now because I'm transcoding video over my network

On one side is a Gigabyte GA-G31M-ES2L. Realtek 8111C with driver version: 5.720.327.2009 (3/27/2009). XP Pro x86.

The other is a Gigabyte GA-MA74GM-S2, 8111C, driver version 5.708.1030.2008 (10/30/2008). XP Pro x86.

I wired my house with cat6, through a patch panel, through a TRENDNet TEG-S80g unmanaged gigabit switch.

Re-try your jperf test with TCP WIndow set to 64k. I bet you get better results.

Try jperf/iperf between some other machines. Any machine with a gigabit NIC on the PCI express bus should do 800+ MB/s, assuming everying's working properly. All of your machines except for John appear to have their NICs attached to the PCIe bus or directly to their chipsets. This should all be good enough. Try between any of them except for John.

 

[mtber]

Awesome,

Here's the JPerf output (Kbits/sec) with TCP Window Size set to 64 K.

So, does this mean that I need to set my Vista and XP systems to 64 K TCP Window size? (or determine optimal if I get really adventeruous)?

(and update the drivers that I can find)

Thanks,

John


bin/iperf.exe -c 192.168.1.101 -P 1 -i 1 -p 5001 -w 64.0K -f k -t 10
------------------------------------------------------------
Client connecting to 192.168.1.101, TCP port 5001
TCP window size: 64.0 KByte
------------------------------------------------------------
[128] local 192.168.1.105 port 49936 connected with 192.168.1.101 port 5001
[ ID] Interval Transfer Bandwidth
[128] 0.0- 1.0 sec 97664 KBytes 800063 Kbits/sec
[128] 1.0- 2.0 sec 105656 KBytes 865534 Kbits/sec
[128] 2.0- 3.0 sec 111296 KBytes 911737 Kbits/sec
[128] 3.0- 4.0 sec 109440 KBytes 896532 Kbits/sec
[128] 4.0- 5.0 sec 101240 KBytes 829358 Kbits/sec
[128] 5.0- 6.0 sec 99240 KBytes 812974 Kbits/sec
[128] 6.0- 7.0 sec 110872 KBytes 908263 Kbits/sec
[128] 7.0- 8.0 sec 110896 KBytes 908460 Kbits/sec
[128] 8.0- 9.0 sec 110280 KBytes 903414 Kbits/sec
[128] 9.0-10.0 sec 110320 KBytes 903741 Kbits/sec
[128] 0.0-10.0 sec 1066912 KBytes 873927 Kbits/sec
Done.

 

[00Roush]

Awesome,

Here's the JPerf output (Kbits/sec) with TCP Window Size set to 64 K.

So, does this mean that I need to set my Vista and XP systems to 64 K TCP Window size? (or determine optimal if I get really adventeruous)?

(and update the drivers that I can find)

Thanks,

John


bin/iperf.exe -c 192.168.1.101 -P 1 -i 1 -p 5001 -w 64.0K -f k -t 10
------------------------------------------------------------
Client connecting to 192.168.1.101, TCP port 5001
TCP window size: 64.0 KByte
------------------------------------------------------------
[128] local 192.168.1.105 port 49936 connected with 192.168.1.101 port 5001
[ ID] Interval Transfer Bandwidth
[128] 0.0- 1.0 sec 97664 KBytes 800063 Kbits/sec
[128] 1.0- 2.0 sec 105656 KBytes 865534 Kbits/sec
[128] 2.0- 3.0 sec 111296 KBytes 911737 Kbits/sec
[128] 3.0- 4.0 sec 109440 KBytes 896532 Kbits/sec
[128] 4.0- 5.0 sec 101240 KBytes 829358 Kbits/sec
[128] 5.0- 6.0 sec 99240 KBytes 812974 Kbits/sec
[128] 6.0- 7.0 sec 110872 KBytes 908263 Kbits/sec
[128] 7.0- 8.0 sec 110896 KBytes 908460 Kbits/sec
[128] 8.0- 9.0 sec 110280 KBytes 903414 Kbits/sec
[128] 9.0-10.0 sec 110320 KBytes 903741 Kbits/sec
[128] 0.0-10.0 sec 1066912 KBytes 873927 Kbits/sec
Done.

Looks like things are working just fine on the network side, but I would still update your drivers.

Well you can't really set Windows to use a particular window size. You can set the maximum but generally from Windows 2000 and up the window size is auto tuned. In my experience Windows will use a window size of 60k-64k on a small local LAN.

So now you have to move onto other things that could cause your problems. It is quiet odd though that all of you machines are having slow file transfers. Have you tried switching your test to using a single large file of around 512 MB-1GB?

00Roush

 

[mtber]

Updated the drivers on Eric (Vista 32) and WHS. No significant difference in Windows Explorer file copy speed.

Tried a single file 1GB copy versus same files as 351 inviduals in a folder. Single 1GB file transfered at 50 MB/Sec versus 26 for the collection.

Obviously nothing is simple ... The reason I asked about setting the TCP Window Size to 64 kbits is that when I Googled setting teh Window size I saw lots of web pages about problems with Auto Tuning and instructions for XP registery editing to set the window to 64 K. (http://www.psc.edu/networking/projects/tcptune/OStune/winxp/winxp_stepbystep.html) So, not a good idea to jump to any of those measures (at least yet) in my case?

I don't have any data to support it, but I suspect that my slow transfer speeds problem existed before I added the WHS. I wonder (but don't know how to investigate) if there is some underlying conflict problem that causes re-sends as the copy progresses. I had noticed that on very large transfers using the Vista 64 system (Kelsey) that they would start off fast and in short order slow to a crawl (<10 MB/s, like down to 2-3).

Thanks,

John

 

[Jay_S]

Awesome,

Indeed! This confirms that your NICs, cables and switch are OK. And defines the upper limit of what's possible. Now you have to test everything resting on top of that.

You'll need FAST drives on both ends. Once file sizes exceed available cache (drive cache, system RAM, CPU cache, etc.) of either system, your transfer speed will not exceed the speed of that systems HDD. For example, your WHS has 2GB of ram, Eric has 4GB. If you transfer a file larger than 2GB from Eric to WHS, you exceed WHS's cache and will be limited by the physical ability of the WHS's HDD to write data to the platters - regardless of Eric's performance.

You can use iozone to test local disk speed. As Tim suggests in post #4, make sure you test file sizes larger than the available system ram. If you have 2GB of ram, test up to 4GB file sizes. Your original iozone command:

iozone -Rab Test2.wks -i 0 -i 1 -+u -f s:\test\test.tmp -r 64k -n 64k -g 1g -z

...tests from 64k to 1GB. Bump that up to something higher. I started at 256m for a run this morning:

So, roughly 73-77 Mb/s non-cached writes, 115 MB/s non-cached reads. This is the fastest drive in my house.

Run iozone locally on Eric's WD Blue drive (which should be the 2nd fastest drive you own, next to the 7200.12 in WHS). Let's see where that goes.

 

[Jay_S]

I had noticed that on very large transfers using the Vista 64 system (Kelsey) that they would start off fast and in short order slow to a crawl (<10 MB/s, like down to 2-3).

Kelsey has the Seagate 7200.9. I also own this drive, and they're slow. See Anandtech's HD Tach results: at best 65MB/s sequential reads. At worst around 35 MB/s. And I've found HD Tach to be pretty optimistic! Either way, assuming it's really capable of 65MB/s, once you factor in SMB peculiarities, file copy engine performance, network protocol overhead, etc., I wouldn't be surprised if you saw half of that in real life.

This doesn't explain why your performance drops off like that. I'm just saying you're not starting with a quick drive.

 

[mtber]

If I can get these issues sorted out, I may need to put a better drive in Kelsey since I pirated the better 500 GB drives out of her system to build the WHS.

Ok, a couple of plots (full sacle and expanded) from two IOZone runs from Eric to WHS today (with read/write data from 18-Sep for comparison) are attached.


Thanks,

John

 

[00Roush]

If I can get these issues sorted out, I may need to put a better drive in Kelsey since I pirated the better 500 GB drives out of her system to build the WHS.

Ok, a couple of plots (full sacle and expanded) from two IOZone runs from Eric to WHS today (with read/write data from 18-Sep for comparison) are attached.


Thanks,

John

How full is the drive in Eric?

00Roush

 

[mtber]

used 159 GB, free 305

defrag is scheduled on a weekly basis

 

[mtber]

WHS Disk Usage

The primary is 23% full and the two 500 GB drives are each 36% used.

 

[00Roush]

With your current disk usage levels I still think you should be able to see higher speeds between those computers for the 1 GB file you tested earlier. From looking up the information about your drives I would guess that the disks would be capable of reading and writing at around 60-70 MB/sec in there current state. Add a bit of overhead for the file transfer and in my opinion you should be able to see 55-65 MB/sec. This is just a guess though and it also assumes that no other software or service is trying to use the disks. Anti virus/spyware software, file duplication, and system restore can all interfere with file transfers. So if possible disable all of these when testing. I suppose your 50 MB/sec for the 1 GB file could be right but I still think it is too low given your hardware.

I think Jay_S wanted you to test your C drive in Eric with Iozone and see what kinds of speeds you get for just local reads and writes. A command line something like this iozone -Rab c:\iozone\results.xls -i 0 -i 1 -+u -f c:\test.txt -y 64k -q 64k -n 256m -g 8G -z. It would be great if you could test the drives on your WHS machine as well but I don't know if you can use iozone on WHS. Basically this gives a good idea of what your actual read and write speeds for the drives you are using in their current state. If file copies over the network are within 5-10 MB/sec of the speeds seen locally chances are you won't see much better unless you swap to faster drives.

Do you have Cool n Quiet enabled on your server? I noticed in your screen shot from post #7 that you were at 40% of maximum frequency so I assume at least that machine is using Cool n Quiet. If so you might consider disabling it in your BIOS while testing. I just recently did some testing at home and found that when enabled on the server or client transfer speeds went down. When enabled on the server it was more noticeable as it dropped read and write speeds down 10-20 MB/sec. Not sure if it is just my setup or what but definitely something to consider.

I say you take your WHS and Eric machines and slim down all of the software to just the bare essentials running. Is any particular software on all of your PC's that might be causing a slow network. It could be something that looks harmless. Virus... Spyware? Figured I would throw that out there. Also make sure you are not playing any media while you are trying to transfer files to/from a Vista PC. Try using just IP addresses to connect to your WHS machine.

If you get a chance could you get a screen shot of the Resource Monitor during a 1GB file transfer writing from Eric to WHS or reading from WHS to Eric?

00Roush

 

[mtber]

Thanks. I'll try those suggestions as I gat a chance to tinker this week. I would like to create a table of measured HD performance and target throughput for all of my systems. Then I could see how close each is so that I'm not wasting everyone's time on chasing diminishing returns. As a get a chance to run the single disk IOZone tests, I'll add the info to the table.

I did run HD Tune on all of the drives last night - I'm trying to figure out how to poast in table format so it would be easy to compare.


Would using the average be a good approximation of max rate to expect to be able to approach? (knocking off some for other inefficiencies, etc?)

Thanks,

John

 

[mtber]

WHS primary drive

 

[Jay_S]

Hunting bottlenecks can be really frustrating. One might have a nice raid controller capable of 800MB/s local read/write performance, but that doesn't matter if you're bottlenecked by a gigabit LAN. One might have a gigabit LAN, which doesn't matter if the PC on the other end is incapable 100MB/s I/O. And so on.

After you've done a fair bit of bottleneck hunting, you'll settle on a methodology that works for you. My methodology has been developed around free software and lots of thinking, mainly because I have more time & energy than I have money! I start by testing low-level stuff first (the network infrastructure: NICs, cables, switches, etc.), using jperf/iperf. Then moving up to HDDs, using IOZone locally. I've found IOZone to provide a more useful picture of HDD performance than any of the HD Tune/Tach/ATTO etc., mainly because the user can custom tailor the test to better simulate their specific application. Then I test network HDD performance using IOZone. And finally I do some timed network file copy tests.

This gives 4 data points:

1. iperf results = the upper limit. Period. You can't exceed the "wire speed" of your network.
2. local IOZone results = the upper read/write limits to your client PC's HDD.
3. remote IOZone results = the upper read/write limits for networked file transfers.
4. network file copy tests = the actual read/write limits for networked file transfers.

You'll discover that network I/O is much slower than local I/O. This isn't surprising, but the reasons are not obvious. And there are lots of reasons, unfortunately! And most do not lend themselves to direct testing, so lots of research, discussion and reasoning is needed. Two reasons I am most comfortable discussing are:

1. Network transfers will always be slower than local transfers because of the way our networks work. Each layer of the network protocol stack chops up data and encapsulates it before sending it to the next network layer. Data is chopped and encapsulated several times before being sent out on the wire. The meta-data that encapsulates the actual data is "wasted space" on the wire - or, as it's typically called, overhead. This is why tricks like large TCP windows, jumbo frames, etc., sometimes work - they permit sending a higher ratio of data to meta-data (more data per header). Plus, TCP is a "reliable" protocol (it requires acknolwedgements), which adds another slow-down. Applications have their own overhead as well. Often, FTP file transfers are faster than Windows Explorer drag-n-drop file transfers because of what Explorer adds on top of all the overhead present in TCP be default. FTP is less convenient, though.
2. As you discovered in post #14, Vista copies faster than XP. Vista SP1 brought a lot of improvements to the windows file copy engine. I previously posted a link to Tim's The Vista Difference, in which he posted a link to Mark Russinovich's Inside Vista SP1 File Copy Improvements that discusses this at length - well worth the read. 00Roush has written about this as well in his sticky.

 

[mtber]

Jay_S & 00Roush, thanks for all of your help.

I'll get local IOZone tests run in the next couple of days.

My objective in all of this is just make sure my set-up is not unnecessarily hamstrung. I don't need to tweak the maximum performance out of it, just want to make sure my stock Camero isn't running like a Chevette.

After I get data on all four of Jay_S's points, I guess there will be a pretty good picture of where things stand.

I have taken a look at the XP versus Vista file copy links. I wonder what the story will be with Windows 7? I plan to get the Home Premium Family Pack and move the Eric, Kelsey, and Dorene systems to Win 7 in the next few months.

Thanks again,

John