It must be pretty difficult to consider changing the testing setup when you have a ton of existing results representing several different vendors and time and effort spent.
However, I wish to say that I've never really been happy with the iozone results -- from the onset, where it produced some very pretty and impressive-looking 3D graphs, which were on closer examination pretty useless, to later on, when for supposedly networked remote machine tests, the results would sometimes show performance better than possible with the given network speed.
Strip out the material which is IMO misleading due to significant cache impact, and you're left with something which gives you very little to graph, and can be shown with several other tools, perhaps more quickly and reliably.
Well, enough ranting. I think that the steps you've taken to test the testbed are great, and I strongly encourage you to go farther. For one, as some posters have done here (e.g. Dennis Wood), you should IMO ground your tests with real world application performance. If the tests do not give you results which even roughly correspond to real file transfers, they're not useful, and you should change the tests. The "small file tests" IMO are in this category -- if you did real small file transfers, you'd never see performance exceeding the network bandwidth, and would very often see performance much less than that for large files.
Intel has recently provided a tool which is interesting. They're apparently captured a bunch of different application access patterns, bunched them together in a test suite, with an executor. This is really good in theory, but still needs to be explicitly correlated with real-world tests IMO. There are a couple of things that I don't like about the Intel test -- e.g. it's very insular about the platforms it supports, but if you happen to get past this, and also validate the test for a specific client against some meaningful real-world benchmarks, then perhaps the test may be useful.
I understand that testing is a multi-dimensionally complex endeavor, and that even something seemingly as simple as file transfers vary bizarrely in practice, but I feel that the oddities in the results represent motivation and opportunity for improvement.
[Article Link]
However, I wish to say that I've never really been happy with the iozone results -- from the onset, where it produced some very pretty and impressive-looking 3D graphs, which were on closer examination pretty useless, to later on, when for supposedly networked remote machine tests, the results would sometimes show performance better than possible with the given network speed.
Strip out the material which is IMO misleading due to significant cache impact, and you're left with something which gives you very little to graph, and can be shown with several other tools, perhaps more quickly and reliably.
Well, enough ranting. I think that the steps you've taken to test the testbed are great, and I strongly encourage you to go farther. For one, as some posters have done here (e.g. Dennis Wood), you should IMO ground your tests with real world application performance. If the tests do not give you results which even roughly correspond to real file transfers, they're not useful, and you should change the tests. The "small file tests" IMO are in this category -- if you did real small file transfers, you'd never see performance exceeding the network bandwidth, and would very often see performance much less than that for large files.
Intel has recently provided a tool which is interesting. They're apparently captured a bunch of different application access patterns, bunched them together in a test suite, with an executor. This is really good in theory, but still needs to be explicitly correlated with real-world tests IMO. There are a couple of things that I don't like about the Intel test -- e.g. it's very insular about the platforms it supports, but if you happen to get past this, and also validate the test for a specific client against some meaningful real-world benchmarks, then perhaps the test may be useful.
I understand that testing is a multi-dimensionally complex endeavor, and that even something seemingly as simple as file transfers vary bizarrely in practice, but I feel that the oddities in the results represent motivation and opportunity for improvement.
[Article Link]
