High Speed data storage with 20+ users

[splintah]

Need High Speed data storage with 20+ users

Hey there,

we are looking for a speedier NAS.

at the moment we have a raid 5 linux machine with fedora
core2duo, 2gig ram
4x 2TB WD green
hp pro curve switch

there are about 25 different computers accessing the files at the same time.
the files are up to 2.5gigabyte
some files are only 65K depending on what they are doing
if they are copying the final render its 2.5 gig
if they are rendering they are constantly accessing and writing 65k to 1 mb files
so there is LOADS of traffic.

the question is
how can we achieve better availability of the data
we have the problem that when all machines are rendering (compositing rendering = high data flow)
the workstations cant work off the server anymore because everything is ultralame. also the rendermachines spend more time waiting for the data than actually rendering.

so we need higher network speeds urgently

is there any possibility to get higher availability and higher throughput ?

we found that adaptec raid controller with ssd caching.
would that solve many issues ?

 

[00Roush]

Are you looking to replace your current setup completely or just upgrade it? How quickly do you need higher network speeds. Most core2duo setups should have enough CPU power to support 100+ MB/sec sustained network throughput. Moving to 4 GB of ram might help. The Adaptec RAID card also might help. It really does depend on where your bottleneck is though. My guess is it is either a network bottleneck or a hard drive bottleneck. Also I am not sure of Fedora... (haven't used it) do you know if the OS is setup optimally? If you only have a single gigabit connection that is being shared between 20 clients the best you can hope for is about 6 MB/sec for each client. If you need more throughput than that a dual port NIC would probably help. With only 4 hard drives chances are your disks are the main bottleneck. A quick test with my server here at home (3 disk RAID 5) shows that under heavy load (random and sequential reads/writes from multiple computers) my disk throughput falls to about 30 MB/sec. If I had 20 clients reading and writing data at the same time each one would probably at most get about 1.5 MB/sec worth of throughput.

Since I am not sure if you are looking to replace or upgrade your current setup I will just give you an idea of what I might look at doing.

Most of the time I start with the OS first. Making sure everything is setup to provide the best performance possible. For example this might include just doing some performance tests with a single client to see if a best case scenario can do 100+ MB/sec. I might try out some other OSes... FreeNAS, NexentaStor, Ubuntu Server, or Windows Server as I know these OSes have good performance right out of the box.

As for hardware... With 20+ clients needing the fastest data transfer speeds possible I would consider at least getting a dual port gigabit NIC. Then based upon how important my data is I would look at my disk setup. If I really don't need a lot of total storage and I could deal with losing my data I would go with a RAID 0 array of as many small fast disks I could afford. Another option would be a 2 disk RAID 0 array of SSD drives. Heck a single SDD could support that many clients. If I could stand losing a days worth of work I might still do a RAID 0 array for the main storage then a 2 disk RAID 1 array that would sync every night. If no data can be lost I might start looking at ZFS for the filesystem. Get about 5-7 disks or more with a SSD for a cache drive and performance should be good.

Not sure if that helps or not. Let me know.

00Roush