Looking for fast, big and reliable storage solution for video use

[Captations]

Hello everyone

I need a new storage solution to work with video editing software : premiere pro, da vinci resolve etc...

I'm looking to have between 16 to 24TB.
I don't like USB3. So I could work with esata or better yet, a special pci card i could insert. It does not necessary have to be networked.
It has to be very fast (400 to 800MB/s, so 3 to 6Gbs), and good access time.
I looked for 8 bay apparatus, quite oftenly NAS (like synologi 1818? but I'm not sure that's what I need.
I thought about putting P300 drives ? I'm contemplating Raid 5 or 6 if it can be quick. Or 10 if that helps.
I'm using win7 x64.
Can anyone put me in the right direction ?

Thank you !

 

[System Error Message]

on the low end, premade nas is an option. For the mid or high ends, DIY NAS is the best.

For example you could put together a decent CPU, board, ram, combined with a 2U case that has the front full of hotswap bays of the drive size you plan to use. You could use a tower that has many 5 inch drive bays and just buy lots of hotswap bays for the front instead.

I recommend SAS/PCIe over onboard sata for raid as sata has the habit of tying up other drives if 1 drive has an issue.

Raid 5 or 6 is better than 10 if you plan to expand in the future. Neither are quick when rebuilding.

Windows is a poor choice for a NAS, go use linux. I myself use openSUSE.

 

[thiggins]

If you want fast and don't have to share, direct attached is best. If your PC supports Thunderbolt, then QNAP has good Thunderbolt options.

 

[Captations]

Hi guys
Thank you so much for your answers.
@thiggins, alas my mobo does not support thunderbolt (it is a high en d asus p9x79WS-E) or at least it seems that pcie thunderbolt cards are not compatible with it. So the next best thing would be usb3.1 (10Gbs), which I do not know how it would perform.
@SysErrMessage
How do you think I should attached that home made apparatus with my video station ? Through LAN ? 10Gbs ?

 

[System Error Message]

SFP+. If its just going to be a drive you'll attach, you can go for thunderbolt. You can get thunderbolt based PCIe cards.

You should first determine the bandwidth you'll need.

 

[Captations]

SFP+. If its just going to be a drive you'll attach, you can go for thunderbolt. You can get thunderbolt based PCIe cards.

You should first determine the bandwidth you'll need.

Thunderbolt would be fine, but there is no thunderbolt pcie cards for x79 chipset based cards...
So the next best thing is either USB 3.1 or 10Gb lan.
What woudl be the best in your opinion ?

Oh, and a real life bandwidth of 600/800MB would be great, if not to much latency. It is a bad experience to edit with a sluggish system.

Here are my specs : i7 4930K, 64G ram, asus P9X79WS-E, GTX1080

 

[System Error Message]

usb 3.1 can do 1GB/s if it is 10Gb/s. If you want your system not to be sluggish, you should do these few things.
First you should grab a couple of SSDs and raid 0 them on PCIe. You can put your OS or software here including the files you are working on. 2SSDs easily achieves 1GB/s so make sure you have at least 4 lanes of PCIe for them.
Then for your massive storage, you can get a multi port usb 3.1 card, maybe 4 ports, connect 4 hard drives and set them up in raid 5 or raid 0 via the OS. You could also get an enclosure that can fit 4 drives but that would cost more.

 

[Captations]

Hello
I already have a pretty fast volume for OS, software and cache (raid 0 of samsung Pro SSD)
I'm contemplating a Qnap solution of 4 to 8 disks in raid 10, in either usb 3.1 or 10GBe.
Do you know how fast such a system would be for video editing ?
Should I go the USB or GBe way ?

 

[System Error Message]

for your needs, qnap is too pricey because you dont need such software.
Lets build from scratch. Get a midi tower case filled with 5 inch bays. Buy 5 inch hot swap drive bays.
Grab a used CPU and board with DDR3 ram. Even a used server will be fine, but at least have it be the iseries or amd bulldozer.
Use the onboard sata to connect up to 6 drives or more (depending on motherboard).
Use one drive for the OS to set up software raid
Install a used sfp+ card on both PCs

You now have a very high performing storage server with SFP+ and using SFP+ to SFP+ direct is much cheaper than used 10GBe cards. However the issue with sata is that if 1 drive is holding up the PC, all drives will be on hold. Im sure even the qnap uses sata too. You could get a SAS card and just use the SAS to many sata it offers.

Its still highly dependent on if you need network connectivity or not. From what i've seen, the best low end NAS is a prebuilt one. The best high end NAS is a self built one.

 

[Captations]

OK. What do you think of that :
- I buy 8 sata Hdds and put it in a box.
- I buy a raid controller with two mini SAS plug external like so :
https://www.amazon.com/dp/B01M6TV7LE/?tag=snbforums-20
- I take Dual Mini SFF8087 to SFF8088 adapter for having an external plug
- I buy two SFF8088 to split 4 sata cable for joining everything.

Do you think that would work ?

 

[System Error Message]

it would but are you gonna place it into your own PC or another PC on the network?
It doesnt matter if you place it in your own PC or another but just make sure you can give it at least 4 PCIe lanes.

Raid 5 will give you performance + 1 redundancy. You can go with raid 6 for 2 redundancies. I suggest against raid 10 because the redundancy isnt as flexible as with raid 5 and 6 where raid 5 and 6 allows losing any 1-2 drives and still function while raid 10 only allows losing specific drives despite raid 5/6 being more cpu expensive however from my experience i've been running 5 drives in raid 5 on an old phenom ii with the main CPU use being SFTP.

Running this in your own PC is the fastest. If you dont need network access you can save yourself the cash by having it in your own PC.

 

[abailey]

I would strongly urge you to stay away from RAID5. Raid5 used to be very popular when we all had small hard drives. With larger hard drives you are looking at many of tears if there is a failure. On the business side I don't know anyone that recommends Raid5 anymore. The chances of a catastrophic failure after you loose a drive (before a new one can be rebuilt) is too high. If you want performance go RAID10, if you want better price/storage ratio go with RAID6.

 

[Captations]

Hi SysErrMessage
Thank you for all your time.
I thought about placing the raid card in my own PC, but putting the 8 HDDs in another "box". I'm having trouble getting that, a box to put HDDs in. Specifically regarding power...
I don't really need a great redundancy scheme because all the datas on said raid will be backed up with cold storage (HDDs). Less CPU use is better though; since the software I use are very CPU intensive.
I don't need network access.

So I need a good Raid Card, and find "the box".
I think I'm going to go with the seagate ironwolfs. What do you think ?

 

[degrub]

go to Areca's website. i think you will find what you want there.

 

[sfx2000]

Hello everyone

I need a new storage solution to work with video editing software : premiere pro, da vinci resolve etc...

I'm looking to have between 16 to 24TB.
I don't like USB3. So I could work with esata or better yet, a special pci card i could insert. It does not necessary have to be networked.
It has to be very fast (400 to 800MB/s, so 3 to 6Gbs), and good access time.
I looked for 8 bay apparatus, quite oftenly NAS (like synologi 1818? but I'm not sure that's what I need.
I thought about putting P300 drives ? I'm contemplating Raid 5 or 6 if it can be quick. Or 10 if that helps.
I'm using win7 x64.
Can anyone put me in the right direction ?

Thank you !

You didn't say what platform you're working with, and the use cases...

Thunderbolt DAS is definitely the right path for a stand-alone workstation - and that's a local storage box - not sure how the QNAP items perform next to a direct Thunderbolt chassis there, but I suspect that the chassis is faster...

10Gbe is good for access to a shared content repo...

Going to a 10Gbe network fabric - it's all about the NIC's, OS support and Switches - some better than others - there's a couple of good articles on the main site that discuss that aspect on workflow.

 

[System Error Message]

Hi SysErrMessage
Thank you for all your time.
I thought about placing the raid card in my own PC, but putting the 8 HDDs in another "box". I'm having trouble getting that, a box to put HDDs in. Specifically regarding power...
I don't really need a great redundancy scheme because all the datas on said raid will be backed up with cold storage (HDDs). Less CPU use is better though; since the software I use are very CPU intensive.
I don't need network access.

So I need a good Raid Card, and find "the box".
I think I'm going to go with the seagate ironwolfs. What do you think ?

Its cheaper ofcourse to have it internal. You just need to look for sata based hot swap HDDs for 5 inch bays. I still would advice against raid 10. Either pick raid 0 or raid 5 as the CPU load of raid 5 is pretty much low.

seagate ironwolfs should be fine, but i havent had the opportunity to use them yet as i have them myself but havent yet put them to use. There is a website that tells you the failure rate of various drives and capacities. The capacity matters as well. Since you are going for speed you dont need very large drives. Infact you could fill up your drives with WD blacks in raid form for performance.

When it comes to big files, hard drives are fast at those, slow at small files.

You didn't say what platform you're working with, and the use cases...

Thunderbolt DAS is definitely the right path for a stand-alone workstation - and that's a local storage box - not sure how the QNAP items perform next to a direct Thunderbolt chassis there, but I suspect that the chassis is faster...

10Gbe is good for access to a shared content repo...

Going to a 10Gbe network fabric - it's all about the NIC's, OS support and Switches - some better than others - there's a couple of good articles on the main site that discuss that aspect on workflow.

While a NAS would be elegant, OP only needs to use it for 1 PC. a raid card has more than just 10Gb/s especially the one listed being PCIe x4 V3 giving quite a lot of bandwidth, more than 10Gb/s. While having a NAS is nicer for network access, the raid card is what makes the difference in bandwidth and access. However SFP+ will always be better than 10Gbe mainly because SFP+ direct offers infiniband, dma and many of the good features that improve overheads and latency.

 

[Captations]

I still would advice against raid 10. Either pick raid 0 or raid 5 as the CPU load of raid 5 is pretty much low.

I'm on the fence between RAID 10 and RAID 0.
I'm unsure about Raid 5 or 6 as it appears that the write performance is not optimal and the reconstruction of a 6TB drive in Raid 5 or 6 seems to be looooooooooong and maybe not succesful ?

 

[thiggins]

If you lose one drive in RAID0 your data is gone. Do you really want to take that chance?

 

[System Error Message]

I'm on the fence between RAID 10 and RAID 0.
I'm unsure about Raid 5 or 6 as it appears that the write performance is not optimal and the reconstruction of a 6TB drive in Raid 5 or 6 seems to be looooooooooong and maybe not succesful ?

raid 10 allows up to 2 drives to fail but are very specific about which drives can fail. Raid 5 and 6 allow for failure of any 1 or 2 drives and read/write are for raid 5 and 6 are dependent on the number of drives -1 for raid 5, but raid 5 uses a bit more c alculations that even a bulldozer cpu has no issue keeping up

 

[Captations]

If you lose one drive in RAID0 your data is gone. Do you really want to take that chance?

I'm backing up every content of that RAID on cold storage. So losing it is not that big a chance. I'm always backing up my rushes and source videos at least once.