First DIY NAS, looking for combo HTPC & 30 TB FreeNAS

[Mr. Bungle]

I'm new to FreeNAS, ZFS and RAID-Z, but I've been doing some research on a first build and would greatly appreciate some feedback.

My ideal system would be a combined HTPC and NAS box with up to 30 TB raw disk space, running Windows 7 and virtualized FreeNAS. I'm not sure if this is all possible - just an ideal that I'm shooting for. My budget is flexible - trying to stay on the cheaper end, but I don't want to skimp.

Here is the list of essential hardware that I've put together:

- Asus E25M1-M PRO motherboard (AMD Zacate E-350 APU, 1.6GHz dual-core) - http://www.newegg.com/Product/Product.aspx?Item=N82E16813131697
- (8x) Hitachi Deskstar 0S03230 3TB 5400 RPM SATA 6.0Gb/s HDD - http://www.newegg.com/Product/Product.aspx?Item=N82E16822145493 (will probably add 2 more drives within a few months to end up with 30 TB raw, ~24 TB in RAID-Z2)
- G.Skill Ripjaws 8GB (2 x 4GB) DDR3 1333 (Model F3-10666CL7D-8GBRH) - http://www.newegg.com/Product/Product.aspx?Item=N82E16820231402
- HighPoint RocketRAID 2720SGL SATA/SAS controller card - http://www.newegg.com/Product/Product.aspx?Item=N82E16816115100

Some notes about intended usage and concerns:

- I'm confident that this hardware is more than capable of handling my HTPC needs (up to 1080p smooth Blu-ray playback). As I understand it, the GPU has hardware decoding that will keep the CPU mostly free (worst-case, ~25% load). However, I already own a sufficient HTPC, so using this build for double-duty (HTPC & NAS) is just gravy, and would serve to simplify my home setup. I want this to be a capable NAS first-and-foremost, so I can scrap the HTPC plans if needed.
- I'm overwhelmingly concerned about data integrity above all else. Performance, compression, and other features aren't very important to me aside from that. I'll be more than happy if I can just get half-decent file transfer speeds over the network. Downtime is completely unimportant. I just don't want to lose any data, so I hope to run RAID-Z2.
- I plan to use this mainly for music, movies, and photos, but also Time Machine backup for 2 Macbooks and a Mac Pro. From the HTPC front - Netflix streaming and watching DVDs and Blu-rays. It doesn't need to interface with Windows except for the HTPC's OS within the same box.

Questions:

- Am I trying to do too much with too little hardware? Do I need to move up to more RAM and a more powerful processor?
- Will 8 GB RAM and the E-350 APU be enough horsepower to run ZFS/Z-RAID2? If so, is it close enough that I might want to avoid dual-tasking as an HTPC?
- I chose this motherboard/CPU combo mainly due to its low power consumption and supposed ability to handle 1080p video on the cheap. The form factor is unimportant, though. This thread (http://forums.freenas.org/showthread.php?27-AMD-E-350-Thread-(now-in-new-forum-)&) suggests that the CPU might be a bottleneck with this motherboard. If that's true, should I consider a motherboard and a more powerful discrete CPU? This may be a dumb question, but since the system will usually just be idling (I plan to leave it on 24/7), am I going to save much power anyway - or would a more powerful CPU still use little power at idle?
- I'm assuming that I can use any low-cost HBA/RAID Controller since I won't actually need it for its RAID capability (since I'll be using RAID-Z2). Is that correct? If so, is there a cheaper option even than the HighPoint 2720?
- The drives I'm using (I've already bought them) are SATA 6.0 Gbps, so I originally intended to buy other hardware with support for that. Will that even matter, though, assuming that all traffic in & out of the system will be via gigabit ethernet (i.e., will the latter be my bottleneck anyway even with SATA 3.0 Gbps)?
- Following up on the previous question, would even a couple of PCI 4-port SATA HBA cards (e.g., http://www.newegg.com/Product/Product.aspx?Item=N82E16816124028) or even a port multiplier card (e.g., http://www.newegg.com/Product/Product.aspx?Item=N82E16816124043) be sufficient to not cause a bottleneck with network transfers? Do I need to worry about FreeBSD compatibility if I'm just using them for extra SATA ports (no RAID other advanced features)?
- Will I have any issues creating a RAID-Z2 setup with 8 drives, then adding 2 more identical drives in a few months? Or am I better off just buying 2 more now to start with a 10-drive setup from the get-go?
- If I ultimately go with the HighPoint 2720SGL and need/want the 6.0 Gbps capability, do I need special cables? Will some cheap cables from MonoPrice do the trick (e.g., http://www.monoprice.com/products/p...=10254&cs_id=1025406&p_id=8186&seq=1&format=2) or do I need to get ones built to a certain standard? Unlike with SATA cables, I don't see any SFF-8087 breakout cables specified as SATA II/3.0 or SATA III/6.0.
- Any recommendations on USB thumb drives for the boot volume (brand/size/model/USB 2.0 or 3.0), or does it matter that much?
- Anything else I missed, or gotchas to be aware of? Would anyone recommend different hardware or a different approach?

Sorry for the barrage of questions... Hopefully based on all this info, you can get a feel for what I'm looking for and what I'm still unsure about, and help to narrow the scope of my research a little further. Any help would be very much appreciated!

 

[GregN]

Looks like you've really been doing your research, very cool. Very ambitious.

I can get this started, help with more details as things get refined.

I'm not a fan of overloading, make a NAS a NAS, an HTPC a HTPC, you are layering complexity and possible failure modes on top of each other, something breaks you lose two machines, not just one. And there are performance questions, streaming HD during a large backup? Some performance can be gained HTPC going iSCSI (sorta what I've done), the HTPC is the NAS, but the disk array runs separately.

I'm not sure about the virtualizing of FreeNAS, it is possible, but there have been reports of disk limitations. Which Hypervisor you thinking of?

In terms of layers, you'll have you VM executive, Windows 7, FreeNAS, and then ZFS on top of that. Plus whatever media center you are running, 7MC?

To your questions

- Am I trying to do too much with too little hardware? Do I need to move up to more RAM and a more powerful processor?

As I suggest, I think you are doing too much with the machine...

- Will 8 GB RAM and the E-350 APU be enough horsepower to run ZFS/Z-RAID2? If so, is it close enough that I might want to avoid dual-tasking as an HTPC?

I think that there are enough questions around performance impact, that single tasking the machine would be a good idea.

8gig core is more than enough for most everything these days.

- I chose this motherboard/CPU combo mainly due to its low power consumption and supposed ability to handle 1080p video on the cheap. The form factor is unimportant, though. This thread (http://forums.freenas.org/showthread.php?27-AMD-E-350-Thread-(now-in-new-forum-)&) suggests that the CPU might be a bottleneck with this motherboard. If that's true, should I consider a motherboard and a more powerful discrete CPU? This may be a dumb question, but since the system will usually just be idling (I plan to leave it on 24/7), am I going to save much power anyway - or would a more powerful CPU still use little power at idle?

Not sure, have you considered an i3 if going with VMs?

I think this is down to usage patterns, under normal circumstances, the board would be fine. I personally prefer ATOM for lower power.

- I'm assuming that I can use any low-cost HBA/RAID Controller since I won't actually need it for its RAID capability (since I'll be using RAID-Z2). Is that correct? If so, is there a cheaper option even than the HighPoint 2720?

I think just a SATA controller (4 port PCIe 2.0 x8 ) with port multipliers would be the way I would go. Start with one controller (12 drives), and see the performance, bump up to a second if an issue. I am very interested in the performance using this config.

Any particular reason you went with SAS capable? Take a look at the BackBlaze box for a model.

- The drives I'm using (I've already bought them) are SATA 6.0 Gbps, so I originally intended to buy other hardware with support for that. Will that even matter, though, assuming that all traffic in & out of the system will be via gigabit ethernet (i.e., will the latter be my bottleneck anyway even with SATA 3.0 Gbps)?

Drive speed is not much of an issue I've found. Cache size and number of spindles is more important. More smaller drives offers better performance in a raid config.

- Will I have any issues creating a RAID-Z2 setup with 8 drives, then adding 2 more identical drives in a few months? Or am I better off just buying 2 more now to start with a 10-drive setup from the get-go?

I don't have alot of experience with RAID-Z2, but everything I read tells me that growing the array is one of the primary advantages of ZFS.

Anyone with more experience want to field this?

- If I ultimately go with the HighPoint 2720SGL and need/want the 6.0 Gbps capability, do I need special cables? Will some cheap cables from MonoPrice do the trick (e.g., http://www.monoprice.com/products/p...=10254&cs_id=1025406&p_id=8186&seq=1&format=2) or do I need to get ones built to a certain standard? Unlike with SATA cables, I don't see any SFF-8087 breakout cables specified as SATA II/3.0 or SATA III/6.0.

The advantage of the card you are looking at is the cabling. It is definitely cleaner, but at a cost... Going with port multiplier might be less expensive. I don't think cable quality is a major issue.

- Any recommendations on USB thumb drives for the boot volume (brand/size/model/USB 2.0 or 3.0), or does it matter that much?

I prefer an SATA SSD for the system drive, some folks have had issues with USB system drives... It'll will make your life easier, also swapping is an issue if you are going to be running a VM Config. You'll be running both windows and FreeNAS and your hypervisor all from a USB Stick?

If you go USB, make sure it is readyboost capable (good indicator of performance).

- Anything else I missed, or gotchas to be aware of? Would anyone recommend different hardware or a different approach?

Have you looked at cases? Most cases that handle 10+ drives have heavy cooling, the fan noise as a HTPC might be an issue. Also do you want hot swap?

Hope that helps, if you can point in a direction I think folks here can help further.

 

[Hydaral]

Once you run ZFS inside a VM you will lose ALL its advantages and features, as the hypervisor controls the disks.

You can ADD to a zpool containing a RAIDZ2 vdev, but you cannot EXPAND the vdev itself (this is actually the most requested feature for ZFS). If you initially create a zpool containing a single RAIDZ2 vdev with 6+2 then that vdev has to stay that way forever, if you want to increase the size of the zpool, you have to add another vdev, but that vdev can be any configuration, RAIDZ1, mirrored, etc.

Are those 3TB disks using 512e blocks? If so, you may have significant performance issues with them on ZFS.

I run NexentaStor off a 3 x USB drive mirror, I have had a few problems. The first drives I used, Sandisk Cruzer Blades, died in less than a month. I have had problems with Corsair Voyagers too, they lose their Fdisk table in about 3-6 months and only keep it for about 2 weeks to a month after fixing. I have an OCZ Rally2 that has been running perfectly for 10 months now and an Imation Nano for about 9 months. I just bought a Patriot Axle to replace the Voyager when it fails.

If you are going to use USB flash for the OS then just make sure it is redundant and that you have spare drives on hand. Oh, and make sure you do not use multiples of the same drive.

 

[Mr. Bungle]

Thanks so much for the feedback! Sorry for the slow reply.

@GregN:

I appreciate it - that's very helpful info. You've convinced me to scrap the HTPC half and just focus on a pure NAS box. I'm also (partly based on other feedback, as well) going to give up on the Asus E25M1-M in favor of a Sandy Bridge-based platform. A few people have suggested that I might need the extra horsepower to run RAID-Z2, and not only does it only work out to a ~$50 premium or so, but it allows for an upgrade path and shouldn't consume much (if any) more power at idle due to Intel's SpeedStep technology.

I don't have a preference for SAS, but the majority of the HBAs that I was coming across on NewEgg (at least the ones within what I consider a reasonable price range) were SAS-based - at least the 4x and 8x port units. I've given that some more thought as well, and due to the entry price of an 8x SATA HBA (~$150 give or take, when you include cables and everything), I might as well just find a motherboard with more SATA ports and rely on one or two 2x SATA HBAs to get me to 10 ports if needed, like the popular HighPoint Rocket 620 ($25).

That BackBlaze article was really cool - thanks for the link. I enjoyed reading about that.

I've actually learned that you can't expand a RAID-Z2, at least not in the way I was hoping. You can add additional arrays to make a pool of multiple arrays (not sure if my terminology is correct), but the only way to grow the size of an array that's already set up is to replace every physical drive with a larger one (e.g., go from 8x 2 TB drives to 8x 3 TB drives). So, I'm planning to just make a RAID-Z2 array that's the largest I feel like I'll possibly need for the foreseeable future.

Thanks for the SATA SSD suggestion; unfortunately I might need to stick with USB since I won't likely have an extra SATA port to take advantage of. At least I'll only be running FreeNAS at this point, not trying to run an HTPC simultaneously. I'll do a bit more research to see what's working well for others, and I'm definitely going to follow a backup plan.

Thanks again! I'd love to hear your feedback on my new thread (link to follow soon).

@Hydaral:

Thanks for your input!

That's very good to know about the huge disadvantage of running ZFS in a VM - definitely a deal breaker. I'm glad to know that it's not just a vague question of performance or something - makes it a much simpler decision to avoid that.

The 3 TB drives that I have use a 512 KB sector size (see http://pccasegear.com/index.php?main_page=product_info&products_id=18061) - is that what you're referring to? If so, can you elaborate on the potential performance problems?

My thought on the USB stick for the OS was just to clone it after making any configuration changes. The NAS box will butt up against a wall, so I figure that just using a rear USB port will be secure enough (there's nowhere for it to go). Once this thing is running, I don't plan to make many (if any) periodic changes, so as long as I've made a clone, I figure I can just run to Fry's a grab a replacement USB stick, then move the clone over to that. Downtime is not really a concern. Does that sound reasonable?

Since my goals have changed so much, I'm just going to start a separate thread about my updated build. I'll post a link here as soon as that's up.

 

[Mr. Bungle]

Here's the new thread:

http://forums.smallnetbuilder.com/showthread.php?p=32324#post32324