My DIY SFF ZFS NAS build

[Hydaral]

I have been reading these forums for a couple of years now, so since I have just built my first NAS I thought I would contribute something back. This is also my first ever case mod.

I have been using a DLink DNS-323 for the last few years, but a recent HDD failure made me seriously consider a DIY NAS that I had complete control over. Only recently have I found the filesystem/OS that has all the features I wanted and is at a user-friendliness level I was comfortable with.

I wanted a FS that had the following:

• very easy to grow
• very fault tolerant
• easy to administer
• simple to transfer to another system

ZFS covers all these, and more. Unfortunately, until recently ZFS was only available in Solaris/OpenSolaris and hardware support was pretty restrictive, but there are now several OSes that implement ZFS and are suitable for NAS use, hardware support is also a little better in Solaris based OSes.

After testing FreeNAS, NexentaStor and EON on an ESX VM, I settled on NexentaStor (http://nexentastor.org/), the main reason being that it can have a ZFS root disk, this is something that FreeBSD (and FreeNAS) cannot do, I will be using a ZFS mirror for the OS. It also has a nice web UI. I could not get EON to work.

On to the hardware, all prices in AU$:

• Case: Aywun A1-1001 http://aywun.com/ProductDetails.asp?ID=71 $41
• Mainboard: GA-D525TUD http://www.gigabyte.com/products/product-page.aspx?pid=3549#ov $129
• Memory: Kingston ValueRAM 2GB DDR3 http://www.valueram.com/desktop/memory.asp $55

I had to check the Gigabyte supported memory list: http://download.gigabyte.ru/memory/mb_memory_ga-d525tud.pdf

Other stuff:

• Scythe 4x Hard Drive Stabiliser: http://scythe-usa.com/product/acc/042/scyhdsx4-detail.html $18
• OCZ Rally2 4GB USB flash drive: On hand
• Gigabyte 120mm case fan: On hand
• Wire mesh 120mm fan grille: $6
• Storage HDDs: On hand

Total cost without disks: $249 (cheaper than my diskless DNS-323)

This is what the original case looks like.



It has this plastic window on the side with slits for airflow.



Inside, it has a 5.25 inch bay and a 3.5 inch cage underneath

 

[Hydaral]

The case does come with a 200W PSU, but it is not in these photos because this is actually my second one of these cases. I blew up the first PSU, so I had to buy another case, details later.

First of all I had to find a way to get 3 x 3.5 inch HDDs in the case, this is where the Scythe HDD stabilizer comes in. It can be mounted to an existing 5.25 inch bay from the top and hang down to hold up to four HDDs, although getting a fourth one in may be a tight squeeze due to the cabling.

I had to drill out the rivets holding the 5.25 inch bay in and cut the bottom out of it, then once the two sides were riveted back in I could get the HDD holder in there. It is only holding one disk ATM but I can get two more in above it.



I also wanted extra cooling, there are going to be 3 HDDs in here so it might get a little warm in this small case. The plastic window with the slits was the perfect place, so I drilled some fan mounting holes (brittle plastic, I cracked it a little while drilling) and dremeled a hole for the fan.



I'm using an aluminium and steel mesh grille, I haven't used one of these before so I don't know how well it will keep the dust out.

 

[Hydaral]

Once I mounted the fan and tried to put it on the case, I found a problem I had not foreseen. this is not a slimline fan and it intrudes 25mm into the case.



As you can see from the original case photos, the longitudinal rails are very close to the side of the case, there is only a 10mm gap, I had to find 15mm somewhere.

I found that there was just enough room to move the left side rail by 15mm, fixing it to the back of the case would not be a problem.



At the front of the case however, there is a hole for the drive bay where I needed to attach the rail. I cut a small piece from the discarded bottom of the 5.25 inch bay and riveted it in place to hold the rail.



This would also mean the 5.25 inch drive bay would be narrower and would not fit the Scythe HDD holder, so I had to flip the right side 5.25 inch bay side around to the other side of its rail.

 

[Hydaral]

This removed some of the case's rigidity as the bay side was riveted to the case front, so I bent a small piece of metal and attached it to the front of the case and the bay side. No photo for this, as it's in a tight spot, you can just see the front rivet and the end of the piece of metal at the far right of the above photo. Close up below.



Now the HDD holder fits and there is just enough room for the fan.



As I have seen others mention about this case, the PSU fan is louder than desirable, I decided to use the series diode method.

For those that haven't heard of this, diodes only allow current to pass in one direction, this particular trait isn't what we are interested in though, as a side effect of this property, the diode causes a voltage drop of around 0.75 volts (mine measured 0.77 V). Connecting a bunch of these in series reduces the voltage to the fan but does not decrease the current like resistors do, so the fan will not stall on startup.

I created small circuit boards for both the PSU fan and the case fan (once I slowed the PSU fan, the case fan seemed a lot louder ). I originally opened up the PSU to see if I could connect the 12V fan wire to 5V, but after closing the PSU up after deciding not to do it this way and then powering it back up, there was a large pop and flash of light, signaling the violent death of the PSU. Since I couldn't find anywhere close that sold a separate, not very expensive mATX PSU, I just ordered another case.

This is the board for the PSU fan.

 

[Hydaral]

This is the one for the case fan.



Connected up.



The boards have jumpers on them so I can choose how many diodes I want to use, the PSU fan is using 8, the case fan is using 4, they are both heatshrinked for electrical safety. The silkscreen on the second board is messed up because I was trying out the PulsarProFX PCB printing kit. It's the first board I made using it.

After these fan mods it is barely audible from 1 metre away in a quiet room. Even slowing the fans down, I still have positive pressure in the case, I always prefer this as it allows me to control where the air enters the case and to filter the dust out.

Externally, it looks pretty much the same as it did before, just with a mesh fan grille.



I have no idea what its power draw is, I don't have a Kill-a-Watt, I expect that it's about the same as any other Intel Atom D525 based machine + 3 HDD.

That's the hardware done, now onto the software.

As mentioned before, I am using NexentaStor v3.0.4, to install this I connected up an IDE CD drive and ran a bootable CD. I installed onto a 4GB OCZ Rally2 USB flash drive I will later get a second flash drive and mirror the system drive.

I will be running a 3 disk ZFS mirror, the Gigabyte board has 4 SATA ports but when I was trying to test using a spare disk I had, NexentaStor would not detect it. I found that I had to change the HDD driver in the BIOS to AHCI, NexentaStor found the disk fine after that.

As I will be mirroring disks and not using RAIDZ1 or 2 and the NAS is for a small home network, I didn't think I would need the 4GB of RAM recommended, so I am only using 2GB ATM. I can always add more if I need to.

Once I had the test HDD installed and running, I did some network speed tests using dd from a Windows XP box.

Read: ~17MB/s
Write: ~29MB/s

Yeah, I know it doesn't look correct. That's what I have written on my notes though. I'll have to do some more tests once I get my mirror disks in. Although it's significantly faster than my current DLink DNS-323 (~10MB/s max for both read and write).


Problems:

It looks like the Solaris kernel doesn't have very good ACPI support, my machine will shutdown, but it won't poweroff from the command line. Shouldn't be too much of a problem for a NAS though, my current one has only ever been shutdown once when I had to replace the faulty HDD. The workaround is to push the power button once, this will tell the machine to shutdown, then once the flash drive activity light has gone off (I can see it through the mesh at the front or the side window) I push the power button again and it powers off.

As mentioned before, NexentaStor only detects the HDDs when they are set to AHCI in the BIOS.

After the final assembly, the machine would not power on, I thought I had killed another PSU, but after checking that it was fine by grounding the green wire, I found that the front power button was not completing the circuit to trigger power to the mainboard. It turns out that the solder connection to the front switch is very fragile and I had broken it after taking the front fascia off so many times. I soldered it back and wrapped some self-fusing tape around it and it was fine.

 

[Kristijan]

Hi Hydaral,

Many thanks for this post, it's saved a lot of digging around.

I'm in the process of doing the exact same thing. I've picked out all my parts, and the motherboard is exactly the model that you used in yours.

Did you have any driver issues with NexentaStor?

My plan was to use either Solaris 10 9/10, or failing that, NexentaStor. I know it's a fork of the old OpenSolaris project, which was based on Solaris.

Many thanks,
-Kristijan

 

[Hydaral]

This was my concern when I was choosing my hardware, I couldn't find anyone using a Gigabyte board in the same family, let alone the same board, but as it is quite new, I guess that was to be expected.

In my research I found several people using the Intel D510MO boards with Solaris based OSes without problems, but they only have 2 SATA ports, I wanted at least 3. I assumed that since Solaris supports the D510 with the NM10 chipset, it *should* support the Gigabyte D525 board with NM10, it does.

The Gigabyte board does use the Gigabyte GSATA interface for two of the SATA ports, my test drive is connected to one of these. As mentioned in my earlier posts, there was an issue with drive detection but once I changed to AHCI it worked fine. I don't know if the detection problem was due to the GSATA ports or just the board in general.

The other concern I had was with the Realtek RTL8111E network chip, I had read that this was very similar to the RTL8168. I found a forum post that indicated that someone had got this chip working with Solaris after a bit of work:

http://forums.sun.com/thread.jspa?threadID=5103112

This post was back in 2006, so again, I took a leap of faith and hoped that the RTL8111 would be supported, if not, I was prepared to do a bit of hacking to get it to work. Luckily it worked out-of-the-box.

The USB works fine, I'm booting from a USB flash drive so no problems there.

The only other possible problem is the audio, but as I'm using this for a NAS I don't care about the audio. I'm not even sure how to test it from the command line in Solaris, if you do, let me know and I'll give it a go and let you know the results.

So, all in all it worked out quite well, as far as I know I am the first one to use this board with a Solaris based OS and post about it. That was my main reason for starting this thread, so others don't have to just assume that things will work.

I would think that if NexentaStor worked for me, then Solaris or OpenSolaris, from which it is based, would also work for you.

 

[cairo]

Hi Hydaral,
I'm just installing Nexenta 3.0.4 RC6 on the same mobo (GA-D525TUD), so it's nice to have someone to follow!
The point is that, in the second installation screen, the installer said it cannot found a network adapter...
I decided to install nexenta anyway, but in your post you said that it worked out-of-the-box (regarding the Realtek nic)
Did you use some specific driver, or make some post-install configuration?
Thanks
Cairo

 

[Hydaral]

Hmmm, strange. The version I am using was the "NexentaStor-Community-3.0.3-1.iso.zip" file, they call this the "3.0.4 Beta1".

I used the Community Edition, I presume you are using this too, although I wouldn't think it's any different to the commercial one?

Have you checked the BIOS, is the adapter turned off? Do you know if you have a new mainboard revision?

Come to think of it, I do recall some warning or error coming up on install, I think it may have been about the NIC, but after the install and reboot everything worked fine.

 

[Hydaral]

I had some flash drive issues just this last week so I thought I'd post an update.

I originally purchased two Sandisk Cruzer Blades to run the mirrored root pool on:

http://www.sandisk.com/products/computing-products/sandisk-cruzer-blade-usb-flash-drive

Just short of a month of using them, I get 97 write errors on one of them, so I detach it from the mirror and try to fdisk and format, nothing. So I attach a spare 4GB flash drive I have laying around so it gets redundancy back.

Just in time. The very next day, the other Cruzer fails with 147 write errors, same deal, no-go on the fdisk and format, neither of them format in WinXP either.

I took them back to the store and exchanged them for two different other brands, but of course after getting them home and trying to attach them to the mirror, I find out they are about 20 cylinders too small.

So back I go to the store to get an 8GB drive, just to be sure, an Imation Nano, 4x the price of the 4GB, but at least my data is safe. I now also have two 8GB Corsair Flash Voyagers on the way, just in case.

The moral of the story is: never use identical brand and model drives in a RAID. I know what you are thinking "but you just said you bought two identical Corsair drives". I do not intend to use them concurrently, I will replace one of the existing flash drives I am using now and keep the other as a spare. If they are ever running concurrently then their in-use times will at least be staggered and only for a short period of time while I procure replacement drives.

 

[cairo]

Hmmm, strange. The version I am using was the "NexentaStor-Community-3.0.3-1.iso.zip" file, they call this the "3.0.4 Beta1".

I used the Community Edition, I presume you are using this too, although I wouldn't think it's any different to the commercial one?

[...]

Come to think of it, I do recall some warning or error coming up on install, I think it may have been about the NIC, but after the install and reboot everything worked fine.

Actually, you are right!
I Used the Community Edition too, the 3.0.4RC4; during the install, at the beginning, the installer warn me that it doesn't found any NIC.
But, choosing to install the system anyway, at the end the NIC works perfectly; I noticed that it uses the gani driver (not the rge... probably the bug you mention in opensolaris was solved months ago), without any disconnetions or other problems.
So, the NIC is a non-issue!
Thanks
Davide

 

[00Roush]

I had some flash drive issues just this last week so I thought I'd post an update.

I originally purchased two Sandisk Cruzer Blades to run the mirrored root pool on:

http://www.sandisk.com/products/computing-products/sandisk-cruzer-blade-usb-flash-drive

Just short of a month of using them, I get 97 write errors on one of them, so I detach it from the mirror and try to fdisk and format, nothing. So I attach a spare 4GB flash drive I have laying around so it gets redundancy back.

Just in time. The very next day, the other Cruzer fails with 147 write errors, same deal, no-go on the fdisk and format, neither of them format in WinXP either.

I took them back to the store and exchanged them for two different other brands, but of course after getting them home and trying to attach them to the mirror, I find out they are about 20 cylinders too small.

So back I go to the store to get an 8GB drive, just to be sure, an Imation Nano, 4x the price of the 4GB, but at least my data is safe. I now also have two 8GB Corsair Flash Voyagers on the way, just in case.

The moral of the story is: never use identical brand and model drives in a RAID. I know what you are thinking "but you just said you bought two identical Corsair drives". I do not intend to use them concurrently, I will replace one of the existing flash drives I am using now and keep the other as a spare. If they are ever running concurrently then their in-use times will at least be staggered and only for a short period of time while I procure replacement drives.

That is a bit weird that your some of your flash drives are having errors in such a short time... it could be due to flash memory in general. Same basic problem as SDD drives have. Flash memory can only be written to a limited amount of times. I wonder if the OS is writing to the flash drive a lot?

BTW great write up on the case mods and build! I actually gave NexentaStor a run through last week and was quite surprised how good the web interface was. Definitely an OS more people need to check out.

00Roush

 

[Hydaral]

I was fully aware of the write-cycle limitation of flash drives when I was designing my NAS and was prepared to replace the drives once, maybe twice per year. They are cheap enough to do this. But less than a month?

Feel free to correct any assumptions or calculations following.

I know that it was writing to the drive quite a lot, the replacement drive, the Imation Nano, has an activity light. Watching it flash I guesstimate, at worst, the drive is 40% busy, I'll assume that 2/3 of that are writes. It's updating log files I guess, so random writes.

So doing some hand-wavy calculations, with the drive having an estimated random write speed of 2 MB/s for small files, the OS is tring to write an average of 0.528 MB/s, let's round that up to 1 MB/s for ease of calculation and errors in estimation.

30 days is 2,592,000 seconds, assuming the drive is being written to 24/7, that's 2,592,000 MB. The drive had about 1 GB free space, so assuming perfect wear leveling, that is 2,592 complete erase/write cycles. I know that erase/writes are sector based, but I really don't know how to work that out accurately.

From everything I have read, modern flash drives are pretty much guaranteed for 100,000 writes, normally you can expect over 1,000,000. I know the Cruzer blade was a cheap flash drive, but this was a Sandisk drive, not some unknown Chinese brand, so I don't think it's too much to ask for it to reach at least 10,000 writes.

I found this document on Sandisk wear leveling and calculation of drive lifetime from 2003, but couldn't decipher their ambiguous equation. Even their worst example says the flash memory will last 80 years though.

http://sandisk.com/Assets/File/OEM/WhitePapersAndBrochures/RS-MMC/WPaperWearLevelv1.0.pdf

I have marked these failures up to a bad batch of drives, I purchased them from the same store at the same time, off the same rack, so they were likely manufactured at exactly the same time.

 

[Hydaral]

I just found this site that seems to have a decent equation for calculating flash endurance:

http://www.edn.com/article/466503-Accurately_judging_endurance_for_solid_state_storage.php

Here it is with some modification by me:

All sizes in MB, speeds in MB/s, percentages as fraction of 1, eg 80% is 0.80.

Years = ((total capacity-used capacity)*maximum write cycles*estimation confidence)/(((estimated write speed*60)*write busy percent)*minutes in year)

So for my previous post:

years=((4000-3000)*10000*0.8)/(((2*60)*(0.4*.66))*525600)

years=(1000*10000*0.8)/(120*0.264*525600)

years=8000000/16651008

years=0.48

So even with 10000 write cycles, I should still be getting about 6 months.

 

[00Roush]

LOL I was just speculating... you went and figured it all out.

Kinda wondered what the life span of a USB flash drive would be.

Well a quick search on google and I found this... http://docs.sun.com/source/820-4893-13/compact_flash.html Looks like there are ways to minimize the writing that the OS does.

When I saw your performance I wondered what your bottleneck might be. It seems a bit low to me. I was going to ask if you have used dd to test the local speed of your zpool.

00Roush