Raid 4 Drives

[coxhaus]

OK. So you have hot swap hardware. Yea I built a server 2008/R2 with hot swap at my house using a SuperMicro chassis with a hardware hot swap chassis. I used a RAID card with a gig of cache and battery backup. It was older technology when I bought it. I got a good deal. I still have it but it will not do 6 gig/s with the newer drives so I don't use it any more.

I was thinking software RAID running off a motherboard.

 

[coxhaus]

ZFS exposes you with RAM failure because it uses dynamic RAM. Read my links above.

 

[sfx2000]

ZFS has a fair amount of redundancy to it, and of course, there is the journal - helps that it was really intended for server grade HW, so ECC is a given with memory...

 

[sfx2000]

I was thinking software RAID running off a motherboard.

And that is implied in my example - as this is what the NAS vendors are doing...

 

[sfx2000]

With mdadm/lvm, it's really easy...

One has to go into mdadm, mark the failed drive, tell mdadm it's going to be removed, put the drive into standby, and then remove and replace, copy the partition map, and add the replaced drive back into the array, and start the rebuild.

One word of advice to others - in a RAID5/RAID10 array with 4 disks - once one has hit a place where one has failed, please consider replacing the rest of the drives - mean time between failures is a good number to go by, and in my experience, after about 3 years in active 24/7 use, one starts getting close to where things can go awry...

And it normally shows up on the rebuild, when a second disk will start reporting unrecoverable errors... and with RAID5, one is now looking at a restore from backups - RAID5 can tolerate one failed drive in a 4 disk array, it won't live with 2 failed drives.. RAID10 can be a bit better there...

 

[System Error Message]

if you use bad ram, you'll have issues using ZFS. my old 2TB WD Black already maxes out a 1Gb/s connection with its 200MB/s speeds. but they'll not exceed sata 3 spec which is 5Gb/s about 640MB/s, hard drives havent reached those speeds yet in a single drive, currently about 300MB/s on a fast performing drive.

@coxhaus in the past, ram was prone to failure which is why it was recommended to use ECC ram to run ZFS but for the past decade, as long as you avoid generic/value ram and stick with standard decent brand or better, you can run ZFS all day, all year. When you buy ram, run memtest on it for a few runs, if it passes, then great, if not return it and get another one. Infact one could get generic or value ram now and still have reliable ram, whats happened now is that manufacturers have turned up latency on those ram to avoid errors, thanks to not having to stick to JEDEC/SPD as one could have the timings and frequencies written onto a chip on the stick so the motherboard can just read it and use those settings.

 

[coxhaus]

The weakness of ZFS is using dynamic RAM. ZFS exposes you to corruption if you have a power failure where a cached RAID controller does not because they have an onboard battery on the controller card. I know you say just use an APC and you should be fine. But there is that small possibility of some kind of power failure with ZFS which does not exist with the RAID controller.

Read what I posted above it explains it better than me.

Yes the new hard drives are much faster and can move more data than the old ones. We are almost to the point where RAID is not needed any more. We now have large 12TB drives which can move 6Gb/s. This is what RAID was designed to do. The only thing missing is redundancy. Now days if I was building a RAID for a small shop or home I might consider not running RAID and doing cross backups for just the important folders. The RAID maintenance operations take so long to run using large drives that I am not sure it is worth it. RAID is putting a lot of wear and tear on your drives which would not be there if you did not run RAID. Think of the extra space you will have by not running RAID. The extra space can be as high as 50%.

 

[sfx2000]

@coxhaus in the past, ram was prone to failure which is why it was recommended to use ECC ram to run ZFS but for the past decade, as long as you avoid generic/value ram and stick with standard decent brand or better, you can run ZFS all day, all year

ZFS is especially vulnerable to RAM errors, as it assumes that the ARC is "safe", and random bit flips (cosmic rays, etc) in RAM happen often enough that ECC is pretty much a must. One data is committed to disk, everything is fine.

There is a flag to help there - which enables checksumming for the ARC cache, but it does come with a cost in performance.

 

[sfx2000]

Yes the new hard drives are much faster and can move more data than the old ones. We are almost to the point where RAID is not needed any more. We now have large 12TB drives which can move 6Gb/s. This is what RAID was designed to do. The only thing missing is redundancy.

RAID is in no danger of going away - mostly due to the redundancy you mentioned - many of my servers in my old role would have RAID1 boot/system drives to ensure high availability. Persistent data was managed outside with things like ZFS or NetApp clusters.

Looking at cloud scale - things have evolved in very interesting ways - check out OpenStack's Cinder Project, and Red Hat's ceph storage is also a good representation of what is going on in the cloud.

 

[coxhaus]

I run all ECC RAM in my servers. Even when I ran pfsense I ran ECC RAM. When I ran Untangle for years I ran ECC RAM. My home server is ECC RAM. It is a security blanket I like.

 

[sfx2000]

The weakness of ZFS is using dynamic RAM. ZFS exposes you to corruption if you have a power failure where a cached RAID controller does not because they have an onboard battery on the controller card. I know you say just use an APC and you should be fine. But there is that small possibility of some kind of power failure with ZFS which does not exist with the RAID controller.

Power is definitely a risk with ZFS - having a HBA with it's own battery helps - many ZFS deployments will use the HW RAID adapters as plain HBA, no RAID, as ZFS can do a better job there at the file system level.

 

[coxhaus]

ZFS is a multl-layer system and you do not need to run all the layers.