Linux File Server (NAS)

[Sheepdisease]

Could someone please suggest the most suitable hardware for this project? Let me go over the functionality of this so that it is clear.

Power, reliability and efficiency are the most important, in terms of working optimally for file transfer, encoding and decoding and conversion whilst remembering that the file server will be on 24/7.

I want to be able to access the server remotely and I know that some linux distros can do this from installation, whilst others can have this functionality added. I would like a version of linux I can use to text network security and learn how to use it properly. I suppose this would be best on a command-line installation, perhaps the option of both would be better.

I'd like to use a motherboard which can be upgraded in the future to accept more powerful hardware, but only want to spend money on the most necessary hardware for the specific purpose of a file sharer (RAM and Processor?).

Is there a PSU/USP hybrid out there to ensure that nothing is compromised by a powercut?

Lastly, I want the largest amount of storage space for the least amount of money.

The plan is that all files will be saved to the server initially, but I would like copies automatically saved to the computer in use as well.

I need ideas on how to ensure that I have a backup of all the files on the server, without using external drives or relying on separation of HDD space using partitions. I plan on using RAID 5 to facilitate as much space as possible. I've already got 4 x 1TB HDDs, 2 x 500GB HDDs.

 

[PaulC]

There are so many choices, and trade-offs, it becomes a very personal set of choices. There really isn't a 'best' choice, and you will probably find that your needs change over time, and so what's 'best' now may not be in six months.

You might want to take a look at a Drobo, connected to something small/power efficient running Linux.

I'd suggest that you might want to reverse the backup strategy - work on files on your desktop, and use real-time backup software to have them copied to your NAS in the background. I use Memeo on my Windows PCs, but there are several other choices.

An external UPS is usually the way to go for power failure protection, just make sure it's one you can monitor from the server, so it can shut down cleanly, automatically. Take a look at the 'nut' project. I use a MGE Pulsar personally, but they're not cheap.

Largest space for the least money means denser drives. The Seagate 1.5TB models are the cheapest per-GB at the moment (~$126 last time I looked).

How 'remote' are you planning to be when accessing this box? If you're on the same network, it's trivial (use OpenSSH). If you're the other side of a firewall, take a look at OpenVPN, that's working well for me (and many others). If you'd like a graphical desktop over the network, I've had good experiences with NXServer.

For distributions, you have several choices. My personal favorite is Fedora, but Ubuntu is probably more popular at this point. There's also Centos, often used for 'mission critical' machines in the enterprise.

Anyhow, just some thoughts. Hope it helps,

Paul

 

[Sheepdisease]

There are so many choices, and trade-offs, it becomes a very personal set of choices. There really isn't a 'best' choice, and you will probably find that your needs change over time, and so what's 'best' now may not be in six months.

You might want to take a look at a Drobo, connected to something small/power efficient running Linux.

I'd suggest that you might want to reverse the backup strategy - work on files on your desktop, and use real-time backup software to have them copied to your NAS in the background. I use Memeo on my Windows PCs, but there are several other choices.

An external UPS is usually the way to go for power failure protection, just make sure it's one you can monitor from the server, so it can shut down cleanly, automatically. Take a look at the 'nut' project. I use a MGE Pulsar personally, but they're not cheap.

Largest space for the least money means denser drives. The Seagate 1.5TB models are the cheapest per-GB at the moment (~$126 last time I looked).

How 'remote' are you planning to be when accessing this box? If you're on the same network, it's trivial (use OpenSSH). If you're the other side of a firewall, take a look at OpenVPN, that's working well for me (and many others). If you'd like a graphical desktop over the network, I've had good experiences with NXServer.

For distributions, you have several choices. My personal favorite is Fedora, but Ubuntu is probably more popular at this point. There's also Centos, often used for 'mission critical' machines in the enterprise.

Anyhow, just some thoughts. Hope it helps,

Paul

Hello Paul, first and foremost I would like to thank you for taking to time to write such a lengthy reply and (seemingly) registering to answer my question.

I realise that needs change over time, but that's what upgradable hardware is for. I'd like to say that I'm looking to build this myself for the experience and would like the opportunity to build something operating linux to learn network security.

This system will be operated fairly remotely. It is going to sit in the loft, so even if the laptop and PC are stollen, we still have a copy of all of our precious files (documents, photos and music). So it will be wired to the PC using cat5e cable and accessed wirelessly on the laptop.

I think that it may be a good idea to do what you say regarding saving locally first, although the idea is that everything is accessible from the file server (because the laptop for instance has a small storage capacity). What are your thoughts?

I've checked out the drobo and really it's just an external set of hard drives, I've got 4 x 1TB HDDs and 2 x 500GB HDDs already. I plan on using RAID 5.

I'm open to ideas regarding backup, further to what has been discussed by us so far.

I know some linux distributions offer remote access straight from installation and others support this function with the installation of an application. I'm not fussed about GUI, command line suits me and is less resource intensive (although a hybrid would be preferable I'm sure).

 

[PaulC]

Hello Paul, first and foremost I would like to thank you for taking to time to write such a lengthy reply and (seemingly) registering to answer my question.

Glad to help. And yes, I registered to answer it I was in your position several years ago, and am coming up to the point where it needs to be overhauled again.

I realise that needs change over time, but that's what upgradable hardware is for. I'd like to say that I'm looking to build this myself for the experience and would like the opportunity to build something operating linux to learn network security.

Understood, though with the caveat that upgrades may cost you as much as adding another server appliance from the likes of Synology, QNAP or Theacus. Though I built my own too, so don't let me dissuade you

This system will be operated fairly remotely. It is going to sit in the loft, so even if the laptop and PC are stollen, we still have a copy of all of our precious files (documents, photos and music). So it will be wired to the PC using cat5e cable and accessed wirelessly on the laptop.

That's not 'remote' in networking terms, it's still on your local network.

What climate do you live in? a loft/attic can get both very hot and very cold, beyond the operating range most equipment are specified for. The temperature swings will 'age' hardware faster and almost certainly cause it to fail prematurely. Might want to think of somewhere else that's more stable for temperature (and perhaps humidity?).

I think that it may be a good idea to do what you say regarding saving locally first, although the idea is that everything is accessible from the file server (because the laptop for instance has a small storage capacity). What are your thoughts?

The performance of file access over the network will never be as good as to a local drive. You'll be more productive if the files you're working on are local. If there's continuous automatic backups happening in the background, you'l get the best of both worlds.

I'd also make a distinction between backup and archiving. Since your desktop/laptop has less storage, at some point there will be files on the server that are no longer on any of your desktop/laptops. A failure on the server now means data loss. So you'd either need to make offline copies (e.g. on DVD/blu-ray) or have a second copy somewhere that wouldn't also be taken out by hardware failure on the server. More on this below.

I've checked out the drobo and really it's just an external set of hard drives,

It's a lot more sophisticated than that. It gives you automatically-managed redundancy against any single drive failure, supports a set of mis-matched drives, along with automatic data migration when you swap any drive for a larger one.

I've got 4 x 1TB HDDs and 2 x 500GB HDDs already. I plan on using RAID 5.

I've had a RAID-5 setup at home for many years now.

One common mis-conception that needs to be pointed out up front - RAID-5 is not a complete solution to protect your data. It only protects you against a single hard drive failure, not other failures (e.g. file system corruption).

Given the firmware problems like the recent one Seagate has had, there's also the danger of multiple drives failing due to a design or common manufacturing defect. Particularly when they've been purchased at the same time as a 'set'.

I'm running two, four-disk RAID-5 arrays. One is the primary, the second purely for backups. The second isn't even mounted unless a backup is in progress, to help shield it from software/filesystem failures.

Even with that setup, the irreplaceable stuff is also backed up to an entirely separate NAS over the network. This is a recent change, and I may retire the second array once I'm happy it's working well. I've also toyed with the idea of off-site backup (e.g. Amazon S3) but the costs add up for large data sets. Earthquakes are a danger where I live, so I really need a copy of important stuff 'somewhere else'. Up 'til now, that's been a pair of removable drives that I alternately take to work.

I've found upgrading RAID-5 arrays can be cost-prohibitive. I have an 8-port IDE 3ware card. I've already upgraded the array once, and it's close to full again. In the meantime, very large drives have become dirt cheap, and there's been a transition to SATA. Frankly it'd be cheaper, and (practically speaking) just as reliable, to dismantle my entire RAID-5 hardware subsystem, and install three or four 1TB+ drives instead. I could even do some of the clever things that Drobo does (albeit manually) using the logical volume manager (LVM) and software RAID (md).

I know some linux distributions offer remote access straight from installation and others support this function with the installation of an application. I'm not fussed about GUI, command line suits me and is less resource intensive (although a hybrid would be preferable I'm sure).

You won't have a problem with accessing linux boxes over the network, it's integral to linux DNA. The OpenSSH package will give you command-line access, VNC or NXServer will give you GUI access. The mainstream distros make it very easy to add such packages, if they're not there already. I wouldn't worry about that aspect - you'll need a monitor/keyboard/mouse hooked up to install anyway.

-- Paul

 

[Sheepdisease]

I don't have time to respond to every single comment you made, although I had the sourtesy to read all of your valid points.

At the moment I would like to respond to your point regarding the loft. There must be some way to make the loft a good place to store the server, some way to shield it from the heat and protect it from the cold.

Surely something like a cardboard box surround would be good to keep the heat in during the winter, whilst a silver foil cover on the box would prevent heat from directly impacting on the server.