What's new

NAS that handles huge numbers of small files

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

McKulty

New Around Here
I have a homebrew document storage system in my office centered on a Buffalo LS421 using Raid 1 with two drives. The files are 20-80K tif files, monochrome scans of paper documents.

The master library is maintained on an SSD with ~160,000 files in one folder, subdivided and synced to three folders on the LS421, folders of approx 50,000 files in each folder.


I'm surprised how long the LS421 takes to add new files to the distribution folders. The cleanest and most efficient copy process has been the DOS COPY command. COPY requires about 300 msec for each file, ~3 files per second. This is not a problem when I try to update small numbers of files (adding each days new scans).

But it's a monster of a job if I try to duplicate the large folder in any fashion in the NAS. The first few hundred files go quickly, then it bogs down progressively to even slower than 3/sec, such that I come back after leaving a busy DOS window overnight and it's like 1.5-2 seconds per file.

I realize that the NTFS file system takes time to do necessary housekeeping on each new file. But I've reached a point where I'd like to upgrade the NAS, or my technique, in some fashion that makes this a less laborious task.

Should I abandon NAS altogether and set up a file server with 2 SSDs? Is there a NAS that stores large numbers of files without bogging down this way? I'm working on breaking the distribution set into 10 folders instead of 3 but I'd like to know if I could bet better performance from the hardware first.

What slows down the LS421? Is it the fact that I'm using RAID1? Would it be better if I replaced the RAID drives with SSDs?

TIA for any help.
 
Having an external computer move files around on a NAS incurs network overhead.

You need a NAS with a built-in file manager, so the NAS can locally execute the copy. Synology, QNAP and NETGEAR NASes also have built-in or add-on modules for folder syncing.
 
Thanks I see that could be 10-100 times faster if that's all I'm doing.

Edit: Shopping for that it's hard to tell which ones have such features. I'm guessing it's in the $500+ range, not $250?
 
Thanks. But if I'm getting ~95 Mbps with large files, that's about the best I can expect from gigabit ethernet, right? SSD would do no better?
 

Similar threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top