I have a homebrew document storage system in my office centered on a Buffalo LS421 using Raid 1 with two drives. The files are 20-80K tif files, monochrome scans of paper documents.
The master library is maintained on an SSD with ~160,000 files in one folder, subdivided and synced to three folders on the LS421, folders of approx 50,000 files in each folder.
I'm surprised how long the LS421 takes to add new files to the distribution folders. The cleanest and most efficient copy process has been the DOS COPY command. COPY requires about 300 msec for each file, ~3 files per second. This is not a problem when I try to update small numbers of files (adding each days new scans).
But it's a monster of a job if I try to duplicate the large folder in any fashion in the NAS. The first few hundred files go quickly, then it bogs down progressively to even slower than 3/sec, such that I come back after leaving a busy DOS window overnight and it's like 1.5-2 seconds per file.
I realize that the NTFS file system takes time to do necessary housekeeping on each new file. But I've reached a point where I'd like to upgrade the NAS, or my technique, in some fashion that makes this a less laborious task.
Should I abandon NAS altogether and set up a file server with 2 SSDs? Is there a NAS that stores large numbers of files without bogging down this way? I'm working on breaking the distribution set into 10 folders instead of 3 but I'd like to know if I could bet better performance from the hardware first.
What slows down the LS421? Is it the fact that I'm using RAID1? Would it be better if I replaced the RAID drives with SSDs?
TIA for any help.
The master library is maintained on an SSD with ~160,000 files in one folder, subdivided and synced to three folders on the LS421, folders of approx 50,000 files in each folder.
I'm surprised how long the LS421 takes to add new files to the distribution folders. The cleanest and most efficient copy process has been the DOS COPY command. COPY requires about 300 msec for each file, ~3 files per second. This is not a problem when I try to update small numbers of files (adding each days new scans).
But it's a monster of a job if I try to duplicate the large folder in any fashion in the NAS. The first few hundred files go quickly, then it bogs down progressively to even slower than 3/sec, such that I come back after leaving a busy DOS window overnight and it's like 1.5-2 seconds per file.
I realize that the NTFS file system takes time to do necessary housekeeping on each new file. But I've reached a point where I'd like to upgrade the NAS, or my technique, in some fashion that makes this a less laborious task.
Should I abandon NAS altogether and set up a file server with 2 SSDs? Is there a NAS that stores large numbers of files without bogging down this way? I'm working on breaking the distribution set into 10 folders instead of 3 but I'd like to know if I could bet better performance from the hardware first.
What slows down the LS421? Is it the fact that I'm using RAID1? Would it be better if I replaced the RAID drives with SSDs?
TIA for any help.