Improving small file transfer speed?

[vondiesel]

I have an AC86U with an external drive attached. It's an old 1 TB 5400rpm laptop drive in a USB3 enclosure and it's attached to the USB3 port on the router. I use it for simple storage, file transfers, and occasional media streaming. I mostly interact with it using my Windows laptops (using Explorer or FTP) or Android phones (FTP). This setup fits my simple needs very well, and I've been very happy with the AC86U. Of particular note, large file transfers are very fast, so I've been really happy about that.

But as always, transferring a large number of small files is much slower. I know that I'll never get small file speeds as high as large ones, but are there any optimizations that can be made to improve small file transfer speed? For example, should I try ext4 instead of NTFS?

Thanks for any suggestions!

 

[K-2SO]

I mostly interact with it using my Windows laptops

should I try ext4 instead of NTFS?

Sure you want this?
- Health Scanner in router's GUI will stop working
- Drive becomes incompatible with Windows laptops

 

[vondiesel]

Sure you want this?
- Health Scanner in router's GUI will stop working
- Drive becomes incompatible with Windows laptops

Thanks for the notes! I didn't know things like that. That's why I'm asking.

Other than a different file system, is there really anything else I can do to optimize my current setup?

 

[K-2SO]

is there really anything else I can do to optimize my current setup?

Not sure what advice to give you. For me routers do routing. NAS do file transfers and storage. With your setup one or the other suffers from sharing the limited resources. You can't get much better this way.

 

[vondiesel]

OK. That's about what I expected, but I thought it would be good to ask people with more expertise.

I certainly recognize the limitations of my current hardware. I've thought about getting a different drive or a dedicated NAS, but given my minimal usage I can't justify the cost. If anyone else has any suggestions, then I'm happy to try them. Otherwise, I appreciate that I'm probably at the limit of what my hardware can do. Which for my limited needs is still very good.

 

[K-2SO]

I certainly recognize the limitations of my current hardware.

My only other suggestion is to avoid in general USB powered hard drives attached to routers.

 

[dosborne]

Is it an option for you to create an archive (tarball for instance) of the large number of small files so that you end up transfer one large file instead? I usually do that for creating images of android an Linux systems.

 

[vondiesel]

My only other suggestion is to avoid in general USB powered hard drives attached to routers.

Thanks for the tip. I'll look into ways of supplying power externally.

 

[vondiesel]

Is it an option for you to create an archive (tarball for instance) of the large number of small files so that you end up transfer one large file instead? I usually do that for creating images of android an Linux systems.

Yes, I was considering that. I will look into it further.

 

[vondiesel]

Is it an option for you to create an archive (tarball for instance) of the large number of small files so that you end up transfer one large file instead? I usually do that for creating images of android an Linux systems.

Thanks again for that tip. That's what I decided to do, and it helps save quite a bit of time on transfers.

 

[bbunge]

With my old AC66U_B1 I used an external USB 2 enclosure with a WD green 1 TB drive. It worked OK but I had a thumbdrive in the other slot with a 512 meg swap file. I set the USB 3 port to USB 2 mode. Asus recommended setting the port to USB2 mode to avoid interference with wifi. If you use the stock Asus firmware you can still set up a swap file. I did a how to several years ago that should work.

 

[vondiesel]

With my old AC66U_B1 I used an external USB 2 enclosure with a WD green 1 TB drive. It worked OK but I had a thumbdrive in the other slot with a 512 meg swap file. I set the USB 3 port to USB 2 mode. Asus recommended setting the port to USB2 mode to avoid interference with wifi. If you use the stock Asus firmware you can still set up a swap file. I did a how to several years ago that should work.

Interesting. How does that help small file transfer speed?

 

[bbunge]

Interesting. How does that help small file transfer speed?

Reads and writes to NTFS needs plenty of memory. Often more than a router has. May not help with your small file transfer but worth a try. For a cheap NAS you can use a Pi 3B+with the USB drive. Works better than a router.

 

[vondiesel]

Reads and writes to NTFS needs plenty of memory. Often more than a router has. May not help with your small file transfer but worth a try. For a cheap NAS you can use a Pi 3B+with the USB drive. Works better than a router.

Sounds worth a try. If I can get some time I'll try adding a flash swap. As for USB mode, I almost never use the 2.4G band, so interference in USB3 mode shouldn't be a problem.

 

[calvin tran]

How many files you need to transfer? and do you need to save them back after updating since you are sharing them among laptops? As suggested earlier, you might want to zip them up.
The problem with transferring small files is not the bandwidth issue but overhead for staring a new transfer for each and writing a new file at the dest.
If you don't want a NAS , you might store them on Google drive so you can access anywhere from any device; just a suggestion and may not work for your case.

 

[vondiesel]

How many files you need to transfer? and do you need to save them back after updating since you are sharing them among laptops? As suggested earlier, you might want to zip them up.
The problem with transferring small files is not the bandwidth issue but overhead for staring a new transfer for each and writing a new file at the dest.
If you don't want a NAS , you might store them on Google drive so you can access anywhere from any device; just a suggestion and may not work for your case.

Thanks. About 5 posts above I note that I've started making zip archives. It definitely helps.

Google drive and other cloud solutions are possible, but I'd have to purchase space since I have a lot of data. I don't need all the data synced to every device all the time. The local and network drives I have now take care of my needs for secure storage in multiple sites with infrequent access. And now that I'm zipping the small stuff, transfer speed is fine all around.