How to sync FTP files to USB stick on router?

  • ATTENTION! As of November 1, 2020, you are not able to reply to threads 6 months after the thread is opened if there are more than 500 posts in the thread.
    Threads will not be locked, so posts may still be edited by their authors.
    Just start a new thread on the topic to post if you get an error message when trying to reply to a thread.

ftpusb

New Around Here
Hihi,

I have been searching through the forums on how to download ftp files into my usb storage attached to the asus router but no success.
Purpose is to sync the files from a specified ftp folder to my usb attached to router every day, and delete the files on ftp server as storage is limited in ftp.

I am using Asus AC56U Merlin firmware 384.6
FTP server is in same home wifi network, I could access the ftp server and see the files on my pc and phone.

I have tried using AiCloud smart sync but it is not downloading any files at all, anyway to do it via scripts?
I am noob about script so have to direct and guide me on step by step to writing script with folder to sync and download, and delete files on ftp etc.
Thanks


1608345003597.png
 

Jeffrey Young

Senior Member
If you are not comfortable with script writing, and both the FTP site and the USB storage is setup as a network shares, why not just use your favorite client computer workstation to run a third party sync software?

Otherwise, you could set up a cron job on your router to run a script nightly to do the file transfer. You can use a variety of tools. dd, rsync, etc.


Start with this site for an overview of scripts that Merlin has defined;


Once you have your script written, you can schedule it by calling cru in services-start scripts or init-start script.

Maybe google "linux mount" for help in mounting your ftp location (if the router GUI has not already done so).
 
Last edited:

JGrana

Very Senior Member
If you are willing ssh into the router to install Entware and do some minor editing, here is a script that should do what you want.
This script assumes the remote device/NAS supports SMB/Samba and you have set it up on the remote device. Asuswrt-merlin already has support for remote SMB using cifs.

I have done some testing and it seems to work as I believe you want.

You will need to have Entware installed. Refer to this thread:


Once Entware is installed, install rsync. This will be the application that does the copy and remove:

opkg install rsync

Then, using an editor (I use vi), create a file called "syncstuff.sh" in /jffs/scripts

vi /jffs/scripts/syncstuff.sh

Copy and paste this into the file:

Code:
#!/bin/sh
#
# syncstuff.sh
# Backup, using rsync, remote files to a local directory
# if files copied successfully, remove them from the remote directory
#

# Set remote NAS and local information

REMOTE_DEVICE_IP="192.168.1.210"   # IP of remote NAS/Server
REMOTE_NAME="Multimedia"           # Name of SMB Share
REMOTE_DIR="tst1"                   # Directory of remote files on smb share
SOURCE_DIR="/tmp/mnt/tst1"         # Name of SMB mounted directory (local mount point)
DEST_DIR="/tmp/tst1"               # Where the files should be copied to locally
USER_NAME="username"           # replace username with actual user name used on remote system
USER_PASSWD="password"         # replace password with the actual password of remote user
RSYNC_LOG="/tmp/rsync.log"         # log file for detailed rsync info

logger -t syncstuff "Syncing files to USB"

# if the remote directory isn't mounted, attempt to

if ! grep -qs $SOURCE_DIR /etc/mtab; then
        mkdir -p $SOURCE_DIR
        /bin/mount -v \\\\$REMOTE_DEVICE_IP\\$REMOTE_NAME\\$REMOTE_DIR $SOURCE_DIR -t cifs -o username=$USER_NAME,pass=$USER_PASSWD
fi

date >> $RSYNC_LOG

# if it mounted, attempt to rsync
if grep -qs $SOURCE_DIR /etc/mtab; then
        /opt/bin/rsync -rtu --stats --remove-source-files $SOURCE_DIR/ $DEST_DIR/ | grep files >> $RSYNC_LOG
        logger -t syncstuff "Sync complete"
        umount $SOURCE_DIR
else
        echo "Could not backup files, Remote directory not mounted" >> $RSYNC_LOG
        logger -t syncstuff "Syncing failed, remote not mounted"
fi

Be sure to make the script executable:

chmod +x /jffs/scripts/syncstuff.sh

Edit/Change the variables in UPPER case - I set them to what you showed in your image. Might work ;-) Double check the SMB name etc.
You will need to set the username (you are using root) and the password used to access the remote server.

For rsync, I am using the flags -rtu which says skip files newer copy directories as well as files from the remote.

Any files/directories in the remote directory will be copied to the local directory, then removed from the remote directory. The remote directories will NOT be removed, only the files contained in them. Files in the local directory are never removed - so it will likely grow over time.

The script will log its start and end in /tmp/syslog.log and also a more detailed log in RSYNC_LOG.
The RSYNC_LOG shows the date and time of the last run and how many files/dirs where copied. For example:

Sat Dec 19 10:20:39 EST 2020
Number of files: 5 (reg: 3, dir: 2)
Number of created files: 4 (reg: 3, dir: 1)
Number of deleted files: 0
Number of regular files transferred: 3

You can run it manually by simply executing it. You should try this a few times to make sure it's working and doing what you expect.
After running, you can tail syslog and should see a few lines:

tail /tmp/syslog.log
Dec 19 13:42:38 RT-AX88U-26A8 syncstuff: Syncing files to USB
Dec 19 13:42:38 RT-AX88U-26A8 syncstuff: Sync complete

You can also have it run on a schedule using cru. For example, for a once a day backup, add the following line to /jffs/scripts/services-start:

cru a "USB Backup" "30 3 * * * /jffs/scripts/syncstuff.sh"

This will run the syncstuff script at 3:30AM every day.

Good luck, hope this helps.
 

ftpusb

New Around Here
@JGrana thanks for the tutorial but I a still having problems,
got an error message while executing the script:

mount: mounting \\192.168.1.200\Multimedia\/mnt/mmc/record/ on /tmp/mnt/tst1 failed: Connection refused

This is the folder on the ftp server that I want to sync:
ftp://192.168.1.200/mnt/mmc/record

and this is the directory which the usb is attached to the router:
/mnt/sda1/2GB

Where did I do wrong in the config file below:

REMOTE_DEVICE_IP="192.168.1.200" # IP of remote NAS/Server
REMOTE_NAME="Multimedia" # Name of SMB Share
REMOTE_DIR="/mnt/mmc/record/" # Directory of remote files on smb share
SOURCE_DIR="/tmp/mnt/tst1" # Name of SMB mounted directory (local mount point)
DEST_DIR="/mnt/sda1/2GB" # Where the files should be copied to locally
USER_NAME="root" # replace username with actual user name used on remote system
USER_PASSWD="XXXXXX" # replace password with the actual password of remote user
RSYNC_LOG="/tmp/rsync.log" # log file for detailed rsync info
 

Jeffrey Young

Senior Member
Mount may not be what you want to use. You are trying to mount a cifs file system (smb1) , not a ftp site. Not sure mount has file system option to do that. You may have to download the curlftpfs package from entware and use that to mount your ftp site.

Opkg install curlftpfs

I don't have an ftp site to try this on. There is another thread here on snb with an example of how to use curlftpfs. Try a search (I would, but doing this on a phone screen is frustrating at the moment).
 

Jeffrey Young

Senior Member

JGrana

Very Senior Member
@JGrana thanks for the tutorial but I a still having problems,
got an error message while executing the script:

mount: mounting \\192.168.1.200\Multimedia\/mnt/mmc/record/ on /tmp/mnt/tst1 failed: Connection refused

This is the folder on the ftp server that I want to sync:
ftp://192.168.1.200/mnt/mmc/record

and this is the directory which the usb is attached to the router:
/mnt/sda1/2GB

Where did I do wrong in the config file below:

REMOTE_DEVICE_IP="192.168.1.200" # IP of remote NAS/Server
REMOTE_NAME="Multimedia" # Name of SMB Share
REMOTE_DIR="/mnt/mmc/record/" # Directory of remote files on smb share
SOURCE_DIR="/tmp/mnt/tst1" # Name of SMB mounted directory (local mount point)
DEST_DIR="/mnt/sda1/2GB" # Where the files should be copied to locally
USER_NAME="root" # replace username with actual user name used on remote system
USER_PASSWD="XXXXXX" # replace password with the actual password of remote user
RSYNC_LOG="/tmp/rsync.log" # log file for detailed rsync info

A few things. Does the remote server support SMB/Samba? What kind of device is it? If it's a NAS, you should be able to enable SMB support.
If it is a Linux server, putting SMB (aka Samba) on it is fairly easy.

The REMOTE_NAME should be the name of the "Share" that you create when you enable SMB. In my case, it was called "Multimedia". That would be the first problem - you likely don't have a share called Multimedia (at least yet).
The second is that the REMOTE_DIR should not start with a "/". Sorry, I should have coded it better. I would try "mmc/record" assuming the mmc directory is on the Share that you defined.
For example, on my NAS box (an older QNAP box) I have enabled SMB and allowed it to share the "Multimedia" folder/directory on the NAS. So, it shows up as \\192.168.1.210\Multimedia.
In the Multimedia folder I have a variety of other folders/directories. Using your ftp example, I would have a folder/directory called mmc. In that folder, another folder called record.
So, the SMB path would be \\192.168.1.210\Multimedia\mmc\record.

As @Jeffrey Young mentioned, if all you have is an ftp server, I can play around and make a version that uses curlftps as the method to attempt to mount. I will need to look into this (and enable ftp on my NAS).

It would be a bit easier (IMHO) to use SMB. Then, you could also use Windows machines as servers.
 

JGrana

Very Senior Member
Ok, lets try it your way ;-)

Here is a version that uses curlftpfs as an ftp method to mount the remote filesystem. Rather than go through the Samba/SMB setup, see if this works.

I have already changed some of the variables in UPPER_CASE. Double check them and also put in the password you have been using for ftp.

I did a quick test on my AX88U (which is starting to have some jffs CRC issues!!!!) and it seems to work fine.

Again, it requires Entware to be installed. But, it will attempt to install curlftpfs if it is not already installed.

Good luck!

ps: I'm actually impressed with how fast it transfers files. I might change all my backup scripts to use curlftpfs!

Code:
#!/bin/sh
#
# syncftp.sh
# Backup, using rsync, remote files using curlftpfs to a local directory
# if files copied successfully, remove them from the remote directory
#

# Set remote NAS and local information

REMOTE_SERVER="192.168.1.200"   # IP of remote FTP Server (CHANGE)
REMOTE_DIR="/mnt/mmc/record"   # Full path to directory of remote files (CHANGE)
FTP_MOUNT="/tmp/mnt/ftpfs"         # Where the remote filesystem will mount to - can leave alone
DEST_DIR="/mnt/sda1/2GB"               # Where the files should be copied to locally - usually USB (CHANGE)
USER_NAME="root"                  # ftp user name (CHANGE)
USER_PASSWD="password"            # ftp users password (CHANGE)
RSYNC_LOG="/tmp/rsync.log"         # log file for detailed rsync info - can leave alone

logger -t syncftp "Syncing files to USB"

# if the remote directory isn't mounted, attempt to

# make sure Entware is installed

if [ ! -f /opt/bin/opkg ]; then
        echo "Entware is not installed. Please install Entware and try again"
        exit
fi

# check for curlftpfs

if [ ! -f /opt/bin/curlftpfs ]; then
        echo "Installing curlftpfs"
        opkg install curlftpfs
fi



if ! grep -qs $FTP_MOUNT /etc/mtab; then
        mkdir -p $FTP_MOUNT
        curlftpfs -o allow_other $USER_NAME:[email protected]$REMOTE_SERVER $FTP_MOUNT
fi

date >> $RSYNC_LOG

# if it mounted, attempt to rsync
if grep -qs $FTP_MOUNT /etc/mtab; then
        /opt/bin/rsync -rtu --stats --remove-source-files $FTP_MOUNT$REMOTE_DIR/ $DEST_DIR/ | grep files >> $RSYNC_LOG
        logger -t syncftp "Sync complete"
        umount $FTP_MOUNT
else
        echo "Could not backup files, Remote directory not mounted" >> $RSYNC_LOG
        logger -t syncftp "Syncing failed, remote not mounted"
fi
 

Jeffrey Young

Senior Member
Cool, I was in the process of building a win10 VM to test this (and to play with a new thing) when company arrived and I had to put the computer away. Glad @JGrana got things going for you to try.
 

ftpusb

New Around Here
A few things. Does the remote server support SMB/Samba? What kind of device is it? If it's a NAS, you should be able to enable SMB support.
If it is a Linux server, putting SMB (aka Samba) on it is fairly easy.

The REMOTE_NAME should be the name of the "Share" that you create when you enable SMB. In my case, it was called "Multimedia". That would be the first problem - you likely don't have a share called Multimedia (at least yet).
The second is that the REMOTE_DIR should not start with a "/". Sorry, I should have coded it better. I would try "mmc/record" assuming the mmc directory is on the Share that you defined.
For example, on my NAS box (an older QNAP box) I have enabled SMB and allowed it to share the "Multimedia" folder/directory on the NAS. So, it shows up as \\192.168.1.210\Multimedia.
In the Multimedia folder I have a variety of other folders/directories. Using your ftp example, I would have a folder/directory called mmc. In that folder, another folder called record.
So, the SMB path would be \\192.168.1.210\Multimedia\mmc\record.

As @Jeffrey Young mentioned, if all you have is an ftp server, I can play around and make a version that uses curlftps as the method to attempt to mount. I will need to look into this (and enable ftp on my NAS).

It would be a bit easier (IMHO) to use SMB. Then, you could also use Windows machines as servers.


Mine is a Xiaomi IP camera with FTP/telnet enabled, main use is to dump the files into the usb attached storage.
 

JGrana

Very Senior Member
Mine is a Xiaomi IP camera with FTP/telnet enabled, main use is to dump the files into the usb attached storage.
Interesting, very clever!
Hope it works out.
 

Similar threads

Latest threads

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top