Smart Sync "upload to Cloud" not working (FTP)

[john9527]

When going back to 374.43, just take the above into account...

GH

It's also picked up in my fork of 374.43 (along with other security updates). You may want to give it a try if you need to roll back to this level.

http://forums.smallnetbuilder.com/showthread.php?t=18914

 

[opheliagrimm]

Great work on the fork John.

It just so happens that I do plan on setting up a plex server as a Christmas 2014 project to feed kid videos to the tablets in the house.

Will give your build a try to avoid any potential problems when the project starts.

Thanks for the heads up to you both.

 

[huotg01]

No luck so far.
No transfer, no error message, no log... I don't know where to look to debug this...

I think I'm going to install rSync. At least it is going to leve some traces, and it is a proven solution...

The images:

• screenshot from my side
• screenshot from other side (in fact there is a moving arrow)
• the log (empty)

 

[huotg01]

I installed Rsync. Worked well when testing. At least I know what I'm doing and what is going on. Rsync is very efficient and I will be able to keep the pace with the firmware upgrades...

Now trying to automate the process using cron. The unsolved part is about having Rsync working exclusively within a time window (let's say starting everyday at 01:00 and stopping at 06:00 even if the backup is not finished). Someone knows how to do that ?

GH

 

[RMerlin]

I installed Rsync. Worked well when testing. At least I know what I'm doing and what is going on. Rsync is very efficient and I will be able to keep the pace with the firmware upgrades...

Now trying to automate the process using cron. The unsolved part is about having Rsync working exclusively within a time window (let's say starting everyday at 01:00 and stopping at 06:00 even if the backup is not finished). Someone knows how to do that ?

GH

There is no clean way of stopping it. What you could do is to have a cron job at 6:00am that will run "killall rsync" to kill any process named rsync. It might leave you with a partial backup however.

 

[huotg01]

[...] It might leave you with a partial backup however.

Thanks. Good solution. It will do it. I have over 1.2Tb to copy over. The first run is going to take many days and I don't want to affect the response time during the day.
Also, if I well understand Rsync, it will eventually catch up, and after that "turning point", only new and modified files will have to be send over ?

GH

 

[RMerlin]

Thanks. Good solution. It will do it. I have over 1.2Tb to copy over. The first run is going to take many days and I don't want to affect the response time during the day.
Also, if I well understand Rsync, it will eventually catch up, and after that "turning point", only new and modified files will have to be send over ?

GH

That's correct. I extensively use it in my customer's datacenter to sync data and configs between production servers and failover servers. I also rsync the music between my desktop and my HTPC every hour, takes only a few seconds to go over ~80GB of files.

 

[opheliagrimm]

judging from the images you posted it looked like the process had begun.

Did you test it with a small share (say two or three 1mb files) to see if it was working before going for the 1.8gb backup?

I was very interested in Rsync before finding the router-to-router sync feature on this asus RT series.

Could you walk me through how I would go about using rsync instead?

It sounds more reliable and keeping up with the firmware would be great as well.

Thanks in advance

 

[huotg01]

judging from the images you posted it looked like the process had begun.
Did you test it with a small share (say two or three 1mb files) to see if it was working before going for the 1.8gb backup?

I noticed that "something was alive", but there was no trace of any backuped files on the remote side.

I was very interested in Rsync before finding the router-to-router sync feature on this asus RT series.
Could you walk me through how I would go about using rsync instead?
It sounds more reliable and keeping up with the firmware would be great as well. Thanks in advance

Good idea. We will make a small tutorial together. I will open a new thread and write down the steps by step to get it working in a ssh tunnel. You will try the draft procedure and come back with questions/remarks/corrections, and so on. This done, we will add the final result as a tutorial in the wiki. Is it ok with you?

That said, in short, the Rsync part is not a problem (I will explain "how to") if we don't care about entering the remote password, but we do care... ;-)

The problem is that if we want to start the daily Rsync without entering the remote password (this being not very practical in an unattended cron job, in fact impossible with ssh), we have to create a private and a public rsa keys. What is supposed to be easy is not working well for me, even if I have the impression that everything is well done. Usually I have no problem solving problems, but in this specific case the ssh engine is something called Dropbear (and Dropbearkey to generate the keys). As long as I can see, the version provided with RMerlin firmware doesn't have the debug options. It is therefore quite difficult to track down the problem (in fact what I'm doing wrong). But I'm confident that we will overcome this problem with the help of the more senior members.

GH

 

[opheliagrimm]

Sounds like a plan!

 

[RMerlin]

You might as well work directly on the Wiki, that way you will be able to edit one another's content rather than do a lot of back and forth in a forum thread

 

[huotg01]

You might as well work directly on the Wiki, that way you will be able to edit one another's content rather than do a lot of back and forth in a forum thread

Usually the forum is a better place to present a problem and solve it. When it is solved, the wiki is a good place to store the proven working recipe. Since, in the present case, it requires more than just "tuning" and documenting a procedure to make it a good tutorial, the forum seems to me a better place to interact with other members, isn't ?
In fact the only problem still unsolved is the use of the RSA keys to prevent the need to enter a password when using ssh to rsync the backup.
I'm trying to track down the problem but Dropbear is not very talkative... I don't even know if someone ever had any successes using a pair of private/public RSA keys to prevent using the password.

GH

 

[huotg01]

I will start a new thread to solve the "problem" with Dropbear/ssh/Rsa keys and the move on the wiki to document the process.

 

[opheliagrimm]

It was always my assumption that asus was using rsync buried in aicloud for the router-to-router sync.

Would they be able to shed any light on what method they used to avoid the issue?

 

[huotg01]

@opheliagrimm
The problem I had is now solved in thread:
http://forums.smallnetbuilder.com/showthread.php?t=21070

I tested with Rsync scheduled in a Cron job.
I should start to produce a tutorial on Friday in the wiki (if you are still interested)

RMerlin would be the best person to confirm, but Asus didn't use Rsync in its SmartSync feature. One thing I did found out after the fact is that Asus is building a PPTP VPN in its solution.

 

[RMerlin]

It was always my assumption that asus was using rsync buried in aicloud for the router-to-router sync.

Would they be able to shed any light on what method they used to avoid the issue?

Asuswebstorage relies on its own proprietary protocol, and the rest of AiCloud relies on webdav AFAIK (hard to tell for sure, it's all closed source).

 

[opheliagrimm]

Thanks for the clarification from you both.

And yes...I am definitely still interested in the wiki instructions for rsync

 

[huotg01]

And yes...I am definitely still interested in the wiki instructions for rsync

I started to build the tutorial in the wiki, just the outlines for now. "Lego" parts are working : ssh connection, rsync, etc, however I need to have a better understanding of what is going on with rsync. At the terminal prompt it starts, begins the transfer (there is messages on the terminal) and suddenly just "suspend" its progress, but there seems to have no errors reported. I think I will start a new thread with the specifics of my rsync command.

 

[opheliagrimm]

damn, I am sorry to hear that.

Was hoping rsync would be a more robust solution than the router-to-router sync.

Thankfully that is still working.

 

[huotg01]

I think it is going to work. There's another thread opened to solve the rsync problem.