Rsync to Online Storage? Recommendations?

[eddomak]

Hi,

I am looking at getting a NAS, and notice that my 2 top choices (QNAP TS-209 II, and Synology DS207) have a remote replication/backup option that is based on rsync.

Can anyone recommend a cheap online storage facility that supports rsync? I am looking at backing up around 30GB-50GB. I am considering http://www.s3rsync.com/.

I would love it if there were any rsync to Mozy solutions out there - I am using Mozy on the laptop and loving it.

Thanks in advance.

 

[a.k.a.]

I know this is a sleeper of a thread, but I thought I'd pitch in here. It won't answer your question, but it will give you a necessary warning.

I shopped around quite a lot for online backup, and all I can say is, its time has not yet come.

My requirements were 1) 1+TB storage, 2) nothing gets deleted unless I approve, 3) cheap, and 4) encrypted.

I ended up going with ElephantDrive, because Mozy deletes any "missing" files since the last time you synced. There are a few other services out there that run as cheap as ED, but ED seemed pretty reliable, running on Amazon's A3 storage platform. To their credit, ED's support team is incredibly, incredibly helpful. They're working very hard to make it work for users.

The problem is that the upload rate is nil. I have about 250GB to back up, and after a full week of leaving my laptop running on a T1/T2 campus ethernet line, I was only at 3% done. This is not an exaggeration. They promise that the delta-sync after you finish the first backup goes much, much faster. But it's really not worth the front-loaded effort.

I cannot imagine that other online backup services are significantly faster in their upload rates. You should, however, test out the "free accounts" for upload speed, and try the top contenders out for a month.

It's not a matter solely of bandwidth. What you'll find is that all of these services have a file verification protocol that jams up the works. One service may have a faster one than another. But, to take an example: If you built an ISO file of of CD size (~700MB), you may have to wait 3-4 hours for that to upload. If any of the data packets don't get delivered, it has to start over, either in part or in full. It left me flabbergasted.

Be prepared for a messy breakup with whichever service you choose. And, if you have time to test the upload speed of ED versus other services, it would be great if you could post back here and let us know.

Cheers!

 

[eddomak]

Thanks for the reply and experience. I figured out a solution that wasn't ideal, but is manageable. Here it is:

1) Purchased QNAP TS-209 II (this wouldn't make any difference to the solution)

Essential Documents/files
2) Use Windows "Offline Files" functionality for essential documents
3) Backup (2) by configuring Mozy to include the Offline Files Cache directory

Less accessed documents/files that change less frequently
4) Use another PC (my laptop) to run Acronis True Image for reading files across the network and creating incremental backups to an online FTP server

Advantages

• this avoids having all documents on my laptop which can't fit 1TB anyway, and which is the point of a NAS)
• the incremental backups are handled by Acronis, so it reduces the size/time that my laptop is taken up (usually a couple of hours overnight)
• Acronis can handle backup/restore directly to/from FTP servers
• depending on how frequently you backup, this determines the window of lost data
• I stand to lose 1 week's worth of non-essential documents
• Does not require an online rsync server (which seems to cost a lot of money!)
• all essential documents are backed up constantly and exist in several places
  
  [*] NAS Copy
  [*] PC Cached Copy
  [*] NAS Backup Copies
  [*] PC Mozy Backup Copies​
  
  

Disadvantages

• Requires a separate computer to run the backups
• Need to spend money on Acronis if you don't already have it (I already had it)
• There has been some technical issue (that I have overcome with a workaround) with Acronis and backing up from some NAS solutions
• Can utilise FTP server space (some places have unlimited storage for a very low fee)

All in all, I am pretty happy with this despite the disadvantages.

Cheers!

 

[a.k.a.]

Wow. Dividing things up does sound ideal.

Can I just ask for clarification, however, out of unblissful ignorance?

Which is it? You have a desktop and ...

1. One NAS hooked into the desktop, to which, at nighttime, you hook up your laptop, and using Acronis, do incremental backups to an online FTP Server?

2. A P2P ethernet link between desktop and laptop, with the laptop driving the backup of the desktop?

You have several local NASes? (I was confused by the references to "NAS copy + NAS backup copy" and also to a "very low fee" "online" FTP.)

If the FTP Server is actually offsite, which I guess it is, which kind do you use?

And is that store encrypted?

Any idea what your transfer rates are? -- though I understand that overnight it doesn't really matter.

Thanks!
a.k.a.

 

[eddomak]

Hi,

Hopefully the attached diagram illustrates the lot...

The NAS is attached to the network, and not to any desktop. (NAS by definition is connected to the Network, as opposed to Direct Attached Storage (DAS) typically being a USB drive connected Directly to the PC).

Laptop1 has Acronis on it, and Acronis reads from the NAS (dotted blue line - which strictly speaking should go through the router, then into the Laptop), determines incremental changes, encrypts, and sends the snapshot data to the FTP server (multiple blue boxes).

If you look for the instances of "EM Docs" on the diagram, you'll see how there are multiple versions in different locations at any given time.

NAS Backups are in the Dark Blue Boxes.
Mozy Backups for my laptop are in the Light Blue Boxes.

I have not implemented any rsync backup (as indicated by the green line).

Cheers,
Edwin.

No idea of transfer rates.

 

[claykin]

Mr

www.jungledisk.com

Uses Amazon S3 for storage and supports Rsync.

 

[noamt]

jungledisk.com rsync

Rrync to Amazon through jungledisk don't work well. The backup time is very long, It always re upload the modify files all over again.

S3rsync.com implement full Rsync protocol that mean only small parts of modified files are upload, so my backup is much faster.

noam

 

[QuickThinking]

Is everyone here looking at this from a business perspective? From a home user perspective these solutions aren't practical because your ISP would pull the plug on your connection if you tried to upload this much data.

I have a post up called Off Site - NAS Backup which I'm hoping to get some feedback on soon. Using rsync with either an SSH or VPN tunnel seems ideal but I'm just not sure what is the most cost effective way to set that up. Ofcourse my solution relies on the idea of doing the initial backup locally and moving a unit off site somewhere to keep it mirrored from there on. Any suggestions?

 

[thiggins]

Is everyone here looking at this from a business perspective? From a home user perspective these solutions aren't practical because your ISP would pull the plug on your connection if you tried to upload this much data.

That's an interesting point. Is your cap daily, weekly, monthly? No options to buy additional bandwidth.

The reason I mention this is that cloud backup is a reasonable option for small businesses or home users that aren't trying to back up Terabytes of ripped videos, but, say, their financial and personal docs.

I personally have been beating on NAS vendors to add cloud back up options. It would be stupid if ISPs screwed this up due to bandwidth caps.

 

[QuickThinking]

That's an interesting point. Is your cap daily, weekly, monthly? No options to buy additional bandwidth.

I had to do a bit of research, but pulled directly from Cox Cable's Acceptable Use Policy and Limitations of Service:

1. Premiere with PowerBoost Package
   • Max Speed: 10 - 20 Mb down ; 1 - 2 Mb up
   • Max Limit (monthly): 60 GB down ; 15 GB up
2. Preferred with PowerBoost Package
   • Max Speed: 5 - 9 Mb down ; 512 Kb - 2 Mb up
   • Max Limit (monthly): 40 GB down ; 10 GB up
3. Value Package
   • Max Speed: 256 Kb - 1.5 Mb down ; 256 - 384 Kb up
   • Max Limit (monthly): 4 GB down ; 1 GB up
4. Economy Package
   • Max Speed: 512 - 768 Kb down ; 256 Kb up
   • Max Limit (monthly): 3 GB down ; 1 GB up

Personally I'm on the Preferred Package, which is probably sufficient to deal with my month to month additions to my backups (pictures and music mostly) but nowhere near what I would need in order to initially set up my offsite backup. Therefore, I feel I have no choice but to configure two devices, initially synchronize them, and then move one of them offsite and configure the necessary VPN / port forwarding / etc in order to get them backuping up ONLY the changes.

My ISP just can't accomodate either the initial backup, or the restore for that matter.

PS> Sorry about the numbered list, I wanted to paste a table but found it suprisingly difficult and gave up.

 

[thiggins]

Thanks for the info. I'll see if I can get a comment on this from NAS manfs...

 

[thiggins]

I had to do a bit of research, but pulled directly from Cox Cable's Acceptable Use Policy and Limitations of Service:

I'm still researching, but NETGEAR pointed out that Comcast backed off these low limits last October and raised the residential monthly limit to 250 GB.

 

[claykin]

Rrync to Amazon through jungledisk don't work well. The backup time is very long, It always re upload the modify files all over again.

S3rsync.com implement full Rsync protocol that mean only small parts of modified files are upload, so my backup is much faster.

noam

When testing Jungledisk, you must subscribe to the Jungledisk Plus service ($1/month) to have the Delta file copy capability. S3 does not natively support Rsync (Delta copying) so Jungledisk did this via an Amazon EC2 server they deployed.

Also, I find Jungledisk to be very speedy over my 8/2 cable connection. I recommend you test drive Jungledisk again and this time use the Plus service. Also, if you have a high bandwidth connection Jungledisk has an option to enable large TCP windows. This helps with my 2Mb/s upstream to keep the data flowing...

With Jungledisk all data is encrypted using AES256 and you are the only one with the keys. You maintain control at all times. Steve Gibson has verified the way Jungledisk handles security and he has given it his stamp of approval. Read more here: http://blog.jungledisk.com/2006/06/06/encryption/

Jungledisk also now supports Rackspace.com (their new owners). Rackspace is similar to S3 but Rackspace is not charging bandwidth and request fees. Read more here: http://blog.jungledisk.com/2009/03/12/jungle-disk-26-released-now-with-cloud-files-support/

No, I do not work for Jungledisk. I'm a user of theirs for 1.5 years and am very happy with the service.

 

[Antwerp]

I know this is a sleeper of a thread, but I thought I'd pitch in here. It won't answer your question, but it will give you a necessary warning.

I shopped around quite a lot for online backup, and all I can say is, its time has not yet come.

My requirements were 1) 1+TB storage, 2) nothing gets deleted unless I approve, 3) cheap, and 4) encrypted.

I ended up going with ElephantDrive, because Mozy deletes any "missing" files since the last time you synced. There are a few other services out there that run as cheap as ED, but ED seemed pretty reliable, running on Amazon's A3 storage platform. To their credit, ED's support team is incredibly, incredibly helpful. They're working very hard to make it work for users.

The problem is that the upload rate is nil. I have about 250GB to back up, and after a full week of leaving my laptop running on a T1/T2 campus ethernet line, I was only at 3% done. This is not an exaggeration. They promise that the delta-sync after you finish the first backup goes much, much faster. But it's really not worth the front-loaded effort.

I cannot imagine that other online backup services are significantly faster in their upload rates. You should, however, test out the "free accounts" for upload speed, and try the top contenders out for a month.

It's not a matter solely of bandwidth. What you'll find is that all of these services have a file verification protocol that jams up the works. One service may have a faster one than another. But, to take an example: If you built an ISO file of of CD size (~700MB), you may have to wait 3-4 hours for that to upload. If any of the data packets don't get delivered, it has to start over, either in part or in full. It left me flabbergasted.

Be prepared for a messy breakup with whichever service you choose. And, if you have time to test the upload speed of ED versus other services, it would be great if you could post back here and let us know.

Cheers!

Thanks for that!

Online backup gets my vote, it's so much more easier to use.

 

[Corneth]

Hi,

I am looking at getting a NAS, and notice that my 2 top choices (QNAP TS-209 II, and Synology DS207) have a remote replication/backup option that is based on rsync.

Can anyone recommend a cheap online storage facility that supports rsync? I am looking at backing up around 30GB-50GB. I am considering http://www.s3rsync.com/.

I would love it if there were any rsync to Mozy solutions out there - I am using Mozy on the laptop and loving it.

Thanks in advance.

The only one I'm aware of that provides

o) rsync over secure SSH
o) mounting the storage as a local drive, using SMB / CIFS over secure VPN
o) browser based file manager
o) mobile phone support, WebDAV over secure SSL
o) secure scp
o) secure sftp
o) truecrypt support

is Strato :

http://www.strato-hosting.co.uk/

http://www.strato.de/

and several other Strato sites.

Usually, see "footnotes" of main Strato pages in several countries to find the other countries where they provide HiDrive.

Found these specs here:

http://www.deltamatica.nl/home/archive/online/

No need to say that rsync is convenient / efficient to use.
Well, my $0.02