two way sync with rsync

[soldstatic]

And don't say "unison" please!

If anyone can help me out, I'd greatly appreciate it.

I am using a NAS box in my house (western digital world book something or other) in my basement, a desktop vista PC, an ARM internet tablet (n770 from nokia), a laptop with vista, a shared host web server, a win xp machine at work, and a coworkers' vista desktop. I want to use my NAS box as a central file synchronization storage thing.

My goal is to have all of my home PCs and the 770 sync to the NAS box, my coworker's machine sync to the webserver, and nightly have the webserver sync with the NAS box. That way everybody will always have the same files. (then later, and I already found some great examples, have rsync create multiple snapshot backups on the NAS box so if something gets messed up I'm not in much trouble).

rsync is great at one way copies, and I know how to do that, but I could use some help creating the perfect rsync command to make a two way sync.

Problem is deletions. If I do a one way sync with --delete and -u, and then do the same sync but switch source / dest around, I will never get brand new files from the first destination on the new machine.

Script I've set up so far:

[Mr. M]

Well, AFAIK rsync has newer been intended for 2 way synchronization, so you should be using unison . However, I think even if you are using unison for the scenario you described you are likely to get into trouble, as it is likely that you and your co-worker change the same file at some point and you may lose your work accidentaly. I use unison to synchronize multiple computers im working from, but I always make sure to synchronize when I being working and at the end, like this the central repository is newer out of sync. Difficult to do with more than one person.

What I recommend is using a version control system, like subversion or git so that you and your co-worker can work on the same stuff. I'm not sure if you can use ssh+svn (i.e. subversion over ssh) in combination with your NAS box for this, you should investigate this option. You can then sync the web server by checking out the latest version and then use rsync (assuming the files on the webserver don't change).

[soldstatic]

hahahaha

Well, I don't want to do subversion because my coworker is a 70 year old man and I don't want to have to install subversion and then explain to him how it works. Besides, we will very rarely (in fact basically never) be working on the same files. For the most part, he acts as an accountant and researcher putting together docs. The sync would streamline our business cause I would get the docs so that I can read them later (i rarely make changes on them, just use them for reference). So I'm not too worried about us both editing one file. I'm worried mostly about when one of us moves or deletes somethings. Say we move a folder from one location to another, depending on the RSYNC we'd get that folder back doing it from the out-of-sync PC and then doing the inverse sync we'd get a copy of that folder in a new location (the destination of the move).

So I'm not worried about losing edits as i would be if we were working on code or something, plus I don't want to have to do svn checkouts from any of the PCs or the tablet, svn always lent itself to getting a new version of one file and a far too old version of another one with me, and it would probably screw us up somehow.

I'm using SyncToy by microsoft on my desktop and laptop to sync with the NAS on some samba shares, it might be able to work over FTP I don't know. Maybe try mapping an FTP drive from the coworker's location. Then I will also set him up on more of his PCs with synctoy as well. But I still have the problem of syncing the contents of the NAS box and the webserver nightly.

So my checklist of syncs:
laptop <-> NAS (synctoy) X
desktop <-> NAS (synctoy) X
coworker <-> webserver (synctoy) ?map ftp drive?
corworker <-> other company PCs (synctoy)
770 <-> NAS (? rsync / unison / svn ?) ?
NAS <-> webserver (? rsync / unison / svn ?) ?

Once I get everything syncing with the NAS box correctly including the webserver, I will probably just use rsync to do the backups. I found a great example that explained it, and I will keep a few 10min backups, a few hourly, a few daily, a few monthly, and one yearly for the last 3 years. Sounds like it's going to take a ton of space, and we do have a huge amount of data, but we really are only adding to the data instead of changing stuff, so I don't think it will actually end up taking up an inordinant amount of space.

[Mr. M]

For the rsync backups, look at the link-dest option, as described here http://www.mikerubel.org/computers/rsync_snapshots/. We use this to backup a 600GB RAID array to a 1TB disk nightly, as long as not too many things change we can keep backups for one week.

[soldstatic]

yea that's exactly what I plan to do.

So, nobody else has any suggestions to make rsync work for a two way sync??

[soldstatic]

well sadly i tried to install unison, but the package manager for the western digital my book world edition (linux based box I want to run everything off of) doesn't have it. So i dunno. Guess I need to compile it by hand or something?

[flybynite]

It's hard to tell exactly what you need but do look at rdiff-backup.

http://www.nongnu.org/rdiff-backup/features.html

It uses rsync to make an (almost) exact copy of a dir on a remote machine. The part that makes it interesting is it keeps diff's to make it possible to roll back changes to the dir like a version control system would. It uses rsync to only send the changes across the network. You would only need to copy, mount or rsync the dir to other boxes skipping one hidden dir that contains the version history info.

It can easily work over ssh and transfer from arbitrary boxes, like from box a you can backup box b to box c. Restoring would be just as easy.

Its a python script that calls librsync so it it very portable.

[soldstatic]

ok i'm looking at rsnapshot project. Looks very promising. And the config file helped me set it up so that rsnapshot will run these commands for me:

[ppurka]

Try to see whether there is an option in rsnapshot to turn off the -R flag of rsync. That is the argument required to provide to rsync to create such (relative) directory structures.
If you can't turn it off, try to see if you can provide --no-R as an option.
_________________
emerge --quiet redefined | E17 vids: I, II | Now using kde5 | e is unstable :-/

[soldstatic]

brilliant that worked perfectly.

Thank you!

[soldstatic]

OK When I first ssh in and run rsnapshot, I get an error that says