| View previous topic :: View next topic |
| Author |
Message |
John the Kiwi
n00b
Joined: 03 Dec 2003
Posts: 39
|
 Posted: Tue Feb 24, 2004 5:35 pm Post subject: Network Backup with Rsync - Best Practises? |
 |
|
I have a web server that has a site with 500K+ files in it. It's about 15 gigs in size and I'm wanting to use rsync to back it up to a Windows share I have open on another server (this share is compressed).
I want to make a cron job that mounts the share and then copies only the changed data to the share. It seems that I would need to use rsync.
Does anyone have any suggestions for this? Should I instead gzip a flat file that only adds the changed files into the archive?
Here's roughly what I'm thinking:
-----------------------------------------------------------
mount -t smbfs -o username=user,password=pass,gid=backup,uid=backup //xxx.xxx.xxx.xxx/backup /mnt/backup
rsync /home/website /mnt/backup
line to send a confirmation email with a log
umount /mnt/backup
-----------------------------------------------------------
I'm depending on the compressed drive to make the files a lot smaller but really as long as only the changed files are being copied each night I don't really care how it's done.
If anyone has a better way to do this or could clean up the code I put in there to achieve my goals I would be very grateful and would write up a faq for the site somewhere.
TIA
John the Kiwi |
|
| Back to top |
|
 |
John the Kiwi
n00b
Joined: 03 Dec 2003
Posts: 39
|
| Posted: Fri Feb 27, 2004 5:40 pm Post subject: |
|
|
Shameless bounce.
Anyone at all got any suggestions for an online backup that doesn't require expensive hardware?
A URL perhaps?
Sample rsync scripts?
Is rsync even the right tool for the job?
Surely some of us backup our file systems?
Or is everyone just thinking I'm too lazy to do it myself? Because I will you know. I'll do it myself and post my solution here - you can't stop me.
I was just hoping the wheel had already been invented.
john the Kiwi |
|
| Back to top |
|
|
georwell
Guru
Joined: 25 Jun 2003
Posts: 430
Location: Uppsala, Sweden
|
| Posted: Fri Feb 27, 2004 9:54 pm Post subject: |
|
|
Just use tar and it will work the same and you can only backup the files that have changed since the last backup. This way you can also encrypt using gzip or bzip2 while backing up. Take weekly full backups if your paranoid and then incremental daily backups and repeat the process.
Never forget the most important rule of all when backing up! Randomly restore files from the backups at least once a week. Otherwise you might never know if something wrong. You do have an office peon to make do this right?  |
|
| Back to top |
|
|
creese
n00b
Joined: 13 Aug 2003
Posts: 58
Location: Folsom, CA
|
| Posted: Fri Feb 27, 2004 11:18 pm Post subject: |
|
|
I use rsync to perform daily backups that are hardlinked together. This means each backup only takes the room of a incremental, but appears to be a full backup when you look at the directory. I've attached my script if you're interested. I don't claim the code to be pretty...
| Code: |
#!/usr/bin/python
import os
import sys
import time
# rsync
# -a Archive
# --whole-file Don't do partials so hardlink copy works
# --one-file-system Stay on the specified filesystem (so root can be done)
# --delete-after Remove files after rsync
# --rsh="ssh -x" Use ssh without X forwarding
RSYNC_CMD = "rsync -a --numeric-ids --whole-file --one-file-system --delete-after --rsh=\"ssh -x\""
BACKUP_DIR = "/backup/rsync"
TIME_BETWEEN_BACKUPS = 11 * 60 * 60
BACKUP_LIST = [ "diskbox:/",
"mediabox:/",
"athlon:/data/Mystuff",
"athlon:/",
"laptop:/win2k/work",
"laptop:/" ]
#
# Execute a shell command line
#
def execCmd (cmd, args=[], allowFail=0):
command = cmd
for a in args:
command += ' ' + a
status = os.system(command)
if (status != 0) and (not allowFail) :
raise Exception("Command '" + command + "' failed with status=" + str(status))
return status
#
# Convert a backup source into a destination directory base name
#
def getOutputBase (source):
parts = source.split(":")
remoteHost = parts[0]
remotePath = parts[1]
if (remotePath == "") or (remotePath == "/"):
remotePath = "root"
else:
remotePath = "root" + remotePath.replace("/", ".")
return BACKUP_DIR + "/" + remoteHost + "_" + remotePath
#
# Backup a source
#
def doBackup (source):
print "*******************************************************"
print "Starting: " + source + " at " + time.ctime()
print "*******************************************************"
host = source.split(":")[0]
print "Checking if " + host + " is up..."
execCmd("ssh -x " + host + " echo \"Node is up\"", [])
latestDir = getOutputBase(source) + "_latest"
outputDir = getOutputBase(source) + "_" + time.strftime("%Y_%m_%d")
try:
os.stat(outputDir)
print "Backup exists: " + outputDir
except:
print "Creating: " + outputDir
os.mkdir(outputDir)
try:
os.stat(latestDir)
print "Copying last backup using hard links..."
execCmd("cp -Rl", [latestDir + "/*", outputDir + "/"])
except:
pass # No older backup available
inputDir = source.rstrip("/") + "/"
print "Performing Rsync..."
execCmd(RSYNC_CMD, [inputDir, outputDir])
try:
os.unlink(latestDir)
except:
pass # Don't care if this failed
print "Marking the latest backup..."
os.symlink(outputDir, latestDir)
print "Done!"
#
# The main backup loop runs forever
#
while (1):
print "Processing Backups..."
for source in BACKUP_LIST:
try:
doBackup(source)
except Exception, e:
print "Unexpected error: ", e
print "Failed!"
print "*******************************************************"
print "Complete at " + time.ctime()
print "Waiting for next backup iteration..."
time.sleep(TIME_BETWEEN_BACKUPS)
|
|
|
| Back to top |
|
|
taskara
Advocate
Joined: 10 Apr 2002
Posts: 3763
Location: Australia
|
|
| Back to top |
|
|
|
Networking & Security
