Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
HOWTO: Easily make a full system backup (stage4)
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4 ... 16, 17, 18  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
jbannon
Tux's lil' helper
Tux's lil' helper


Joined: 15 Mar 2005
Posts: 99
Location: Paisley Scotland

PostPosted: Sun Apr 10, 2005 10:07 pm    Post subject: Reply with quote

Nice script, well presented and easy to understand. Works fine for me. I haven't been brave enough to wipe my system and try a full restore but listing it certainly seemed fine (no reports of any corruption). Only mods I made were the placement of the backup file on my external iomega drive and to include /usr/portage/distfiles (I wanted this because some packages I had to manually download). One mod I will do is to create the backup on the fast disk and then move it to my iomega as running bzip on that is mighty slow. I'll also use it as a basis for doing some others, like a daily backup of my data and so forth.
_________________
Best Regards,
Jim Bannon
(When in doubt, try honesty!)
Back to top
View user's profile Send private message
javock
Tux's lil' helper
Tux's lil' helper


Joined: 01 Jan 2004
Posts: 106
Location: Babeland... yeah! I wish!

PostPosted: Mon Apr 11, 2005 1:10 am    Post subject: Reply with quote

Hey nice script...

I found it usefull, but, correct me if I am wrong, to make the backup, it first creates the tar, and then compresses it right?
I can't afford to duplicate the space of my instalation, so I made some corrections to the script.
First, as you clearly need some files on the excluded directories, and tar wont accept to exclude the directory and include some file, I create a list with the excluded files, on this list I omit the files to be included on the tar. Then I pass the -X option and this created file to tar and modify the script to pipe the result from tar to gzip/bzip2...

PM me if you are interested on the sugested behavior...

Regards.
_________________
The heart has reasons that the reason will never understand.

-- La Renga. El final es donde partí
Back to top
View user's profile Send private message
slydini
Tux's lil' helper
Tux's lil' helper


Joined: 30 Oct 2002
Posts: 129
Location: Virginia Beach, VA

PostPosted: Tue Apr 12, 2005 4:38 am    Post subject: Reply with quote

Quote:
Strange, me too, I might try tar.gz as space isn't an issue on my external hard drive. I'm guessing that bz2 doesn't find large achives attractive? Perhaps tar.gz will perform better. Integrity check failure.

"Unexpected EOF (End Of File)"


I have seen bad media or certain brands of cdr's cause the above "Unexpected EOF" type error. I have also seen the file size not be correct when doing a long listing on a linux machine (shows 20Mb instead of 1G) but read the correct file size on a windows machine. The only difference being a change in media to fix the problem and be able to read the tar.gz files on the linux machine. BTW, I was able to extract the needed files from the seemingly bad media by copying the tar balls to an nfs mount from a windows machine to a linux machine. Then I was able successully extract the files.

What type of cdr's is everyone using? I have had good luck with the sony brand.
_________________
_______________________________________
Oh how I love Linux, especially on gentoo.
Back to top
View user's profile Send private message
slydini
Tux's lil' helper
Tux's lil' helper


Joined: 30 Oct 2002
Posts: 129
Location: Virginia Beach, VA

PostPosted: Tue Apr 12, 2005 4:49 am    Post subject: Reply with quote

Javoc, what are the files that are being excluded that you say are clearly needed? If they are needed then I would be interested in your modifications. I have not restored using this particular script yet but it appears to be viable. I have a spare hard drive so I think I will try to restore to it some time this week.
_________________
_______________________________________
Oh how I love Linux, especially on gentoo.
Back to top
View user's profile Send private message
BlinkEye
Veteran
Veteran


Joined: 21 Oct 2003
Posts: 1046
Location: Gentoo Forums

PostPosted: Tue Apr 12, 2005 5:23 am    Post subject: Reply with quote

javock wrote:
Hey nice script...

I found it usefull, but, correct me if I am wrong, to make the backup, it first creates the tar, and then compresses it right?
I can't afford to duplicate the space of my instalation, so I made some corrections to the script.
First, as you clearly need some files on the excluded directories, and tar wont accept to exclude the directory and include some file, I create a list with the excluded files, on this list I omit the files to be included on the tar. Then I pass the -X option and this created file to tar and modify the script to pipe the result from tar to gzip/bzip2...

PM me if you are interested on the sugested behavior...

Regards.

thanks for your pm. i had a similar version once. i don't like the fact of creating first the stage4.tar and then zipping it neither. but i chose transparency and simplicity for the script as others are using it too. give me some time, i'm testing your idea and maybe will change the script. a tar gets so big i almost have no choice ...

slydini wrote:
Javoc, what are the files that are being excluded that you say are clearly needed? If they are needed then I would be interested in your modifications. I have not restored using this particular script yet but it appears to be viable. I have a spare hard drive so I think I will try to restore to it some time this week.

don't worry, i haven't forgotten something. what he meant is the issue about tar not backing up files which are in a subfolder of an excluded folder
_________________
Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick
Back to top
View user's profile Send private message
Coume
Guru
Guru


Joined: 11 Jun 2004
Posts: 306

PostPosted: Tue Apr 12, 2005 9:18 am    Post subject: Reply with quote

Hello,

I just come accross this script but have no time right now to test it :/
But after reading the thread, I'm wondering one thing.

couldn't it be possible to generate directly backups CDs which include the LiveCD on the first one, so it could be possible to create an automated restauration script? Cos' the HUGE advantage I see with this script, is to recover a fucked up system while travelling and not having Internet access.
So if it was possible to only carry the backups CDs and knows that everything is in there... It would be perfect.

Just my 2 Euro cents
Ludo
_________________
MythTVtalk.com - The MythTV Community forum
Back to top
View user's profile Send private message
javock
Tux's lil' helper
Tux's lil' helper


Joined: 01 Jan 2004
Posts: 106
Location: Babeland... yeah! I wish!

PostPosted: Tue Apr 12, 2005 12:36 pm    Post subject: Reply with quote

slydini wrote:
Javoc, what are the files that are being excluded that you say are clearly needed? If they are needed then I would be interested in your modifications. I have not restored using this particular script yet but it appears to be viable. I have a spare hard drive so I think I will try to restore to it some time this week.


As blinkeye said... those files are not left away in the script, it is a matter of implementation... tar does not allow you to use --exclude=<dir> and then try to explicity include a file that exists inside the excluded dir. So right now, the script first creates a tar with these files, and then appends the rest of the backup... simple, but as always it has a trade off, it first creates a full tar of the system (in my case around 8G 8O ) writing it to disk *and* afterwards it compresses it. For me this was not an option as I cannot spare another 8G :roll: so I prefered to pipe the tar to gzip and only use the disk space to store the backup until I got to write it to a DVD. :lol:

The solution was to create an exclude file with where you explicitly list the files to exclude, and in creating this file you /omit/ :wink: the files you want to keep. Not so simple, but a little more efficient.

BlinkEye wrote:
but i chose transparency and simplicity for the script as others are using it too. give me some time, i'm testing your idea and maybe will change the script. a tar gets so big i almost have no choice ...


As you can see my sugestion 8) is being taken into consideration... I hope it makes it to the stable branch :lol:
_________________
The heart has reasons that the reason will never understand.

-- La Renga. El final es donde partí
Back to top
View user's profile Send private message
feld
Guru
Guru


Joined: 29 Aug 2004
Posts: 593
Location: USA

PostPosted: Tue Apr 12, 2005 5:04 pm    Post subject: Reply with quote

If anyone else is having problems compressing with the bzip2 method please take the .tar and attempt to manually bzip2 it with the -s (--small) switch. I was having problems not being able to make it past 400mb and currently with the --small switch I have made it to : 1.3gb (almost done). The speed compressing also appears to be the exact same.

I guess this is supposed to be more memory-friendly and not thrash and stress your RAM so much.

I know my RAM isnt faulty cuz i just tested it with memtest86+ not too long ago... and I have plenty (1gig).



-Feld
_________________
< bmg505> I think the first line in reiserfsck is

if (random(65535)< 65500) { hose(partition); for (i=0;i<100000000;i++) print_crap(); }
Back to top
View user's profile Send private message
BlinkEye
Veteran
Veteran


Joined: 21 Oct 2003
Posts: 1046
Location: Gentoo Forums

PostPosted: Tue Apr 12, 2005 6:27 pm    Post subject: Reply with quote

feld wrote:
If anyone else is having problems compressing with the bzip2 method please take the .tar and attempt to manually bzip2 it with the -s (--small) switch. I was having problems not being able to make it past 400mb and currently with the --small switch I have made it to : 1.3gb (almost done). The speed compressing also appears to be the exact same.

I guess this is supposed to be more memory-friendly and not thrash and stress your RAM so much.

I know my RAM isnt faulty cuz i just tested it with memtest86+ not too long ago... and I have plenty (1gig).



-Feld

please not that the issue about bzip2 and broken bzip2 files has nothing to do with the script.

Feld just dropped me a note and he "confessed" that his bzip2 isn't working - so, the --small switch does no good. i don't know what your problem is - i just tested the script over some of my mp3 files (resulting tar was 25G) and after 2.4GB of the resulting tar.bz2 i gave up. bzip2 is working flawlessly here and obviously is able to handle large files. i suggest you open a new thread about your issue ...
_________________
Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick
Back to top
View user's profile Send private message
BlinkEye
Veteran
Veteran


Joined: 21 Oct 2003
Posts: 1046
Location: Gentoo Forums

PostPosted: Tue Apr 12, 2005 6:34 pm    Post subject: Reply with quote

Coume wrote:
Hello,

I just come accross this script but have no time right now to test it :/
But after reading the thread, I'm wondering one thing.

couldn't it be possible to generate directly backups CDs which include the LiveCD on the first one, so it could be possible to create an automated restauration script? Cos' the HUGE advantage I see with this script, is to recover a fucked up system while travelling and not having Internet access.
So if it was possible to only carry the backups CDs and knows that everything is in there... It would be perfect.

Just my 2 Euro cents
Ludo

if you backup the stage4 on another media than the to-be-fsck-harddisk you are set. boot to your fsck system with a regular livecd (whatever that might be) and untar the stage4. i thought about the solution of creating a livecd, but it's simply not worth the effort. i don't have a DVD burner yet and a stage4 of <670MB is simply very small. even if i had a DVD burner i don't gain anything if i'd create a livecd with the stage4: you get the same result just burning it on a cd/dvd and using another livecd. someone posted a link about creating a bootable livecd - so, if you really want to do it, just follow the link. i created a custom livecd, so, it's working. but if i fsck my harddisk/sytem i'd want a current stage4, so, i burn the cd/dvd if in need.
_________________
Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick
Back to top
View user's profile Send private message
Coume
Guru
Guru


Joined: 11 Jun 2004
Posts: 306

PostPosted: Tue Apr 12, 2005 6:45 pm    Post subject: Reply with quote

Erm, so u really just have to untar the file and you are set??

Sounds too good to be true... :) But it will keep in memory, rights, user, etc. ? everything?? no need to chmod thingie, ?

Ludo
_________________
MythTVtalk.com - The MythTV Community forum
Back to top
View user's profile Send private message
BlinkEye
Veteran
Veteran


Joined: 21 Oct 2003
Posts: 1046
Location: Gentoo Forums

PostPosted: Tue Apr 12, 2005 6:54 pm    Post subject: Reply with quote

Coume wrote:
Erm, so u really just have to untar the file and you are set??

Sounds too good to be true... :) But it will keep in memory, rights, user, etc. ? everything?? no need to chmod thingie, ?

Ludo

i mentioned that in "5. What now". it's not too good to be true, it IS true.
_________________
Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick
Back to top
View user's profile Send private message
ctachta
n00b
n00b


Joined: 09 Apr 2005
Posts: 39
Location: Glasgow

PostPosted: Tue Apr 12, 2005 7:07 pm    Post subject: Reply with quote

Everyone,

Can I ask something stupid? :oops: :D

When the script is interrupted with ctrl-c does it leaves everything somewhere at the hard disk to float. Is that right? Where is that at tha hard drive?

Thanks a lot,
ctachta
Back to top
View user's profile Send private message
BlinkEye
Veteran
Veteran


Joined: 21 Oct 2003
Posts: 1046
Location: Gentoo Forums

PostPosted: Tue Apr 12, 2005 7:11 pm    Post subject: Reply with quote

did you change something? if not - it's in /mnt/backups/stage4/
_________________
Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick
Back to top
View user's profile Send private message
ctachta
n00b
n00b


Joined: 09 Apr 2005
Posts: 39
Location: Glasgow

PostPosted: Tue Apr 12, 2005 7:16 pm    Post subject: Reply with quote

No I did not. I told you it was stupid question I just figure it out just now. :D

Thanks VERY VERY much for the script. :D
Back to top
View user's profile Send private message
ctachta
n00b
n00b


Joined: 09 Apr 2005
Posts: 39
Location: Glasgow

PostPosted: Tue Apr 12, 2005 7:25 pm    Post subject: Reply with quote

Guys,

I did try to make the script perform the operation silently by redirecting the output to a file like this

sh mkstage4.sh > stage4.log

and I've got this error/warning

Code:

Total bytes written: 348160 (340KiB, 93KiB/s)

tar: /var/lib/dbus/system_bus_socket: socket ignored
tar: /var/run/mysqld/mysqld.sock: socket ignored
tar: /var/run/cgisock: socket ignored


is it safe to ignore it? What does this mean and what is caused from?

Thanks
Back to top
View user's profile Send private message
BlinkEye
Veteran
Veteran


Joined: 21 Oct 2003
Posts: 1046
Location: Gentoo Forums

PostPosted: Tue Apr 12, 2005 7:31 pm    Post subject: Reply with quote

yes. this is safe to ignore. such messages result from files which have changed/were deleted while tar is backing them up.
_________________
Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick


Last edited by BlinkEye on Tue Apr 12, 2005 10:22 pm; edited 1 time in total
Back to top
View user's profile Send private message
slydini
Tux's lil' helper
Tux's lil' helper


Joined: 30 Oct 2002
Posts: 129
Location: Virginia Beach, VA

PostPosted: Fri Apr 15, 2005 1:29 am    Post subject: Out of the box? Reply with quote

A question for BlinkEye: Is there any way you can eliminate the zip command in your script? What is the purpose of zip? I found that in order to use your script on most of my machines it is nescessary to install zip first. That kind of blows the "out of the box" goal out the window. I really like the script otherwise.

I have some older machines that never get updated at all that I would like to backup with your script but I can't because of the need for zip to be installed. These machines are not connected to the net in any way.

If not, it would be great if someone could create a live-cd that had zip and your script on it.
_________________
_______________________________________
Oh how I love Linux, especially on gentoo.
Back to top
View user's profile Send private message
BlinkEye
Veteran
Veteran


Joined: 21 Oct 2003
Posts: 1046
Location: Gentoo Forums

PostPosted: Fri Apr 15, 2005 9:39 am    Post subject: Reply with quote

well, $zip is a variable for either bzip2 or gzip - depending on your choice at the beginning. these are tools, i'd say every system has installed. at the beginning of the script i set a list
Code:
command_list="echo tar hostname date split zip bzip2"

which are the tools needed.

ah i see - this is a typo. zip should be gzip. the command_list should be:
Code:
command_list="echo tar hostname date split gzip bzip2"

thanks. updated v.1.4.0 to 1.4.1
_________________
Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick
Back to top
View user's profile Send private message
slydini
Tux's lil' helper
Tux's lil' helper


Joined: 30 Oct 2002
Posts: 129
Location: Virginia Beach, VA

PostPosted: Fri Apr 15, 2005 1:34 pm    Post subject: What a relief. Reply with quote

Now it makes sense. After looking at the script it looked like zip was a variable but I couldn't figure out why it was in the command list. So I checked on portage for a program called zip and sure enough there was one. I emerged zip and the script worked fine after that. So the last few days I have been trying to figure how to get that little zip program on my older "never been updated in almost 2 years" gentoo machines.

I should probably note for the beginners out there that this script or any other for that matter will give you the following error if you copy the script from a windows machine without converting the text file:
Code:
# ./stage4.sh
: No such file or directory.


You can use the dos2unix utility to convert any such files to the proper format.

Code:
emerge dos2unix
dos2unix stage4.sh


Not knowing the above caused great frustration for many hours on some of my older machines that were not on the network. I have a 40gig firelite usb drive that I use to Xfer files between machines. Anyways, just somthing to consider.
_________________
_______________________________________
Oh how I love Linux, especially on gentoo.
Back to top
View user's profile Send private message
gdi2k
Tux's lil' helper
Tux's lil' helper


Joined: 15 Oct 2004
Posts: 88
Location: Dubai

PostPosted: Sat Apr 16, 2005 1:35 pm    Post subject: Thanks! Reply with quote

Just wanted to say thanks for an excellent script! I used it to convert my 2 identical hard drives into a RAID 1 array today, without the fear of having to reinstall gentoo again from scratch. It took me a few hours to shift all the data around, and get the disks synched, but it appears to have worked nicely :D
Back to top
View user's profile Send private message
thomasa88
Tux's lil' helper
Tux's lil' helper


Joined: 13 Apr 2005
Posts: 143
Location: Sweden

PostPosted: Sun Apr 17, 2005 3:23 pm    Post subject: Reply with quote

I mounted a ntfs disc through samba and run the script.

When the tar file is about 2GB this error appears in the script and it starts bzipping.

Code:

./backup.sh: line 196: 24302 File size limit exceeded$tar_command_append


I suppose my tar isnt complete or? :?

Edit.

Seems like smbfs has problems with making tars over networks:
https://forums.gentoo.org/viewtopic-t-130241-highlight-tar+size+limit+exceeded.html
_________________
- thomasa88
Back to top
View user's profile Send private message
slydini
Tux's lil' helper
Tux's lil' helper


Joined: 30 Oct 2002
Posts: 129
Location: Virginia Beach, VA

PostPosted: Sun Apr 17, 2005 7:31 pm    Post subject: Reply with quote

Damn, I tried to restore and as it was restoring I noticed the leading slashes. So now I am lost as how to get the .tgz file to extract without writing over the currently running Live CD filesystem???? I can't chroot into anything because there is nothing there to chroot too. It's like a catch-22. It makes more sense to have backed up each main filesystem sepearately then use relative pathes so I could restore to the /mnt/gentoo and /mnt/gentoo/boot directory. See what I mean?

If there is another way to restore using the Live-CD and a separate Data CD I am all ears.

Here is the commands I used to restore from my data-CD after I repatitioned and formated:

Code:
mount /dev/hda5 /mnt/gentoo
mount /dev/hda3 /mnt/gentoo/boot
cd /mnt/gentoo
tar xzvpf /mnt/cdrom/host-stage4-16.04.2005-custom.tar.gz .
........After only 6.5 minutes
/.reiserfs_priv/
tar: .reiserfs_priv: CAnnot mkdir: Permission denied
/stage4.sh
/lost+found/
tar: Error exit delayed form previous errors


A backup isn't a backup unless you can restore it.
_________________
_______________________________________
Oh how I love Linux, especially on gentoo.


Last edited by slydini on Sun Apr 17, 2005 11:25 pm; edited 1 time in total
Back to top
View user's profile Send private message
BlinkEye
Veteran
Veteran


Joined: 21 Oct 2003
Posts: 1046
Location: Gentoo Forums

PostPosted: Sun Apr 17, 2005 8:48 pm    Post subject: Reply with quote

slydini wrote:
Damn, I tried to restore and as it was restoring I noticed the leading slashes. So now I am lost as how to get the .tgz file to extract without writing over the currently running Live CD filesystem???? I can't chroot into anything because there is nothing there to chroot too. It's like a catch-22. It makes more sense to have backed up each main filesystem sepearately then use relative pathes so I could restore to the /mnt/gentoo and /mnt/gentoo/boot directory. See what I mean?

no, i don't.

slydini wrote:
Here is the commands I used to restore from my data-CD after I repatitioned and formated:

Code:
mount /dev/hda5 /mnt/gentoo
mount /dev/hda3 /mnt/gentoo/boot
cd /mnt/gentoo
tar xzvpf /mnt/cdrom/host-stage4-16.04.2005-custom.tar.gz
........After only 6.5 minutes
/.reiserfs_priv/
tar: .reiserfs_priv: CAnnot mkdir: Permission denied
/stage4.sh
/lost+found/
tar: Error exit delayed form previous errors


A backup isn't a backup unless you can restore it.

well, sure it is. but you simply forgot that the above command extracts your stage4 to /mnt/cdrom. either copy your stage4 to /mnt/gentoo or use
Code:
tar xzvpf /mnt/cdrom/host-stage4-16.04.2005-custom.tar.gz -C /mnt/gentoo

which does extract your stage4 to /mnt/gentoo
_________________
Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick
Back to top
View user's profile Send private message
slydini
Tux's lil' helper
Tux's lil' helper


Joined: 30 Oct 2002
Posts: 129
Location: Virginia Beach, VA

PostPosted: Sun Apr 17, 2005 10:54 pm    Post subject: Reply with quote

I actually forgot to put the dot at the end of the tar command in my post to indicate the current directory. I did attempt to cp the tarball to /mnt/gentoo and then extract it but I got the same error. And just think about it a minute if you tar up your system using absolute pathes then / is the same root your running from the Live-CD!!!!!!! So absolute paths don't make any sense at all while booted to a Live-CD. The gentoo stage1-3 tarballs are done with relative paths which makes sense to me. However if there is another method for doing this I am ready to learn.

If anyone has the steps they use to restore using a regular gentoo-live-cd please post your steps. Creating the tarball with this script works great it's the restore steps that need to be clarified.

The error I am getting happens after about 6.5 minutes of extracted files scrolling up the screen then:
Code:
tar: .reiserfs_priv: Cannot mkdir; Permission denied


Here are the steps I am doing so far:
Code:
1. Boot off of Live-CD and repartition and create filesystems as necessary.
2. Reboot, use the option: gentoo docache
3. umount /mnt/cdrom
4. Remove the Live-CD and insert the CD with the backup info on it.
5. mount /dev/cdrom /mnt/cdrom
6. Restore MBR using: dd if=/mnt/cdrom/mbr.img of=/dev/hda bs=446 count=1
7. mount /dev/hda5 /mnt/gentoo
8. mkdir /mnt/gentoo/boot
9. mount /dev/hda3 /mnt/gentoo/boot
10. mount any other filesystems you have on your system.
11. cd /mnt/gentoo
12. tar xzvpf host-stage4-16.04.2005-custom.tar.gz .
13. cd /
14. umount /mnt/cdrom
15. remove backup cd
16. Reboot


Only trouble is it doesn't work for me.
_________________
_______________________________________
Oh how I love Linux, especially on gentoo.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3, 4 ... 16, 17, 18  Next
Page 3 of 18

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum