Saving downloadtime when downloading KDE by using xdelta

blackpenguin · n00b Joined: 09 Mar 2004 Posts: 43 Location: Germany

KDE mirrors release xdelta files - but portage does not use them.

If you dont know what xdelta-files are - these are much similar like diff-patches but for binary files.

So downloading xdeltas reduce the size of your downloads.
For example instead of downloading ~300 MB for the complete kde-sources (incl. kde-i18n-3.2.1.tar.bz2) you would download only ~50 MB xdelta-files.

I have written a small script, that does this for you now.

You can find the script here: http://linux01.gwdg.de/~nlissne/kdedelta.sh

Have a look into the script before starting it
(for example you can/should change the packages and the kde-mirror to be used for download...)

The script checks if you have the previous versions (referred as $KDE_OLD and $ARTS_OLD in the script) and if so it will download the xdelta file and use it together with the old version of the tar.bz2-file to create the new one.
If you dont have the old version in your $DISTDIR it just downloads the new version.

Just start it before upgrading your KDE.
Of course you should have installed xdelta to use it. (just emerge xdelta)

Script is prepared to download xdelta-files for arts-1.2.1 and kde-3.2.1 but can easily be changed for future versions.

I hope this is useful for someone.
Have fun.

jcc · Posted: Tue Mar 09, 2004 11:36 pm Post subject:

man, this is cute. I think I had to say a quick thank you :)

Redeeman · l33t Joined: 25 Sep 2003 Posts: 958 Location: Portugal

i have thought of stuff like this too, but never saw it before this.

X-Ryl669 · Tux's lil' helper Joined: 16 Feb 2004 Posts: 135

What a great proggy...

I will have a look at it before downloading any more tar.bz2...

Thanks
_________________
Think it twice, think it thrice, you never think enough...

alec · Apprentice Joined: 19 Apr 2002 Posts: 270 Location: Here

There's a program called deltup in portage that is suppose to add this functionality to portage (but automated), however it doesn't seem to ever have gotten off the ground.

rewt · n00b Joined: 19 Feb 2004 Posts: 58

Sounds cool. The only drawback I see is that you have to have the previous version to make use of it
Maybe it would be possible to store the deltas somewhere and apply them one after another if that would be faster?
In any case I'm sure this will save a lot of people huge amounts of time and bandwidth, nice job!
_________________
Because sometimes peace is another word for surrender... and secrets have a way of getting out

blackpenguin · n00b Joined: 09 Mar 2004 Posts: 43 Location: Germany

Good Idea, I have changed the script to do that.
It is probably not faster for the whole script, but useful on a
dialin-line to download all xdeltas first.

occam2003 · Posted: Sat Mar 13, 2004 3:25 am Post subject:

VERY good idea - pity I did not check this forum before starting the emerge -U world. Oh well, next time..

skimitar · n00b Joined: 30 Dec 2003 Posts: 20

Very useful. Made me get off my butt to upgrade to 3.2.1.

Just a note for people: you'll need the orginal kde and arts distfiles in /usr/portage/distfiles.

If you've deleted them to clear space, you'll need to restore the relevant bz2's from backup (assuming you've made one).
_________________
"Internet! Is that thing still around? " - HJS

ferringb · Retired Dev Joined: 03 Apr 2003 Posts: 357

To those interested, glep 25 is another attempt at getting distfile patching support into portage- the utility it's based on (diffball, which I've been writing) supports reconstructing files via xdelta patches.

Pretty much, assuming the glep were accepted (it's currently a draft) the existing kde xdelta patches could be used automatically. Give the glep a read if your interested- feedback would be quite appreciated.

mantis · Posted: Sun May 02, 2004 9:37 am Post subject:

I do hope this does get accepted.

Wish it were available at the moment.