The best way to update Gentoo without network?

[derekdai]

Hi, I have some MIPS based devices run in places which may or may not have network.

So if users want to update those devices, he or she might have to download pre-built binary packages to memory sticks, HDDs or burn a CD ROM.

I love Gentoo and Portage. It is a powerful environment to do cross development. Once I finished my software, the next step is to package and distribute it and its dependencies in pre-built binary. But if .ebuilds (and scripts been used to build source code) need to be maintained on devices, that may consume lot of storage space and in fact, those files are meaningless to end users. So, is it possible to distribute binary package without .ebuild but keep version and dependence information? Or any other suggestions for my problem?

[BradN]

Edit: re-read your post and saw you intend to use binary packages. This isn't a problem but you'll want to make sure your host system compiling them is an actual mips box with the same make.conf CPU configuration (and same libraries where there are incompatible versions available). Cross compiling "should" work, but there's so many times it doesn't that I hesitate to recommend it for anything but the simplest purposes. The quickpkg command is your friend - it will build binary packages on your host system and put them I believe in /usr/portage/packages - these can be copied to the target whatever way you want and installed with emerge -K

---- Normal emerge without network (probably not what you want):

emerge -p --fetchonly can give you a list of downloads you need (plus alternate URL's), between that and downloading a fresh portage tarball when needed should be able to get you going. Beware of use flags not being set the way you want, as this can pull in extra dependencies if you have to change them after the download. If there's packages you're not sure about what version you want (compatibility or bugs, etc), you might want to download a couple versions just in case.

Suggest having a decent size external hard drive that you dump all needed distfiles to, as this can reduce extra downloading if you've got a lot of stuff in one spot already.

On your target system you could either configure it to use that external drive as the distfiles location, or copy them into the normal distfiles spot. Be aware that distfiles is normally in /usr/portage and that location could easily be deleted on accident when you're unpacking a new portage tarball - so I suggest changing the location to say, /var/distfiles if you plan on keeping package sources on the local system.

[few]

If you ship binaries, you don't need the ebuilds. Make sure that the packages on the end users system are always in a consistent state (exactly the version of dependencies you built a package against). End users would just need to set PKGDIR to the cdrom/whatever. The --rebuild-binaries option should help you with ensuring the consistency of the end users system (assuming the binaries you ship are consistent).

[BradN]

Thanks for that tip, few. I wasn't sure if portage still wanted updated ebuilds to install binary packages or not, but if it doesn't need them, that simplifies a bit.


Another approach that might be beneficial if you maintain a bunch of identical systems, is having one designated as the master build system, and cloning the whole thing to the client machines (except for /etc and /home and parts of /var that are unique to each client). Another possibility that becomes an option then is using squashfs for /usr and cutting down disk footprint a lot, as well as speeding up updates by having most of the system in one file (rsync is nice for performing the updates, and is network capable should you network the machines eventually). This is a more advanced approach so if you're not comfortable enough with setting it up, just stick to either normal systems or cloning without squashfs.

The cloning approach also allows filtering out extra files that may not be needed at runtime, such as portage, *.a *.la files, headers, etc. By combining this with squashfs, you make a system that easily fits within 1GB storage (even 512MB is doable if not much user storage is needed) and might let you cut back on hard disks in favor of cheap and low power USB drives or small solid state IDE. The throughput loss by decompression is usually made up for by the improved locality of data (as well as more efficient cache usage) and less physical IO needed.

As a test case, I set up a system like this on an old pentium 1 class system with a 1GB hard drive and 128MB RAM, and surprisingly even with such a slow CPU, the squashfs decompression was not a significant bottleneck, so modern MIPS boxes should do even better.

Anyway, it's just something to consider

[derekdai]

My devices might have different versions on it.

I mean those devices will not been updated at the same time, some might update daily, some might update weekly or monthly...and manually, I can't control it.

So, the versions and dependencies information are important to my situation, if drop all these .ebuilds, can we maintain these information with just /usr/portage/packages/*.tbz2? Thanks for your reply, that's very helpful!

[BradN]

To answer your last question, if you only intend to install binary packages, you probably don't need the portage tree installed/updated on the clients at all - all the installed programs are tracked through /var/db/portage or /var/portage or something like that - /usr/portage wouldn't be needed.


I think with all the random breakages that happen with gentoo from time to time, you'll want to test each new configuration somewhat thoroughly before you put it in use, and doing a emerge -e world after major upgrades might save you more trouble than the extra time it takes (but depending how slow those mips boxes are it could take a long time)

To make it simple, even if your machines don't all run the same software, you should try to confine them to certain versions or configurations that you've tested, and don't do partial updates.

So you might have 3 machines running config/version A, a couple with version B, and 5 with version C. But you don't want any running version B half way updated to version C, this can cause extra problems that might not happen with one or the other, unless you spend the time to test everything and make sure it works right.

Sorry I can't really offer better advice than that, planning systems for deployment with gentoo is often very purpose-specific

[few]

[derekdai]

End user won't be allowed to install package by himself.

And because our MIPS CPUs run only about 600MHz, I think it might not be a good idea to build packages on it.

If I removed /usr/portage from my devices, can emerge still work? How do I emerge a package with .tbz2? Just put it like /usr/portage/packages/XXX/OOO/OOO.tbz2?

As few mentioned, If a package need extra user interactions to finish merge, can I do it first before I distribute it?

Sorry for extra two stupid questions

[few]

[Veldrin]

basically there are 2 possibilities to used binary packages in gentoo.

The 'normal and correct' way is to have a portage tree, and use emerge -k <pkg>.
Your portage tree has to be in sync with the built binaries in order to work. (e.g emerge --sync; build packages; deploy packages /w portage tree)

The second, more experimental way is to just explode those packages into /. (i.e cd /; tar xvjf <pkg>.tbz)
The downside here, is that you lose any control over the installed packages, and make upgrading, or cleaning you system next to impossible. but you can create a slim system.


About the interaction part...
everything is scriptable. something like the following could work. (assuming that all packages are in /usr/portage/packages, and have tbz2 suffix.)

[few]

[Veldrin]

I assumed, that the OP wanted to get rid of portage on the final system.
If that is not the case, then ignore my previous post...

[few]

[derekdai]

Thanks guys, you gave great advises for me.

So, if prebuilt .tbz2 have all package information, I can install it without ebuilds, right? Just remember to keep /etc/make.profile point to right place.

If not, maybe I should:
1. trim and maintain a portage tree that fits my devices
2. deploy the tree and .tbz2 to a media
3. plug to device and mount to, say /usr/portage
4. emerge world
5. run some scripts to do further configurations

[few]

[derekdai]

Yes, you guys are right! I removed all ebuilds and left package directory and .tbz2 there, then I ran "emerge -eK system" to re-emerge all system package, it simply works! Portage amazing!

Further question, if I want to upgrade my devices with pre-built binaries, do I need portage tree? Or just new .tbz2?

[few]

[derekdai]

Thanks guys, especially few.