View previous topic :: View next topic |
Author |
Message |
frozenpenguin n00b
Joined: 04 Feb 2024 Posts: 33
|
Posted: Sun Mar 31, 2024 4:44 am Post subject: Changing the default directory for compiled binaries |
|
|
I am trying to emerge a package and want the emerged package to be stored on a USB. However, I do not know which variable to change in portage to make portage behave as I wish. I also do not know how to let the terminal know the commands associated with the package if the package isn't in the default package directory. Any assistance will be appreciated. |
|
Back to top |
|
|
Josef.95 Advocate
Joined: 03 Sep 2007 Posts: 4664 Location: Germany
|
Posted: Sun Mar 31, 2024 6:40 am Post subject: |
|
|
The variable for this is PKGDIR
this is documented in "man make.conf" |
|
Back to top |
|
|
frozenpenguin n00b
Joined: 04 Feb 2024 Posts: 33
|
Posted: Sun Mar 31, 2024 8:45 am Post subject: |
|
|
From what I understand, PKGDIR is where binary packages downloaded off the internet are stored. I am trying to find the default directory where the packages compiled from source on my machine are stored. |
|
Back to top |
|
|
flexibeast Guru
Joined: 04 Apr 2022 Posts: 425 Location: Naarm/Melbourne, Australia
|
Posted: Sun Mar 31, 2024 9:15 am Post subject: |
|
|
The variables used during the build-and-merge process are documented in ebuild(5) man page and in the devguide. There's also the ebuild(1) man page which describes the various phases involved.
Depending on exactly what you want on the USB drive (a single binary package? a directory-and-file-tree, given that a single package might install more than one binary, and possibly various libraries and other supporting files as well?), emerge(1)'s `--buildpkg*` options, and the the quickpkg(1) program, might be relevant.
If none of the above helps, please provide more information about exactly what you require. |
|
Back to top |
|
|
Hu Administrator
Joined: 06 Mar 2007 Posts: 22578
|
Posted: Sun Mar 31, 2024 3:10 pm Post subject: Re: Changing the default directory for compiled binaries |
|
|
frozenpenguin wrote: | I am trying to emerge a package and want the emerged package to be stored on a USB. However, I do not know which variable to change in portage to make portage behave as I wish. I also do not know how to let the terminal know the commands associated with the package if the package isn't in the default package directory. | The third sentence leads me to believe you are trying to place the installed program at a prefix other than the typical /usr (from-source) / /opt (prebuilt). The answer to your problem there is to modify your shell's search path, traditionally $PATH, to include the directory of interest. However, that is only useful after you change the prefix, for which there is no general solution if you want Portage to change the path only for this one package. |
|
Back to top |
|
|
frozenpenguin n00b
Joined: 04 Feb 2024 Posts: 33
|
Posted: Sun Apr 07, 2024 1:04 pm Post subject: |
|
|
My idea is to install a package on a USB. When the USB is mounted, the system should treat packages installed on the USB as if the packages were installed on the system's partition. Only the packages binaries and dependencies which are required by those but not by any packages on the system should be installed there, as I do not want packages I periodically use installed on my computer as I wish to prevent storage issues on my system before they arise. |
|
Back to top |
|
|
eschwartz Developer
Joined: 29 Oct 2023 Posts: 213
|
Posted: Sun Apr 07, 2024 2:25 pm Post subject: |
|
|
frozenpenguin wrote: | My idea is to install a package on a USB. When the USB is mounted, the system should treat packages installed on the USB as if the packages were installed on the system's partition. Only the packages binaries and dependencies which are required by those but not by any packages on the system should be installed there, as I do not want packages I periodically use installed on my computer as I wish to prevent storage issues on my system before they arise. |
This will not work because software tends to embed the filesystem paths of the resources they use. Also, shared libraries have to be integrated into the system loader.
If you wish for packages to be installed to a location other than /usr or /bin or /lib, such as a USB drive, then you need to compile the packages specific for that purpose, including nontrivial approaches to ensure they run portably. It is much the same requirements as imposed by Windows application bundles (in fact, Windows application bundles already have to do this, thus *they work by default on a USB drive*).
I think you have two options:
- don't borrow trouble from the future. Simply install the programs you need, and deal with storage issues *if* they ever come up, which they may not.
- buy a bigger hard drive |
|
Back to top |
|
|
Hu Administrator
Joined: 06 Mar 2007 Posts: 22578
|
Posted: Sun Apr 07, 2024 2:45 pm Post subject: |
|
|
frozenpenguin: what are the specifications for the system where you want to do this? How much storage do you think you need for these programs you intend to relocate to USB? I concur with eschwartz that at this stage, it looks like you are trying to avoid a problem that may never come to pass, and are definitely setting yourself up for additional hassle in the attempt. |
|
Back to top |
|
|
pingtoo Veteran
Joined: 10 Sep 2021 Posts: 1214 Location: Richmond Hill, Canada
|
Posted: Sun Apr 07, 2024 3:01 pm Post subject: |
|
|
frozenpenguin wrote: | My idea is to install a package on a USB. When the USB is mounted, the system should treat packages installed on the USB as if the packages were installed on the system's partition. Only the packages binaries and dependencies which are required by those but not by any packages on the system should be installed there, as I do not want packages I periodically use installed on my computer as I wish to prevent storage issues on my system before they arise. |
What you have in mind is possible, however it is advance system admin topic. Have you thought about how you going to maintain such setup? For example when you perform a update of some system level (i.e. a xz library) and yet the USB was NOT connected at the time of update but later you wish to use a program that is on the USB that require use of the older version of system library, Do you want to perform another update just to use the program on USB?
Having said that one possible setup for your idea is using a union filesystem (OverlayFS), You create a layer on USB and map it into system hierarchy. This idea does not solve the system admin issue I made above, it just simplify the installation procedure.
Gentoo is good for customisation, however you also need to take the maintenance of the customisation info account. |
|
Back to top |
|
|
pjp Administrator
Joined: 16 Apr 2002 Posts: 20476
|
Posted: Sun Apr 07, 2024 3:50 pm Post subject: |
|
|
Isn't using the USB simply a sneakernet method?
Am I missing something? Why would that be meaningfully different than downloading them from a network connection? _________________ Quis separabit? Quo animo? |
|
Back to top |
|
|
Hu Administrator
Joined: 06 Mar 2007 Posts: 22578
|
Posted: Sun Apr 07, 2024 6:20 pm Post subject: |
|
|
As I understand the thread, OP is not merely using a USB disk as an alternative to downloading the source files over an Ethernet/Wifi connection. OP wants to use the USB as a conditionally present seamlessly available filesystem hosting finished runnable programs. For example, with USB absent, the system can only run a lightweight desktop, like Xfce4. Plug in the USB, and suddenly you can also choose GNOME or KDE Plasma as desktop options. |
|
Back to top |
|
|
frozenpenguin n00b
Joined: 04 Feb 2024 Posts: 33
|
Posted: Mon Apr 08, 2024 4:36 am Post subject: |
|
|
My current setup on my computer is around 85GB for linux and around 383GB for windows. Now, I could add more space for linux except windows isn't willing to give any more space up from its partition using its disk management tool. I also don't want to delete windows as I may need it again in the future.
As I only want to install packages I rarely use, packages on the USB will only be updated when their use is required. The reason these packages are rarely used is because I never use these packages unless it is required for certain features(such as some sites requiring use of a certain browsers for certain functionality). |
|
Back to top |
|
|
e8root Tux's lil' helper
Joined: 09 Feb 2024 Posts: 94
|
Posted: Mon Apr 08, 2024 3:05 pm Post subject: |
|
|
To allow reducing size of Win10/11 install you need to disable pagefile and hibernation, restart and then defragment your drive.
I would also recommend using folder compression on programs and files you don't really use all that often. Of course already compressed files do not gain much from compressing them again but on average gain can be quite good for typical programs. Most gains are on things like large number of small text files.
As for what you want to achieve in OP I am not sure how to achieve it, or at least in automated way as manually you could probably move some files and perhaps use LD_LIBRARY_PATH to move some .so files to other locations. Doing it manually can quickly lead to issues if you don't know exact dependencies. Automating this... should be doable but sounds like a lot of effort hacking Portage.
Probably (very...) better solution would be to use something like Flatpak or Appimage packages. Especially the latter one you should be able to just put on USB and run as portable programs. Flatpaks I am not sure but should be doable. There are also Snap packages but we neither use nor talk about Snap packages ; ) But seriously it seems you need SystemD which you might have or not and I personally am lest interrested in these even if I myself do use SystemD.
There are also other more source-based solutions like Nix and Guix package managers. The latter one is not free to use anything not free so might not have all programs you would want. Both work in similar way and allow you to build your packages. By the the looks of it it should be possible to achieve having Nix/Guix programs in separate mounted directory without needing anything on your base system. Downside it you would then need to learn new build system and there is some disk space overhead as you will need all the build tools like compilers.
Lastly it is possible to chroot in to existing Linux install and run applications from that. It can be Gentoo and in this case its like separate Gentoo environment which can be system which might not even be bootable/usable standalone but could easily be made as standalone bootable system or some other distro. It requires some tinkering if you want to run GUI apps or whole X11/Wayland servers but it is doable. It is of course abusing chroot and issues are expected, especially if you run something like Ubuntu. There for example Snaps don't work and cannot be made to work. Otherwise Especially for Gentoo itself it might be a good idea to have such system as it can be useful for testing, troubleshooting and such.
Then there are other solutions like LXC or docker.
Depending on your specific needs and compatibility either of those or others which I mentioned and didn't mention can be usable.
Most sane and "just works" solution is however something like Appimage.
Maybe create Appimage using Gentoo? Download program you need from trusted source and just run it? Lots of different ways to do it... _________________ Unix Wars - Episode V: AT&T Strikes Back |
|
Back to top |
|
|
Genone Retired Dev
Joined: 14 Mar 2003 Posts: 9604 Location: beyond the rim
|
Posted: Mon Apr 08, 2024 3:25 pm Post subject: |
|
|
I'd first check why you even need more than 50 GB for your linux install. Most packages only need a couple megabytes at most, so redirecting them to a USB drive wouldn't really help with space issues in the first place.
Try running
Code: | du -sh /* /usr/* /var/* |
to check which filesystem locations take up the most space. I suspect there will be some candidates for cleaning up. |
|
Back to top |
|
|
logrusx Advocate
Joined: 22 Feb 2018 Posts: 2366
|
Posted: Mon Apr 08, 2024 3:57 pm Post subject: |
|
|
Genone wrote: | I'd first check why you even need more than 50 GB for your linux install. |
What about some everyday user data. The time I lived with 85Gb is not that far. It's not enough for a desktop system. I was constantly cleaning up tmp and dist files what not to find space to decompress something.
Best Regards,
Georgi |
|
Back to top |
|
|
Zucca Moderator
Joined: 14 Jun 2007 Posts: 3678 Location: Rasi, Finland
|
Posted: Mon Apr 08, 2024 7:03 pm Post subject: |
|
|
I'd like to add to @Genone's comment that if you want to find "where all the stuff is" then try sys-fs/ncdu. _________________ ..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--
Quote: | I am NaN! I am a man! |
|
|
Back to top |
|
|
NeddySeagoon Administrator
Joined: 05 Jul 2003 Posts: 54572 Location: 56N 3W
|
Posted: Mon Apr 08, 2024 7:24 pm Post subject: |
|
|
Put distfiles, packages and /usr/src onto USB.
They are only needed at update time.
Optionally, ./repos can go to USB too.
If you must have ::gentoo available all the time, use a squashfs snapshot to save space.
Everything to run Gentoo in always available. The bits required for maintenance can be unplugged when not needed. _________________ Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
Back to top |
|
|
Genone Retired Dev
Joined: 14 Mar 2003 Posts: 9604 Location: beyond the rim
|
Posted: Tue Apr 09, 2024 10:28 am Post subject: |
|
|
logrusx wrote: | Genone wrote: | I'd first check why you even need more than 50 GB for your linux install. |
What about some everyday user data. The time I lived with 85Gb is not that far. It's not enough for a desktop system. I was constantly cleaning up tmp and dist files what not to find space to decompress something.
Best Regards,
Georgi |
Was kind of my point that most space is likely taken up by user data, downloaded files, cache data and so on. If you're short on space with a >50GB install, redirecting a couple hundred megabytes of installed packages to another storage device is at best a short-lived temporary workaround. Cleaning up or outsourcing distfiles, kernel sources and build artifacts will likely yield much bigger savings.
Only exception would be if those redirected packages include behemoths that actually install gigabytes of data (e.g. some games). But then it might be worth checking their install locations and maybe simply mount /opt or whatever on USB.
TL;DR: Before working on a (perceived) solution, identify the cause of the problem. |
|
Back to top |
|
|
frozenpenguin n00b
Joined: 04 Feb 2024 Posts: 33
|
Posted: Thu Apr 11, 2024 1:20 pm Post subject: |
|
|
So I got the source files on my flash drive, however when I try to install the package I get this:
Code: | Emerging (1 of 1) www-client/chromium-120.0.6099.129::gentoo
bash: line 1: /usb/dist/.__portage_test_write__: Permission denied
bash: line 1: /usb/dist/.__portage_test_write__: Permission denied
!!! No write access to '/usb/dist'
bash: line 1: /usb/dist/.__portage_test_write__: Permission denied
!!! No write access to '/usb/dist'
!!! File .layout.conf.distfiles.gentoo.org isn't fetched but unable to get it.
!!! File chromium-120.0.6099.129.tar.xz isn't fetched but unable to get it.
* Fetch failed for 'www-client/chromium-120.0.6099.129', Log file:
* '/var/tmp/portage/www-client/chromium-120.0.6099.129/temp/build.log'
>>> Failed to emerge www-client/chromium-120.0.6099.129, Log file: |
The flash drive's filesystem is exfat and has chromium source files compressed in the .zip format. These files were downloaded off the chromium github mirror without using git clone as getting the source via portage isn't possible as google is blocked where I currently am. They were gotten from my phone(as it has a vpn) then moved to the flash drive. |
|
Back to top |
|
|
pingtoo Veteran
Joined: 10 Sep 2021 Posts: 1214 Location: Richmond Hill, Canada
|
Posted: Thu Apr 11, 2024 1:32 pm Post subject: |
|
|
frozenpenguin wrote: | So I got the source files on my flash drive, however when I try to install the package I get this:
Code: | Emerging (1 of 1) www-client/chromium-120.0.6099.129::gentoo
bash: line 1: /usb/dist/.__portage_test_write__: Permission denied
bash: line 1: /usb/dist/.__portage_test_write__: Permission denied
!!! No write access to '/usb/dist'
bash: line 1: /usb/dist/.__portage_test_write__: Permission denied
!!! No write access to '/usb/dist'
!!! File .layout.conf.distfiles.gentoo.org isn't fetched but unable to get it.
!!! File chromium-120.0.6099.129.tar.xz isn't fetched but unable to get it.
* Fetch failed for 'www-client/chromium-120.0.6099.129', Log file:
* '/var/tmp/portage/www-client/chromium-120.0.6099.129/temp/build.log'
>>> Failed to emerge www-client/chromium-120.0.6099.129, Log file: |
The flash drive's filesystem is exfat and has chromium source files compressed in the .zip format. These files were downloaded off the chromium github mirror without using git clone as getting the source via portage isn't possible as google is blocked where I currently am. They were gotten from my phone(as it has a vpn) then moved to the flash drive. |
How is this relate to the topic you are talking about? |
|
Back to top |
|
|
Genone Retired Dev
Joined: 14 Mar 2003 Posts: 9604 Location: beyond the rim
|
Posted: Fri Apr 12, 2024 10:07 am Post subject: |
|
|
frozenpenguin wrote: | The flash drive's filesystem is exfat |
Exfat doesn't support the posix ACL model, so it isn't really suited for use under Linux for anything other than storing user data. The error you see is because under Linux FAT filesystems by default can only be modified by root or the specific user account that mounted the filesystem.
If you want to use a USB drive to store anything but user data, you'll have to use a posix compatible filesystem (like ext4). But of course then the stick will no longer be accessible by Windows without special software. |
|
Back to top |
|
|
frozenpenguin n00b
Joined: 04 Feb 2024 Posts: 33
|
Posted: Mon Apr 15, 2024 11:42 pm Post subject: |
|
|
Quote: | How is this relate to the topic you are talking about? |
I just thought that it would be relevant as as someone suggested putting source on the flash drive. I'll put it in a new thread next time.
Quote: | Exfat doesn't support the posix ACL model, so it isn't really suited for use under Linux for anything other than storing user data. The error you see is because under Linux FAT filesystems by default can only be modified by root or the specific user account that mounted the filesystem.
If you want to use a USB drive to store anything but user data, you'll have to use a posix compatible filesystem (like ext4). But of course then the stick will no longer be accessible by Windows without special software. |
I mounted the flash drive as root so from this, I take it to mean that I can use it without problems as root. However, I get this problem when I try to emerge a package that has the DISTDIR variable set to point to the flash drive.
On another note, I still wish to know how to change the default directory or know where it is in case [/quote]I want packages to run on an embedded device where disk space is limited. |
|
Back to top |
|
|
Hu Administrator
Joined: 06 Mar 2007 Posts: 22578
|
Posted: Tue Apr 16, 2024 1:00 am Post subject: |
|
|
You originally asked to use a USB drive. I, and likely others, took this to mean a removable block storage device backed by either an SSD or a mechanical drive, not a USB flash stick. Regardless, I (and, again, likely others) assumed you would use a proper POSIX filesystem on this storage device. Whether you try to store distfiles or a compiled package on the external device, you will cause yourself considerable extra hassle if you use a filesystem that cannot represent standard POSIX features. ExFAT and NTFS are the most well known unsuitable filesystems, but there might be others.
As I wrote at the start of the thread, there is no generalized mechanism to change the prefix for individual packages to be other than the Portage default. The Gentoo Prefix project can relocate the entire resulting Gentoo system under a prefix, but that's not what you requested.
The general advice for dealing with systems that have limited internal storage is to use existing functionality to trim down which files the package installs, such as by eliminating unnecessary translations, stripping debug symbols, discarding documentation, and so on. If storage is very limited, use ROOT to install a non-self-sufficient system. Excluding compilation tools can save you substantial space. |
|
Back to top |
|
|
Genone Retired Dev
Joined: 14 Mar 2003 Posts: 9604 Location: beyond the rim
|
Posted: Tue Apr 16, 2024 8:04 am Post subject: |
|
|
frozenpenguin wrote: | I mounted the flash drive as root so from this, I take it to mean that I can use it without problems as root. However, I get this problem when I try to emerge a package that has the DISTDIR variable set to point to the flash drive. |
Yes, you as root can use it. But even if you launch emerge as root, it will by default switch to another user account and therefore can't access the root-only filesystem. Also system calls like chown (to change the owner of a file) can't work on a FAT file system, the file ownership instead is determined by the mount options. That is not compatible with portage (and most other Linux/Unix applications that make use of ACLs in some way). My previous statement may have been a bit unclear regarding that.
Quote: | On another note, I still wish to know how to change the default directory or know where it is in case I want packages to run on an embedded device where disk space is limited. |
There is no such thing as a "default install directory". I assume you get that concept from windows and its C:\Program Files\<appname> pattern. This doesn't apply to Linux. There are features like prefix or chroot, but they work quite different than "install location" on Windows.
In practice most package files will end up somewhere in /usr/, but that is not something you want to mess with as you can easily break your system doing so. |
|
Back to top |
|
|
|