Any option to disable premerge check?

lefsha

It's a new feature (for me). Sometimes it's useful sometimes not.
Obviously it's not really useful for emerge -e @world

It does make wrong assumptions and prevents parallel emerge.

Is it possible to turn it off/on optionally by using some flags or settings?

Thx
_________________
Lefsha

cboldt · Veteran Joined: 24 Aug 2005 Posts: 1046

Not sure what "check" you are looking to twiddle. If it's one of the possible system resources checks, those appear in individual ebuilds in the form of variables. I've bumped into these in `emerge -uDN @world` commands, resulting in the command failing, meaning nothing is built.

mike155 · Posted: Fri Nov 29, 2019 2:56 am Post subject:

Are you looking for

javeree · Guru Joined: 29 Jan 2006 Posts: 453

@mike155

I just tried that with
I_KNOW_WHAT_I_AM_DOING=1 emerge net-misc/r8168

which resulted in

lefsha · Posted: Sun Dec 01, 2019 1:08 am Post subject:

Hu · Administrator Joined: 06 Mar 2007 Posts: 21709

lefsha · Posted: Sun Dec 01, 2019 3:59 am Post subject:

krinn · Watchman Joined: 02 May 2003 Posts: 7470

Ionen · Developer Joined: 06 Dec 2018 Posts: 2727

If you don't agree, the I_KNOW_WHAT_I_AM_DOING is a perfectly fine solution anyway, just to make sure you're aware. Unfortunate if an ebuild ignores the flag but in those cases it often mean it'll 100% fail.

To answer the original question, if you _really_ want to force-disable them all or conditionally, you could use a /etc/portage/bashrc to set a dummy pkg_pretend() { true; } that does nothing (at least I think it should work, haven't tried), but well, do this at your own risks, could miss important messages too (I don't recommend making this a default, or using at all). Portage doesn't stop you from enforcing whatever you like.

lefsha · Posted: Sun Dec 01, 2019 3:02 pm Post subject:

lefsha · Posted: Sun Dec 01, 2019 3:20 pm Post subject:

NeddySeagoon · Posted: Sun Dec 01, 2019 4:21 pm Post subject:

lefsha,

You assume that everyone has the same partition layout as you do.
Everything is in one big root partition.

Many users don't do that. The build space check is valuable to them as they are likely to have /var/tmp/portage on its own partition, or even in tmpfs.
With SSDs and >=16G RAM being common, SSD users put /var/tmp/portage into tmpfs to save writes to the SSD that will never be read.

If builds didn't stop for these RAM and build space checks they would never be read until its too late.
You know they are there, why would you leave an overnight build until you have seen that all is well?

I agree that if a build fails, /var/tmp/portage will be full of debris, so further builds may not have enough space but plan for success.
I think that there is a FEATURE to tell portage to clean up after failed builds, so you may use that if you expect builds to fail.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

Ionen · Developer Joined: 06 Dec 2018 Posts: 2727

Yeah the feature is "fail-clean" I use it by default and manually set FEATURES=noclean when I need to inspect things.

Also, for large overnight parallel builds, there is an emerge option "--keep-going", so that when it fails you'll only have the failed package (and things that depend on it if it wasn't a simple rebuild) to build, while everything else will have kept going and be done.

NeddySeagoon · Posted: Sun Dec 01, 2019 4:39 pm Post subject:

Ionen,

--keep-going does not continue after a failed build space check.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

Ionen · Developer Joined: 06 Dec 2018 Posts: 2727

Yeah, I was thinking about normal build failures really. I'm having a hard time to see the pre-merge space check as an issue.

Hu · Administrator Joined: 06 Mar 2007 Posts: 21709

This check is useful for many users, though it is not perfect. Of the failure modes below, which do you want changed and what do you propose be done?

The package claims to need more space than it actually will need. Solution: file a bug to have the requirement adjusted. Document your configuration, since requirements can vary by what features you enable. Workaround: the requirement is probably not wrong by a large margin, so providing a little extra space will get past it anyway.
Downloading the source will bring you below the required amount, and the package does not detect this. As Neddy notes, many people do not download sources to the same volume that will be a build area, so this failure mode does not even apply to those people.
The check runs too early, and the space is used up by other processes in the interim. This is always a risk, and can only be mitigated by the administrator ensuring that nothing else can consume space on the build volume. Even if Portage were perfect here, you would still get in trouble if completely unrelated processes (say, a runaway debug log - we've had people report gigabytes of junk logs from broken programs) exhausted your space. In the worst case, the offender might wait until just after the last time Portage checks before it exhausts your space. This check should be done as early as practical, because you want to let the user walk away as soon as possible, and you also want the user to see the error report before leaving.

I have run Gentoo since before my join date. I like to think I know what I am doing, and yet I have used that override only very rarely, when I wanted to use a very experimental package that, by design, refused to build without that because it was not considered ready for general use.

If you have a specific example of a case where Portage balked, and the package worked with the override and no other changes, I'd like to see a description of what happened and how exactly Portage was wrong in that case.

NeddySeagoon · Posted: Sun Dec 01, 2019 5:21 pm Post subject:

Hu,

In my experience, building on space constrained systems like Raspberry Pis ... from the original 256Mb Pi1B up to the 4G Pi4 the build space estimates are all pretty good.
There is no good reason to ignore them.

The RAM constraints vary. Lots of things build (eventually) with less RAM than advertised.
They are driven into swapping first and the build times go through the roof but on a Pi with 1G RAM, there is nowhere to go.
My conclusion from doing silly things on Pis is that if the RAM space requirement is not met, it might build, eventually but it will take a long time to find out.
It would not be reasonable to expect devs to set the RAM limit at the point of failure.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

John R. Graham · Posted: Sun Dec 01, 2019 6:11 pm Post subject:

The main objection that lefsha seems to be making is that he's started an emerge and immediately walked away, only to have it fail early, in his estimation, for no good reason. The problem with his perceived solution is that, in my estimation, there are so many other things that an emerge can fail early for that's it's a really good idea to stick around to see if the build phase actually starts. The pre-merge checks take very little time to run; also, compared to the build time, the dependency resolution takes a relatively short amount of time as well.

The one thing that I do agree with lefsha on is that dependency resolution is slower today than it was in Ye Olde Days™. The counterpoint is that Portage's dependency resolver does much more than it used to, solving things that at one time it couldn't, resulting in the virtual retirement of previously necessary fix-up tools like revdep-rebuild and python-updater. (I'm not sure about perl-cleaner, but I haven't had to use it in a very long time.) So, sure, it would be nice if it were faster, but, especially on modern hardware, it's not too bad, and it's a Good Thing™ to wait for the the build phase to start before walking away.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.

NeddySeagoon · Posted: Sun Dec 01, 2019 6:29 pm Post subject:

John R. Graham,

John R. Graham · Posted: Sun Dec 01, 2019 6:38 pm Post subject:

Hmm. For what? Mine are never that slow. My big gun workstation is a 24-core dual CPU Xeon running at 3.5GHz but only (!) 32 GiB of RAM with Seagate Barracuda backing store. Here's one exemplar from my home server, though, which is a much more modest machine:

lefsha · Posted: Sun Dec 01, 2019 9:41 pm Post subject:

Neddy,

May be you can help here. I would appreciate it.

I do really feel stupid, because I can't understand why people don't see an obvious problem
with that interruption.

Read again carefully and tell me where I am wrong.

The first story:

1. We start "emerge palemoon"
2. Premerge check will calculate space available. Let it be 10Gb.
3. It will make a decision, whether continue or not. Palemoon need 9Gb
to be compiled.
4. 10Gb > 9Gb so it goes well.
5. Emerge continues and either me or emerge itself because of additional dependencies
will download let say 5Gb. May be it was me, who downloaded a db of all customers of Bank of America...
6. Then it takes 5+ hours till we get to palemoon compilation time.
7. Now we have 10-5-X << 9Gb left. It is not enough to compile palemoon successfully.

The question: What was the purpose to check available space and let compilation go?

The second story:
1. My emerge failed, because not enough space. I do have 10-5 = 5 Gb left.
2. I started "emerge palemoon" again.
3. Premerge check told me - I don't have enough space, BUT didn't interrupt the process.
4. Now I hear the FBI is ringing into my door. And I'm deciding to clean up the space by removing
db of BoA customers, just to make it easier for emerge...
5. Palemoon compiled correctly, because 10-9>0

Please find a mistake in that story.

What we can learn from it? - Give a notice ie provide more information is always a good thing.
Interrupting a process is not. First it doesn't help like in the first story, second it prevents
from successful build in the second one.

What exactly is wrong in these words? It's like 2+2.

lefsha · Posted: Sun Dec 01, 2019 9:58 pm Post subject:

lefsha · Posted: Sun Dec 01, 2019 10:03 pm Post subject:

lefsha · Posted: Sun Dec 01, 2019 10:10 pm Post subject:

NeddySeagoon · Posted: Sun Dec 01, 2019 10:16 pm Post subject:

lefsha,

Your logic is not clear because your underlying assumptions are not stated.
If you have a single partition for Gentoo then what you say is correct.
All processes compete for the same free disk space.

The more general case for your unwritten assumption is that build space and distfiles are on the same partition.

If you have several partitions that may no longer be true. Lets take my install as a worked example.