View previous topic :: View next topic |
Author |
Message |
rzdndr Tux's lil' helper
![Tux's lil' helper Tux's lil' helper](/images/ranks/rank_rect_1.gif)
Joined: 26 Jul 2024 Posts: 77
|
Posted: Tue Jul 30, 2024 8:59 am Post subject: Resuming emerge |
|
|
Hello,
I have read some articles about resuming an emerge that was stopped with (Ctrl+C). But none worked.
I am installing a package, and Ctrl+C'd it to continue later. When I logged into the system and have done emerge --resume, it started with some other package. Since after a Ctrl+C, the object files (or whatever files are generated during a build) would not be deleted, wouldn't it be simple for emerge to restart where it left off? I thought that will be the case with emerge --resume.
What is the best way to resume an emerge when the emerge is stopped to continue later?
Regards |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
szatox Advocate
![Advocate Advocate](/images/ranks/rank-G-1-advocate.gif)
Joined: 27 Aug 2013 Posts: 3329
|
Posted: Tue Jul 30, 2024 9:34 am Post subject: |
|
|
Emerge --resume picks up the last job plan and continues from where it left of.
It makes sure to start each task from a clean state though, so it will not continue an interrupted compilation.
If you want to continue working on a particular task (e.g. manually patch a bug which breaks your build), you can do that with a lower-level interface: ebuild
Ebuild knows the correct order of actions during build process and is smart enough to know which steps have been already completed, so you can just call ebuild merge <path to ebuild> and it will unpack sources, prepare them, patch, build and whatever else I missed, all in one go.
Once this is done, you might try running emerge --resume --skipfirst to remove that item from the list before continuing. _________________ Make Computing Fun Again |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
rzdndr Tux's lil' helper
![Tux's lil' helper Tux's lil' helper](/images/ranks/rank_rect_1.gif)
Joined: 26 Jul 2024 Posts: 77
|
Posted: Tue Jul 30, 2024 9:50 am Post subject: |
|
|
szatox wrote: | Emerge --resume picks up the last job plan and continues from where it left of.
It makes sure to start each task from a clean state though, so it will not continue an interrupted compilation.
If you want to continue working on a particular task (e.g. manually patch a bug which breaks your build), you can do that with a lower-level interface: ebuild
Ebuild knows the correct order of actions during build process and is smart enough to know which steps have been already completed, so you can just call ebuild merge <path to ebuild> and it will unpack sources, prepare them, patch, build and whatever else I missed, all in one go.
Once this is done, you might try running emerge --resume --skipfirst to remove that item from the list before continuing. |
That would mean that I could use ebuild merge <ebuild file> to start the package merge. When I need to stop, can Ctrl+C at any point in the compile phase. Then shutdown the system. After some time, can startup the system, do an ebuild compile <ebuild file> to resume where it left off.
Would that be correct?
I am trying to build libreoffice, and wanted to stop the build at a point and continue tomorrow. Would the process above be possible. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
szatox Advocate
![Advocate Advocate](/images/ranks/rank-G-1-advocate.gif)
Joined: 27 Aug 2013 Posts: 3329
|
Posted: Tue Jul 30, 2024 10:03 am Post subject: |
|
|
Honestly I never tried that, but it should work.
A lot of the work compilers do can be (and is) cached on disk and big programs are made of many files that can be processed independently.
Which BTW reminds me about ccache some people use to speed up rebuilds. It could also be a good way for you to escape the need for manual intervention with ebuild. _________________ Make Computing Fun Again |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
logrusx Advocate
![Advocate Advocate](/images/ranks/rank-G-1-advocate.gif)
![](images/avatars/176594205966916859b5145.jpg)
Joined: 22 Feb 2018 Posts: 2162
|
Posted: Tue Jul 30, 2024 10:16 am Post subject: |
|
|
szatox wrote: | If you want to continue working on a particular task (e.g. manually patch a bug which breaks your build) |
That's not only not necessary at all but afterwards if one runs emerge --resume it'll rebuild that same package as it is part of the job plan it resumes.
You can resume terminated builds with
Code: | FEATURES="keepwork" emerge --resume |
or
Code: | FEATURES="keepwork" emerge package |
You have to take care to clean up PORTAGE_TEMPDIR manually.
You can wait till the interrupted package is finished, terminate emerge, cleanup and resume again without keepwork.
Be warned keepwork may not work as expected or may not work at all.
Best Regards,
Georgi |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
rzdndr Tux's lil' helper
![Tux's lil' helper Tux's lil' helper](/images/ranks/rank_rect_1.gif)
Joined: 26 Jul 2024 Posts: 77
|
Posted: Tue Jul 30, 2024 10:38 am Post subject: |
|
|
szatox wrote: | Honestly I never tried that, but it should work.
A lot of the work compilers do can be (and is) cached on disk and big programs are made of many files that can be processed independently.
Which BTW reminds me about ccache some people use to speed up rebuilds. It could also be a good way for you to escape the need for manual intervention with ebuild. |
I have done a quick experiment with evince. You can actually use the low-level ebuild tool when you have to break up the emerge at some point. You need to remember where to have broken the build, what package, and what phase (compile, merge, etc). ebuild will pick up where it left off and will continue compiling the sources, leaving out the sources that it create object files for.
The only thing is that I did not power of the system. But I would think that when I powered off and powered on again, logged into the system, the cache for the build will be there. That is an assumption.
But you need to be sure what you are doing with this, otherwise you might cause problems.
I wounder if emerge could be made to do this, like continue building when the build process is aborted. Some commands could be added. emerge --continue <package-name> sort of thing. But I am not the person who could be talking about these, at this stage, since I am new to gentoo. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
rzdndr Tux's lil' helper
![Tux's lil' helper Tux's lil' helper](/images/ranks/rank_rect_1.gif)
Joined: 26 Jul 2024 Posts: 77
|
Posted: Tue Jul 30, 2024 10:48 am Post subject: |
|
|
logrusx wrote: | szatox wrote: | If you want to continue working on a particular task (e.g. manually patch a bug which breaks your build) |
That's not only not necessary at all but afterwards if one runs emerge --resume it'll rebuild that same package as it is part of the job plan it resumes.
You can resume terminated builds with
Code: | FEATURES="keepwork" emerge --resume |
or
Code: | FEATURES="keepwork" emerge package |
You have to take care to clean up PORTAGE_TEMPDIR manually.
You can wait till the interrupted package is finished, terminate emerge, cleanup and resume again without keepwork.
Be warned keepwork may not work as expected or may not work at all.
Best Regards,
Georgi |
Doesn't emerge --resume start from the package that was aborted, and will clean out the build cache of that package and start building from scratch?
Regards |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
logrusx Advocate
![Advocate Advocate](/images/ranks/rank-G-1-advocate.gif)
![](images/avatars/176594205966916859b5145.jpg)
Joined: 22 Feb 2018 Posts: 2162
|
Posted: Tue Jul 30, 2024 11:35 am Post subject: |
|
|
rzdndr wrote: | and will clean out the build cache of that package and start building from scratch? |
man make.conf wrote: |
Code: | keepwork
Do not delete the ${WORKDIR} directory after the merge process. ${WORKDIR} can then be reused since this feature disables most of the clean phase that runs prior to each build. Due to lack
of proper cleanup, this feature can interfere with normal emerge operation and therefore it should not be left enabled for more than a short period of time.
|
|
Best Regards,
Georgi |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
szatox Advocate
![Advocate Advocate](/images/ranks/rank-G-1-advocate.gif)
Joined: 27 Aug 2013 Posts: 3329
|
Posted: Tue Jul 30, 2024 11:43 am Post subject: |
|
|
Huh... Looks l confused this one with keeptemp.
Well, keepwork might actually do the trick, though not deleting workdir after merge implies polluting your tmp with potentially big amount of data. There are a few common packages which check for 6-10GB of free space in tmp before even attempting to compile.
Quote: | keeptemp
Do not delete the ${T} directory after the merge process.
keepwork
Do not delete the ${WORKDIR} directory after the merge process. ${WORKDIR} can then be reused since this feature disables most of the clean phase that runs prior to each build. Due to lack
of proper cleanup, this feature can interfere with normal emerge operation and therefore it should not be left enabled for more than a short period of time. |
This quote comes from
man make.conf _________________ Make Computing Fun Again |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
logrusx Advocate
![Advocate Advocate](/images/ranks/rank-G-1-advocate.gif)
![](images/avatars/176594205966916859b5145.jpg)
Joined: 22 Feb 2018 Posts: 2162
|
Posted: Tue Jul 30, 2024 11:49 am Post subject: |
|
|
szatox wrote: |
Well, keepwork might actually do the trick |
It DOES the trick.
szatox wrote: | though not deleting workdir after merge implies polluting your tmp with potentially big amount of data. There are a few common packages which check for 6-10GB of free space in tmp before even attempting to compile. |
man ebuild wrote: | Code: | clean Cleans the temporary build directory that Portage has created for this particular ebuild file. The temporary build directory normally contains the extracted source files as well as a possible "in‐
stall image" (all the files that will be merged to the local filesystem or stored in a package). The location of the build directory is set by the PORTAGE_TMPDIR variable. For information on what
this variable is, run emerge --info, or to override this variable, see make.conf(5).
Note: Portage cleans up almost everything after a package has been successfully merged unless FEATURES contains ’noclean’. Adding noclean to FEATURES will cause a lot of files to remain and will
consume large amounts of space, very quickly. It is not recommended to leave this on, unless you have use for the sources post-merge. Optionally, one may manually clean these files with rm -rf
/var/tmp/portage. |
|
I've had that issue in the past with Chromium. It didn't want to start because it checked for I think it was 11G back then, not knowing it does not have anything to unpack. I think that I solved with ebuild compile.
As I said above, you can clean up the temp dir later and resume so that other packages do not leave temp dirs behind. But given that you have sufficient space, keepwork is the most convenient way to pick up where you left.
Best Regards,
Georgi |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
rzdndr Tux's lil' helper
![Tux's lil' helper Tux's lil' helper](/images/ranks/rank_rect_1.gif)
Joined: 26 Jul 2024 Posts: 77
|
Posted: Tue Jul 30, 2024 11:53 am Post subject: |
|
|
szatox wrote: | Huh... Looks l confused this one with keeptemp.
Well, keepwork might actually do the trick, though not deleting workdir after merge implies polluting your tmp with potentially big amount of data. There are a few common packages which check for 6-10GB of free space in tmp before even attempting to compile.
Quote: | keeptemp
Do not delete the ${T} directory after the merge process.
keepwork
Do not delete the ${WORKDIR} directory after the merge process. ${WORKDIR} can then be reused since this feature disables most of the clean phase that runs prior to each build. Due to lack
of proper cleanup, this feature can interfere with normal emerge operation and therefore it should not be left enabled for more than a short period of time. |
This quote comes from
man make.conf |
If there is keepwork/keeptemp then there must exist a way to clean the temp and work directories in emerge probably.
I would think that you could run ebuild <ebuildfile> clean after it. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
szatox Advocate
![Advocate Advocate](/images/ranks/rank-G-1-advocate.gif)
Joined: 27 Aug 2013 Posts: 3329
|
Posted: Tue Jul 30, 2024 12:17 pm Post subject: |
|
|
No, you just don't keep this feature enabled. Cleaning up after a job is the default behavior _________________ Make Computing Fun Again |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
rzdndr Tux's lil' helper
![Tux's lil' helper Tux's lil' helper](/images/ranks/rank_rect_1.gif)
Joined: 26 Jul 2024 Posts: 77
|
Posted: Tue Jul 30, 2024 12:28 pm Post subject: |
|
|
logrusx wrote: | szatox wrote: |
Well, keepwork might actually do the trick |
It DOES the trick.
szatox wrote: | though not deleting workdir after merge implies polluting your tmp with potentially big amount of data. There are a few common packages which check for 6-10GB of free space in tmp before even attempting to compile. |
man ebuild wrote: | Code: | clean Cleans the temporary build directory that Portage has created for this particular ebuild file. The temporary build directory normally contains the extracted source files as well as a possible "in‐
stall image" (all the files that will be merged to the local filesystem or stored in a package). The location of the build directory is set by the PORTAGE_TMPDIR variable. For information on what
this variable is, run emerge --info, or to override this variable, see make.conf(5).
Note: Portage cleans up almost everything after a package has been successfully merged unless FEATURES contains ’noclean’. Adding noclean to FEATURES will cause a lot of files to remain and will
consume large amounts of space, very quickly. It is not recommended to leave this on, unless you have use for the sources post-merge. Optionally, one may manually clean these files with rm -rf
/var/tmp/portage. |
|
I've had that issue in the past with Chromium. It didn't want to start because it checked for I think it was 11G back then, not knowing it does not have anything to unpack. I think that I solved with ebuild compile.
As I said above, you can clean up the temp dir later and resume so that other packages do not leave temp dirs behind. But given that you have sufficient space, keepwork is the most convenient way to pick up where you left.
Best Regards,
Georgi |
I think a build can be resumed.
from make.conf wrote: |
Code: | keeptemp
Do not delete the ${T} directory after the merge process.
keepwork
Do not delete the ${WORKDIR} directory after the merge process. ${WORKDIR} can then be reused since this feature disables most of the clean phase that runs prior to each build. Due to lack of proper cleanup, this feature can interfere with normal emerge operation and therefore it should not be left enabled for more than a short period of time. |
|
From the https://devmanual.gentoo.org/eclass-reference/ebuild/index.html index on where the temp directories and work directories are (the directories that have been called ${T} and ${WORKDIR}).
ebuild/index.html wrote: |
Code: | WORKDIR = ${PORTAGE_TMPDIR}/portage/${CATEGORY}/${PF}/work
Contains the path to the package build root. Do not modify this variable.
...
T = ${PORTAGE_TMPDIR}/portage/${CATEGORY}/${PF}/temp
Contains the path to a temporary directory. You may use this for whatever you like. |
|
As you said, a huge compile (libreoffice, chromium, or others) could be started using.
Code: | FEATURES="keepwork" emerge --ask --verbose <package-name> |
Then you can safely abort the build. When you login again, you can run the same command above to continue building.
When the build finishes, you then run
Code: | ebuild <ebuild-file-path> clean |
This removes the work and temp directories, refered to above. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
rzdndr Tux's lil' helper
![Tux's lil' helper Tux's lil' helper](/images/ranks/rank_rect_1.gif)
Joined: 26 Jul 2024 Posts: 77
|
Posted: Tue Jul 30, 2024 12:55 pm Post subject: |
|
|
rzdndr wrote: | logrusx wrote: | szatox wrote: |
Well, keepwork might actually do the trick |
It DOES the trick.
szatox wrote: | though not deleting workdir after merge implies polluting your tmp with potentially big amount of data. There are a few common packages which check for 6-10GB of free space in tmp before even attempting to compile. |
man ebuild wrote: | Code: | clean Cleans the temporary build directory that Portage has created for this particular ebuild file. The temporary build directory normally contains the extracted source files as well as a possible "in‐
stall image" (all the files that will be merged to the local filesystem or stored in a package). The location of the build directory is set by the PORTAGE_TMPDIR variable. For information on what
this variable is, run emerge --info, or to override this variable, see make.conf(5).
Note: Portage cleans up almost everything after a package has been successfully merged unless FEATURES contains ’noclean’. Adding noclean to FEATURES will cause a lot of files to remain and will
consume large amounts of space, very quickly. It is not recommended to leave this on, unless you have use for the sources post-merge. Optionally, one may manually clean these files with rm -rf
/var/tmp/portage. |
|
I've had that issue in the past with Chromium. It didn't want to start because it checked for I think it was 11G back then, not knowing it does not have anything to unpack. I think that I solved with ebuild compile.
As I said above, you can clean up the temp dir later and resume so that other packages do not leave temp dirs behind. But given that you have sufficient space, keepwork is the most convenient way to pick up where you left.
Best Regards,
Georgi |
I think a build can be resumed.
from make.conf wrote: |
Code: | keeptemp
Do not delete the ${T} directory after the merge process.
keepwork
Do not delete the ${WORKDIR} directory after the merge process. ${WORKDIR} can then be reused since this feature disables most of the clean phase that runs prior to each build. Due to lack of proper cleanup, this feature can interfere with normal emerge operation and therefore it should not be left enabled for more than a short period of time. |
|
From the https://devmanual.gentoo.org/eclass-reference/ebuild/index.html index on where the temp directories and work directories are (the directories that have been called ${T} and ${WORKDIR}).
ebuild/index.html wrote: |
Code: | WORKDIR = ${PORTAGE_TMPDIR}/portage/${CATEGORY}/${PF}/work
Contains the path to the package build root. Do not modify this variable.
...
T = ${PORTAGE_TMPDIR}/portage/${CATEGORY}/${PF}/temp
Contains the path to a temporary directory. You may use this for whatever you like. |
|
As you said, a huge compile (libreoffice, chromium, or others) could be started using.
Code: | FEATURES="keepwork" emerge --ask --verbose <package-name> |
Then you can safely abort the build. When you login again, you can run the same command above to continue building.
When the build finishes, you then run
Code: | ebuild <ebuild-file-path> clean |
This removes the work and temp directories, refered to above. |
But this might have a problem. When you run emerge (with keepwork) the second time around, wouldn.t emerge clean the work directory to start?
It might be better to use ebuild, after an emerge was aborted. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
szatox Advocate
![Advocate Advocate](/images/ranks/rank-G-1-advocate.gif)
Joined: 27 Aug 2013 Posts: 3329
|
Posted: Tue Jul 30, 2024 12:58 pm Post subject: |
|
|
Quote: | ebuild <ebuild-file-path> clean
This removes the work and temp directories, refered to above. | just use rm _________________ Make Computing Fun Again |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
rzdndr Tux's lil' helper
![Tux's lil' helper Tux's lil' helper](/images/ranks/rank_rect_1.gif)
Joined: 26 Jul 2024 Posts: 77
|
Posted: Tue Jul 30, 2024 1:19 pm Post subject: |
|
|
szatox wrote: | Quote: | ebuild <ebuild-file-path> clean
This removes the work and temp directories, refered to above. | just use rm |
The directory location might change in the future. You might not remember the location of the build files. Wouldn't it be better to use ebuild? |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
Hu Administrator
![Administrator Administrator](/images/ranks/rank-admin.gif)
Joined: 06 Mar 2007 Posts: 22378
|
Posted: Tue Jul 30, 2024 1:21 pm Post subject: |
|
|
The idea of interrupting and resuming emerge comes up every few months, and the answer is always the same. This may work in some cases, but cannot work in the general case. If you interrupt a build step that does not know how to resume itself correctly, then when you do resume, you may get incorrect results. Consider the following: Code: | $ cat Makefile
d: c
c: a b
echo D > d
{ cat a; sleep 5; cat b; } > "$@"
echo D2 >> d
a:
echo "A" > "$@"
b:
echo "B" > "$@"
$ rm -f a b c d
$ make
echo "A" > "a"
echo "B" > "b"
echo D > d
{ cat a; sleep 5; cat b; } > "c"
echo D2 >> d
$ cat d
D
D2
$ rm -f a b c d
$ make
echo "A" > "a"
echo "B" > "b"
echo D > d
{ cat a; sleep 5; cat b; } > "c"
^Cmake: *** Deleting file 'c'
make: *** [Makefile:5: c] Interrupt
$ cat d
D
| The file d now has incorrect contents, because the step to build it was interrupted, and since it was not directly the output of c, Make did not delete it. You can argue that this is a bad build script, but it's perfectly legal according to the grammar, and I have seen plenty of build systems with this pattern. What will a real build system do when you break it like this? It depends. Maybe you get lucky and you get a nice obvious error message. Maybe you get incorrect build results. Portage does not offer an easy way to restart an aborted phase because the only safe time to interrupt a build is at a checkpoint, and the only guaranteed checkpoints are the ebuild phases.
If you must power down the system in the middle of a build, your best option is to use hibernation so that the processes of the build can be brought back exactly as they were. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
rzdndr Tux's lil' helper
![Tux's lil' helper Tux's lil' helper](/images/ranks/rank_rect_1.gif)
Joined: 26 Jul 2024 Posts: 77
|
Posted: Tue Jul 30, 2024 1:35 pm Post subject: |
|
|
Hu wrote: | The idea of interrupting and resuming emerge comes up every few months, and the answer is always the same. This may work in some cases, but cannot work in the general case. If you interrupt a build step that does not know how to resume itself correctly, then when you do resume, you may get incorrect results. Consider the following: Code: | $ cat Makefile
d: c
c: a b
echo D > d
{ cat a; sleep 5; cat b; } > "$@"
echo D2 >> d
a:
echo "A" > "$@"
b:
echo "B" > "$@"
$ rm -f a b c d
$ make
echo "A" > "a"
echo "B" > "b"
echo D > d
{ cat a; sleep 5; cat b; } > "c"
echo D2 >> d
$ cat d
D
D2
$ rm -f a b c d
$ make
echo "A" > "a"
echo "B" > "b"
echo D > d
{ cat a; sleep 5; cat b; } > "c"
^Cmake: *** Deleting file 'c'
make: *** [Makefile:5: c] Interrupt
$ cat d
D
| The file d now has incorrect contents, because the step to build it was interrupted, and since it was not directly the output of c, Make did not delete it. You can argue that this is a bad build script, but it's perfectly legal according to the grammar, and I have seen plenty of build systems with this pattern. What will a real build system do when you break it like this? It depends. Maybe you get lucky and you get a nice obvious error message. Maybe you get incorrect build results. Portage does not offer an easy way to restart an aborted phase because the only safe time to interrupt a build is at a checkpoint, and the only guaranteed checkpoints are the ebuild phases.
If you must power down the system in the middle of a build, your best option is to use hibernation so that the processes of the build can be brought back exactly as they were. |
A very good point. There may be builds (of build files, like the makefile you gave) recover from an interruption, many might not be able to recover. There is no control over that.
So it might be much better to use binary packages for such big packages.
Good to know. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
rzdndr Tux's lil' helper
![Tux's lil' helper Tux's lil' helper](/images/ranks/rank_rect_1.gif)
Joined: 26 Jul 2024 Posts: 77
|
Posted: Tue Jul 30, 2024 1:51 pm Post subject: |
|
|
Hu wrote: | The idea of interrupting and resuming emerge comes up every few months, and the answer is always the same. This may work in some cases, but cannot work in the general case. If you interrupt a build step that does not know how to resume itself correctly, then when you do resume, you may get incorrect results. Consider the following: Code: | $ cat Makefile
d: c
c: a b
echo D > d
{ cat a; sleep 5; cat b; } > "$@"
echo D2 >> d
a:
echo "A" > "$@"
b:
echo "B" > "$@"
$ rm -f a b c d
$ make
echo "A" > "a"
echo "B" > "b"
echo D > d
{ cat a; sleep 5; cat b; } > "c"
echo D2 >> d
$ cat d
D
D2
$ rm -f a b c d
$ make
echo "A" > "a"
echo "B" > "b"
echo D > d
{ cat a; sleep 5; cat b; } > "c"
^Cmake: *** Deleting file 'c'
make: *** [Makefile:5: c] Interrupt
$ cat d
D
| The file d now has incorrect contents, because the step to build it was interrupted, and since it was not directly the output of c, Make did not delete it. You can argue that this is a bad build script, but it's perfectly legal according to the grammar, and I have seen plenty of build systems with this pattern. What will a real build system do when you break it like this? It depends. Maybe you get lucky and you get a nice obvious error message. Maybe you get incorrect build results. Portage does not offer an easy way to restart an aborted phase because the only safe time to interrupt a build is at a checkpoint, and the only guaranteed checkpoints are the ebuild phases.
If you must power down the system in the middle of a build, your best option is to use hibernation so that the processes of the build can be brought back exactly as they were. |
But isn't there a flaw in your example. When you interrupt the build the second time, you do not restart the build.
When you run the build again after you interrupted it, you will get the results correctly.
So this assumption might not be true I think. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
logrusx Advocate
![Advocate Advocate](/images/ranks/rank-G-1-advocate.gif)
![](images/avatars/176594205966916859b5145.jpg)
Joined: 22 Feb 2018 Posts: 2162
|
Posted: Tue Jul 30, 2024 2:12 pm Post subject: |
|
|
rzdndr wrote: |
But this might have a problem. When you run emerge (with keepwork) the second time around, wouldn.t emerge clean the work directory to start?
It might be better to use ebuild, after an emerge was aborted. |
How many times I have to repeat keepwork prevents it from doing so? Once emerge has finished with that package, unless you run it again without keepwork, emerge won't do nothing in regards to it and won't clean it. You need to clean it manually, either with rm or with ebuild.
Best Regards,
Georgi |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
Hu Administrator
![Administrator Administrator](/images/ranks/rank-admin.gif)
Joined: 06 Mar 2007 Posts: 22378
|
Posted: Tue Jul 30, 2024 2:21 pm Post subject: |
|
|
True, you found a flaw in an example I wrote to make the point. Rerunning make will regenerate c, and as a side effect, d. This is because GNU make is generally good at dependency management, and I failed to create a sufficiently pathological example to counter it. I've seen build systems that are just shell scripts, sometimes without any error checking. Such a script might notice the existence of d and decline to rebuild c, without verifying the mtime or contents of any of the files.
Therefore, next, you need to prove that for all possible build systems, it is impossible to interrupt them at any point where a subsequent restart will produce an incorrect result, whether that result is a missing file, an incomplete file, or any other result that would not occur if you let it run to completion. ![Smile :)](images/smiles/icon_smile.gif) |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
rzdndr Tux's lil' helper
![Tux's lil' helper Tux's lil' helper](/images/ranks/rank_rect_1.gif)
Joined: 26 Jul 2024 Posts: 77
|
Posted: Tue Jul 30, 2024 2:24 pm Post subject: |
|
|
Hu wrote: | True, you found a flaw in an example I wrote to make the point. Rerunning make will regenerate c, and as a side effect, d. This is because GNU make is generally good at dependency management, and I failed to create a sufficiently pathological example to counter it. I've seen build systems that are just shell scripts, sometimes without any error checking. Such a script might notice the existence of d and decline to rebuild c, without verifying the mtime or contents of any of the files.
Therefore, next, you need to prove that for all possible build systems, it is impossible to interrupt them at any point where a subsequent restart will produce an incorrect result, whether that result is a missing file, an incomplete file, or any other result that would not occur if you let it run to completion. ![Smile :)](images/smiles/icon_smile.gif) |
I was not saying that all build systems will recover. Again, it is true that there can be build systems that can not resume, or recover from an interruption. So I think there is no need for a proof here ![Wink ;)](images/smiles/icon_wink.gif) |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
eschwartz Developer
![Developer Developer](/images/ranks/rank-dev.gif)
![](images/avatars/180460208566750902dba46.png)
Joined: 29 Oct 2023 Posts: 132
|
Posted: Tue Jul 30, 2024 3:45 pm Post subject: |
|
|
Hu wrote: | You can argue that this is a bad build script, but it's perfectly legal according to the grammar, and I have seen plenty of build systems with this pattern. What will a real build system do when you break it like this? It depends. Maybe you get lucky and you get a nice obvious error message. Maybe you get incorrect build results. Portage does not offer an easy way to restart an aborted phase because the only safe time to interrupt a build is at a checkpoint, and the only guaranteed checkpoints are the ebuild phases.
|
Saying that it is "perfectly legal according to the grammar" is a bit like saying the sentence:
Quote: |
via hello I'm suspicion by next friendly centrality to blackmail vindicated by caned
|
is a valid sentence because a spellcheck verifies that all the words are real words, without verifying semantics.
Hu wrote: | True, you found a flaw in an example I wrote to make the point. Rerunning make will regenerate c, and as a side effect, d. This is because GNU make is generally good at dependency management, and I failed to create a sufficiently pathological example to counter it. |
It's not a matter of GNU Make, it's a basic requirement of POSIX make. If c is out of date, then d is out of date, and the argument in favor of it being a Makefile that even works on the first compilation, is that running c will always create d as well... If you want to prove something by example, it seems like a basic requirement to make it "sufficiently pathological" to actually prove the baseline point. This could for example mean even finding a single implementation of Make (not GNU Make) that does, in fact, break on this.
Hu wrote: |
I've seen build systems that are just shell scripts, sometimes without any error checking. Such a script might notice the existence of d and decline to rebuild c, without verifying the mtime or contents of any of the files.
Therefore, next, you need to prove that for all possible build systems, it is impossible to interrupt them at any point where a subsequent restart will produce an incorrect result, whether that result is a missing file, an incomplete file, or any other result that would not occur if you let it run to completion. ![Smile :)](images/smiles/icon_smile.gif) |
Sure. I've seen build systems that are just shell scripts without any error checking, too. They aren't very good at successfully building the first time around, either. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
eschwartz Developer
![Developer Developer](/images/ranks/rank-dev.gif)
![](images/avatars/180460208566750902dba46.png)
Joined: 29 Oct 2023 Posts: 132
|
Posted: Tue Jul 30, 2024 3:52 pm Post subject: |
|
|
I would argue that in fact the *real* reason why portage cannot guarantee that a package with an existing workdir can be resumed in the middle of an ebuild phase, is because portage cannot guarantee that ebuild files are, fundamentally, resumable at all. Many ebuilds perform destructive actions such as moving a file in src_install instead of copying it, which is arguably good for speed and for avoiding running out of space on the disk containing the portage tmpdir, but pretty terrible for re-running the install phase.
This is not the fault of the build system, since the build system does in fact work fine -- it is simply a choice by ebuild authors. IIRC the kernel is one of those ebuilds (eclasses, rather) |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
Hu Administrator
![Administrator Administrator](/images/ranks/rank-admin.gif)
Joined: 06 Mar 2007 Posts: 22378
|
Posted: Tue Jul 30, 2024 4:09 pm Post subject: |
|
|
I picked Make because I was in a hurry and thought I could quickly write a bad build system in it that would demonstrate my point about not being able to reliably resume. I was wrong. I wrote something that was insufficiently tricky, because I did not leave out enough dependency information to confuse it while still including enough to guarantee that a non-interrupted run would work. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|