| View previous topic :: View next topic |
| Author |
Message |
Stopi
n00b
Joined: 28 Jul 2016
Posts: 20
Location: Laos
|
 Posted: Wed Dec 04, 2019 8:59 pm Post subject: Rebuild libs emerged before a given date (beginner tuto) |
 |
|
Months after migration to profile 17.1 (which didn't go smoothly on one of my machines), I had few issues remaining.
It was like if portage didn't really understand the migration from lib32 to lib.
So many libs seemed to be not recognized as installed, even if they were.
This kind of problem might also happened to you: when you try to emerge packages depending of them, you might get an error saying something like:
| Quote: |
...package xxx not found...
...consider adjusting the PKG_CONFIG_PATH environment variable...
|
Another symptom is when you rebuild or upgrade one of those libs, portage is warning you about file collision, because the lib try to override its own files.
You know a lot of those libs will have to be rebuilt, but you don't want to find them one by one, after each package building fail.
This can be quickly tedious...
And sometimes, if you have lots of packages and a slow machine, you just don't want to rebuild everything with:
If you're in that case, here is a way to help.
I just rebuilt the installed libs from my system, following the 4 steps below.
Note: I recommend to use app-misc/screen when dealing with portage, it's really convenient in so many ways (unless you're emerging screen itself).
First step: Find the libs with these commands:
| Code: |
cd /dev/shm/
PRINT_COUNT_ALWAYS=never eix -I *-libs/* --format '<installedversions:DATESORT>' | sort -n | cut -f2-3 > x
vim x
|
Second step: Once in vim, delete all lines after the date you want (in my case, everything emerged after the profile migration is deleted).
Third step: Next just copy/paste this regex replacement
| Code: |
:%s/.\+\s\+\(.*\)\n/\1 /
|
Now you should have only 1 line with a list of the wanted packages.
Save and exit vim.
Last step: Finally run:
| Code: | | emerge -1av `cat x` |
Success! Portage will merge only the libs you need it to rebuild and consider them to be part of its world again.
Still portage will fail to build some of them.
That's because not everything is inside the *-libs categories (for example, sqlite3 isn't, it's in dev-db).
If you need clues on what to rebuild for a failing package's merge, you can have answers using equery like so:
| Code: | | equery g =media-libs/harfbuzz-2.6.4 |
After you fixed the issue, emerge --resume may not work.
In that case, just restart at first step to have a fresh list of libs still to re-emerge (remember not to skip second step, which is key here for refreshing).
WARNING: some libs can still have an issue, like media-libs/portaudio (see this topic or this bug).
I just excluded it manually from the rebuild list.
Note: You can then forget about the x file, because /dev/shm is supposed to be mounted in RAM.
Hope this will help some people out there.
Just remember the algorithm and adapt the commands to your real situation.
Bye  .
Last edited by Stopi on Fri Dec 06, 2019 7:01 pm; edited 7 times in total |
|
| Back to top |
|
 |
Hu
Administrator
Joined: 06 Mar 2007
Posts: 22753
|
| Posted: Thu Dec 05, 2019 1:35 am Post subject: Re: Rebuild libs emerged before a given date (beginner tuto) |
|
|
| Stopi wrote: | Third step: Next just copy/paste this regex replacement (please note there's one space at the end of line): | Code: | | :%s/.\+-libs\/\(.*\)\n/\1 |
|
I think you could more easily do this by using :%j in Vim. Since you already restricted the list to packages in *-libs, your expression matches every line. Sometimes there is value in retaining the category name to help Portage disambiguate. |
|
| Back to top |
|
|
Stopi
n00b
Joined: 28 Jul 2016
Posts: 20
Location: Laos
|
| Posted: Thu Dec 05, 2019 2:15 am Post subject: Re: Rebuild libs emerged before a given date (beginner tuto) |
|
|
| Hu wrote: | | I think you could more easily do this by using :%j in Vim. Since you already restricted the list to packages in *-libs, your expression matches every line. Sometimes there is value in retaining the category name to help Portage disambiguate. |
Thanks for your comment, I've edited my post and changed the regex from:
| Code: | | :%s/.\+-libs\/\(.*\)\n/\1 |
to:
| Code: | | :%s/.\+\s\+\(.*\)\n/\1 / |
You are totally right: the category is sometimes important.
This new regex makes the code more generic, in case the eix is used on another category.
I've also added an ending slash, so people can't miss the trailing space.
However, I don't see how :%j could work, since we need to remove the date on each line.
If it weren't for filtering manually on date, we could even have done this without vim, in a one-liner program, by redirecting the eix command to sed. |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 22753
|
| Posted: Fri Dec 06, 2019 3:42 am Post subject: |
|
|
I didn't try your output command, so I didn't notice that your substitution was to discard more than just the category name. You are correct that :%j is not appropriate if you need to remove text other than the newlines. You could avoid the manual editing step if you used a filter that inspected the installation date and deleted lines accordingly. For example, PRINT_COUNT_ALWAYS=never eix -I '*-libs/*' --format '<installedversions:DATESORT>' | gawk '{ if ($1 > '"$(date -d '-2 days' +%s)") { print; } }'. You could instead compute the appropriate cut-off time and hardcode it in the gawk. Either way, you would then use the other parts of the pipeline that you wrote above to prepare the output for emerge. | Code: | | PRINT_COUNT_ALWAYS=never eix -I '*-libs/*' --format '<installedversions:DATESORT>' | gawk '{ if ($1 > '"$(date -d '-2 days' +%s)"') { print; } }' | sort -n | cut -f3 | tr '\n' ' ' |
- eix: suppress the summary line, so that a later sed to delete it is not needed.
- gawk: filter by time, print matching packages
- sort: unchanged
- cut: save only package name, no timestamps
- tr: join lines
If you move the sort to before the gawk, then sort has more work to do, but you can use gawk to discard unwanted fields, thus avoiding use of cut entirely. That also lets you move the newline removal into gawk, since nothing later will need to isolate the records. | Code: | | PRINT_COUNT_ALWAYS=never eix -I '*-libs/*' --format '<installedversions:DATESORT>' | sort -n | gawk -vORS=' ' -F'\t' '{ if ($1 > '"$(date -d '-2 days' +%s)"') { print $3; } }' |
|
|
| Back to top |
|
|
Stopi
n00b
Joined: 28 Jul 2016
Posts: 20
Location: Laos
|
| Posted: Fri Dec 06, 2019 6:50 pm Post subject: |
|
|
Thanks for this.
I thought there were some ways to do it at once, but I've been too lazy, plus the idea of this tuto is to provide several step that can be used separately to fit other situations.
I think there are cases where users want to see the actual dates, just to know and select the start of the rebuild.
Anyway, I have integrated to OP your:
| Code: | | PRINT_COUNT_ALWAYS=never |
I wasn't aware of that eix option, it's indeed much more elegant than deleting with sed.
By the way, if we are sending data to emerge without any manual control on date, there's no point to use the sort command.
We end up by solving my specific use case (60 days after the migration) in one line:
| Code: |
emerge -1av $(PRINT_COUNT_ALWAYS=never eix -I '*-libs/*' --format '<installedversions:DATESORT>' | gawk -vORS=' ' -F'\t' '{ if ($1 > '"$(date -d '-60 days' +%s)"') { print $3; } }')
|
|
|
| Back to top |
|
|
Stopi
n00b
Joined: 28 Jul 2016
Posts: 20
Location: Laos
|
| Posted: Fri Dec 06, 2019 7:57 pm Post subject: |
|
|
...just realized it could be done with only 3 steps.
- First step
Find the libs with this command:
| Code: |
PRINT_COUNT_ALWAYS=never eix -I *-libs/* --format '<installedversions:DATESORT>' | sort -n | cut -f2-3 > /dev/shm/x
|
Second step
Edit the "/dev/shm/x" file and delete all lines after the date you want, then save.
Last step
Launch the rebuild:
| Code: |
emerge -1av $(cat /dev/shm/x | cut -f2-2 | tr '\n' ' ')
|
I guess for the sake of a tuto, it's better to have less steps. |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 22753
|
| Posted: Sat Dec 07, 2019 1:19 am Post subject: |
|
|
| Stopi wrote: | | Code: | | $(cat /dev/shm/x | cut -f2-2 | tr '\n' ' ') |
|
This is a useless use of cat. You can get the same result by changing cat /dev/shm/x | cut -f2-2 into cut -f2-2 < /dev/shm/x. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Documentation, Tips & Tricks
