| View previous topic :: View next topic |
| Author |
Message |
dufeu
l33t
Joined: 30 Aug 2002
Posts: 924
Location: US-FL-EST
|
 Posted: Thu Aug 31, 2006 8:00 pm Post subject: Sugs 4 'network portgage masking' was 'architecture' flag |
 |
|
Yeah I know. You read the subject and think to yourself "What! Is he nut's?"
Yep. And since I'm not a programmer, I'm free to make Joe 6-pack type suggestions. 
I've started trying Gentoo in late 2001. In 2002, I finally made the plunge and converted nearly all my PCs to run Gentoo. Over this time, I've noticed two kinds of cruftiness. The first, which I don't really care about that much are inadvertent left over files. The second has become nearer and dearer to my heart and is the focus of my way out there and wild suggestion.
Like many Gentoo users, I'm set to run the bleeding edge of the architectures I have on my network. These are ~x86 and ~amd64. This is great because I get to play with all the latest user packages as they get updated.
The problem, which grows progressively worse over time, is the cumlative effects caused by tool chain upgrades. I call this problem toolchain cruftiness. Basically, different packages with interdependencies are build against slightly different versions of gcc, glibc, binutils and related. These user packages with inter-dependencies end up being sublty out of whack.
So what are these effects?
Well, new packages won't compile for reasons unknow. Other packages stop working even though they appear to compile fine. Stability is threatened.
On one of my main PCs, kmail stopped opening. konqueror stopped opening. Several packages refused to compile. Firefox would have all it's processes die simultaneously at random times. Definitely weird stuff.
I did have some clues that I had tool chain issues, but it certainly wasn't at all clear. Plus, my tool chain packages were installed correctly. I finally decided to completely recompile this PC. And in fact, I've been running the 'emwrap.sh' script multiple times for the last four days. e.g. successively with options -St -Sb -S -wet -W -W {from memory, may not be completely correct}. To compound the problem, I have two other machines I 'cloned' from this PC using 'mkstage4.sh'. While not as bad, I will clearly need to recompile them as well.
Even though the process isn't complete, it is far enough along so that kmail and friends are working properly, packages that failed emerging are now emerged to completion so on and so forth.
Due to the nature of the problem, the only solution when toolchain cruftyness sets in is a complete recompile or reinstall of the system. I have no magical answer for fixing cruftiness problem.
My suggestion relates to preventing or at least postponing the onset of TC cruftiness.
What I'd like to see is a separate architecture flag for the toolchain. e.g. "tc" and "~tc".
The following example matrix would result:
| Code: | "x86 tc" ==> super conservative stable version of Gentoo {default}
"~x86 tc" ==> keep the toolchain stable - get bleeding edge user packages {my personal preference}
"x86 ~tc" ==> not realistic {invalid choice}
"~x86 ~tc" ==> i'm a masochist
i'm also a programing god
{make me bleed} |
When rebuilding the toolchain, emrap.sh finds 256 packages. I would expect all of these packages to fall under the "tc" flag.
When rebuilding the rest of my 'world', emwrap.sh finds 1217 packages.
A stable toolchain would permit a much longer period of time to go by without TC cruftiness setting in.
Certainly, this would increase complexity in portage. On the other hand, this would probably reduce the number of posts reporting bugs in bugzilla and pleas for aid in the forums. I'm sure Jakub would be very happy to see a drop in the number of new bugzilla pointings he looks at each day. 
Does anyone else find merit in this suggestion?
_________________
People whom think M$ is mediocre, don't know the half of it.
Last edited by dufeu on Sat Sep 02, 2006 8:08 pm; edited 1 time in total |
|
| Back to top |
|
 |
brims
Guru
Joined: 19 Apr 2004
Posts: 492
Location: Arizona
|
| Posted: Thu Aug 31, 2006 9:39 pm Post subject: |
|
|
While we're on the subject of suggesting new flags, how about an ACCEPT_KEYWORDS=cvs (or svn or they can be synonymous); you want bleeding edge, there you go. It could go along with the toolchain flags. Something like cvstc.
_________________
Adopt an Unanswered Post
Report violations, duplicates, misplaced, etc |
|
| Back to top |
|
|
Archangel1
Veteran
Joined: 21 Apr 2004
Posts: 1212
Location: Work
|
| Posted: Thu Aug 31, 2006 9:52 pm Post subject: Re: Suggestion for new 'architecture' flag |
|
|
| dufeu wrote: | | When rebuilding the toolchain, emrap.sh finds 256 packages. I would expect all of these packages to fall under the "tc" flag. |
That's far more than just a toolchain - last time I installed a base system it was about 100 packages - and I wouldn't consider most of those part of the toolchain.
brims: What about ACCEPT_KEYWORDS="-*"? For those ebuilds that have a -9999 version, that does exactly what you're after.
_________________
What are you, stupid? |
|
| Back to top |
|
|
brims
Guru
Joined: 19 Apr 2004
Posts: 492
Location: Arizona
|
| Posted: Thu Aug 31, 2006 10:35 pm Post subject: Re: Suggestion for new 'architecture' flag |
|
|
| Archangel1 wrote: | | brims: What about ACCEPT_KEYWORDS="-*"? For those ebuilds that have a -9999 version, that does exactly what you're after. |
Never knew that, neither the -* part nor the -9999 versions. That documented in the documentaion page?
EDIT: ACCEPT_KEYWORDS=cvs would be easier I think.
_________________
Adopt an Unanswered Post
Report violations, duplicates, misplaced, etc |
|
| Back to top |
|
|
Carlo
Developer
Joined: 12 Aug 2002
Posts: 3356
|
| Posted: Thu Aug 31, 2006 10:57 pm Post subject: Re: Suggestion for new 'architecture' flag |
|
|
| dufeu wrote: | | The problem, which grows progressively worse over time, is the cumlative effects caused by tool chain upgrades. I call this problem toolchain cruftiness. Basically, different packages with interdependencies are build against slightly different versions of gcc, glibc, binutils and related. These user packages with inter-dependencies end up being sublty out of whack. |
Slightly different? Either ABI compatible or not - unless the compiler is buggy.
| dufeu wrote: | Well, new packages won't compile for reasons unknow. Other packages stop working even though they appear to compile fine. Stability is threatened.
On one of my main PCs, kmail stopped opening. konqueror stopped opening. Several packages refused to compile. Firefox would have all it's processes die simultaneously at random times. Definitely weird stuff. |
You simply have to rebuild your system as a whole, when the compiler ABI changed. Read this thread.
| dufeu wrote: | | What I'd like to see is a separate architecture flag for the toolchain. e.g. "tc" and "~tc". |
The answer is something between won't and can't. There's not the workforce behind Gentoo to provide infinitive support and to speak about the current change to GCC 4.1.1: GCC 3.x isn't supported upstream anymore either. See also GLEP 19 - if there were enough interested folks behind it, doing the work, it were not as dead as it is.
You wan't to have the cake and eat it. Either you value the benefits of a source distro or you go with a binary distribution.
_________________
Please make sure that you have searched for an answer to a question after reading all the relevant docs. |
|
| Back to top |
|
|
Headrush
Watchman
Joined: 06 Nov 2003
Posts: 5597
Location: Bizarro World
|
| Posted: Thu Aug 31, 2006 11:13 pm Post subject: Re: Suggestion for new 'architecture' flag |
|
|
| brims wrote: | | Archangel1 wrote: | | brims: What about ACCEPT_KEYWORDS="-*"? For those ebuilds that have a -9999 version, that does exactly what you're after. |
Never knew that, neither the -* part nor the -9999 versions. That documented in the documentaion page?
EDIT: ACCEPT_KEYWORDS=cvs would be easier I think. |
Too easy.
Too many people are running unstable packages that shouldn't be already.
Although there isn't much difference between using cvs vs. -*, I beat more people don't understand -* initially and have to do so research first.
You add an arch like cvs and I think people will be more likely to use it including noobies that shouldn't be.
Before you know it the forums are just flooded with problems from noobies when things break. |
|
| Back to top |
|
|
dufeu
l33t
Joined: 30 Aug 2002
Posts: 924
Location: US-FL-EST
|
| Posted: Fri Sep 01, 2006 4:00 am Post subject: |
|
|
| brims wrote: | | While we're on the subject of suggesting new flags, how about an ACCEPT_KEYWORDS=cvs (or svn or they can be synonymous). |
SHUDDERS! Good $DIETY, no!! Not me!
I'm a crazy, slightly masochistic fool. Not a blithering id10T. I'm one of those people you don't want that far on the bleeding edge.  As it is right now, Jakub probably marks two thirds of the bugs I enter either as duplicates or as things wrong at my end. heh. I'm kind of proud of the fact that at least some of my bugs are valid. I wince thinking of his comments if I were to start getting involved in CVS level stuff.
| Headrush wrote: | | Slightly different? Either ABI compatible or not - unless the compiler is buggy. |
In theory, I know you're correct. Still, theory doesn't quite explain my stability issues before and and after recompiling everything. While I'm not a general programmer, I am a programmer of sorts. I program in a dead legacy environment called PICK/Universe. So yes, I fully understand that computers only do and can only do what you tell them, even though I couldn't progam a plug-in to save my life. For what it's worth, the three PCs I'm referring to date back to and have gone through all unstable available versions of gcc since 3.3. The same thing applies to time period equivalents of glibc, binutils etc.
| Quote: | | You simply have to rebuild your system as a whole, when the compiler ABI changed. Read this thread. |
I understand and agree. What I'm after is a little more subtle than just the compiler ABI. When I refer to 'toolchain' I'm really referring to the base system including everything you need to bootstrap in before you can compile meaningful user packages. I suppose labelling the proposed flags as "system" and "~system" would be more descriptive though I'm not certain how much of 'system' should actually be included. Most definitely not my area of expertise.
BTW, I've actually read and followed the gcc upgrade instructions. However, I don't feel the instructions are sufficient which is why I use emwrap.sh. I agree with the contention that you have to compile gcc as well as all associated packages at least twice in order to get a clean toolchain and a clean base for the rest of everything else. I also think there is benefit to compiling the tc and system level stuff in three passes and the rest of world twice.
Two thoughts. A} Of course there are bugs everywhere, including in the tc and other base packages. Otherwise we wouldn't have so many -r# and .#.# revisions. B} Interdependencies including circular dependencies are important considerations. Doing an emerge -t tells me this. To be honest, when doing a gcc or glibc upgrade, I don't see how you can get away with a single re-compile of world.
| Quote: | | The answer is something between won't and can't. There's not the workforce behind Gentoo to provide infinitive support ... |
I'm continually amazed and appreciative of the amount and quality of support that there is. If I could actually do the kind of programming you folk do, I'd be contributing that way too. Testing on different PCs with different processors and different chipsets is about the best I can do to help I'm afraid. My suggestion is simply a wish. You know. In an ideal world and all that.
| Quote: | | You wan't to have the cake and eat it. Either you value the benefits of a source distro or you go with a binary distribution. |
Not quite. I want a stable base to build everything else on. Trust me. I'm very happy on a source distro. Especially this one. It's just that after having built so many and lived long term with Gentoo based systems, I'm coming to realise that there is probably some value to differentiating more strongly between the base set of packages and everything else built upon them. You'll have to pry my Gentoo from my cold, dead hands before I switch to a binary based distro.
| Archangel1 wrote: | | That's far more than just a toolchain - last time I installed a base system it was about 100 packages - and I wouldn't consider most of those part of the toolchain. |
Yep. See above. I don't know how to differentiate out and group the packages involved. I rely on emwrap.sh to do that for me.
Unlike many other people, it doesn't bother me to let my 'puters spend 4+ days rebuilding themselves. I believe in redundancy. That's why I have a back up firewall, backup server and multiple workstations at home. I should really re-try distcc again. The relatively new Gentoo HOW-TO guide for distcc looks simple enough that even I will be able to follow it.
BTW - my main workstation PC just finished up this evening. So far, everything looks incredibly mean, lean, clean and stable. ahhhhh. It's so nice to have everything 'just work' again.
_________________
People whom think M$ is mediocre, don't know the half of it. |
|
| Back to top |
|
|
brims
Guru
Joined: 19 Apr 2004
Posts: 492
Location: Arizona
|
| Posted: Fri Sep 01, 2006 12:13 pm Post subject: Re: Suggestion for new 'architecture' flag |
|
|
| Headrush wrote: | Too easy.
Too many people are running unstable packages that shouldn't be already.
Although there isn't much difference between using cvs vs. -*, I beat more people don't understand -* initially and have to do so research first.
You add an arch like cvs and I think people will be more likely to use it including noobies that shouldn't be.
Before you know it the forums are just flooded with problems from noobies when things break. |
Not many packages have a version 9999. So you can't exactly build an entire bleeding edge cvs system.
_________________
Adopt an Unanswered Post
Report violations, duplicates, misplaced, etc |
|
| Back to top |
|
|
Headrush
Watchman
Joined: 06 Nov 2003
Posts: 5597
Location: Bizarro World
|
| Posted: Fri Sep 01, 2006 2:07 pm Post subject: Re: Suggestion for new 'architecture' flag |
|
|
| brims wrote: | | Headrush wrote: | Too easy.
Too many people are running unstable packages that shouldn't be already.
Although there isn't much difference between using cvs vs. -*, I beat more people don't understand -* initially and have to do so research first.
You add an arch like cvs and I think people will be more likely to use it including noobies that shouldn't be.
Before you know it the forums are just flooded with problems from noobies when things break. |
Not many packages have a version 9999. So you can't exactly build an entire bleeding edge cvs system. |
Its only takes one that breaks another package and they're screaming.
(I would expect to see more as many projects are heating up and advancing fast. How many have been added in the last few months?) |
|
| Back to top |
|
|
R.Smith
Tux's lil' helper
Joined: 20 Nov 2005
Posts: 131
Location: Caerdydd, Cymru.
|
| Posted: Fri Sep 01, 2006 3:06 pm Post subject: |
|
|
| dufeu: May I suggest that you mask the toolchain packages (via /etc/portage/package.mask) and only update them when necessary? That should provide you with the stable base you want. |
|
| Back to top |
|
|
dufeu
l33t
Joined: 30 Aug 2002
Posts: 924
Location: US-FL-EST
|
| Posted: Fri Sep 01, 2006 6:21 pm Post subject: |
|
|
| R.Smith wrote: | | dufeu: May I suggest that you mask the toolchain packages (via /etc/portage/package.mask) and only update them when necessary? That should provide you with the stable base you want. |
I did think of doing this. And your suggestion has merit.
My problem is that I currently have 10 active Gentoo PCs on my network, all different. This means I have to maintain 10 different package.masks. Individually, I can indeed, and probably will set package.mask on each PC. But first I have to figure out what I want to mask. It's not enough to mask just gcc, glibc and binutils.
My experience with simultaneously supporting multiple, wildly different configurations is what led to initiating this discussion in the first place. When you're supporting a network of Gentoo installations, the individual attention each requires tends to eat up one's time.
It probably would be a nifty alternative to have a webmin plugin to globally set the /etc/portage/package.* files across a network. The idea would be to be able to add and delete package mask entries in common to all of them without stepping on individually unique entries. Or even to create a network global /etc/portage/package.* directory with the local package directories retaining individual higher priorities. It's times like these I wish I was a different kind of programmer.
Any volunteers?
_________________
People whom think M$ is mediocre, don't know the half of it. |
|
| Back to top |
|
|
Nick C
Guru
Joined: 18 Mar 2005
Posts: 526
Location: Portsmouth, England
|
| Posted: Fri Sep 01, 2006 8:08 pm Post subject: |
|
|
or you could cheat and just have all pcs on a network share the same stuff by having /etc/portage shared and mounted over nfs
_________________
Please add [solved] to the initial post's subject line if you feel your problem is resolved.
www.monkeydust.net |
|
| Back to top |
|
|
dufeu
l33t
Joined: 30 Aug 2002
Posts: 924
Location: US-FL-EST
|
| Posted: Sat Sep 02, 2006 8:06 pm Post subject: |
|
|
| Nick C wrote: | | or you could cheat and just have all pcs on a network share the same stuff by having /etc/portage shared and mounted over nfs |
I would if I could! I already share /usr/portage/distfiles that way.
The problem is that package.mask is different on each PC.
Actually, the idea of a global /network/portage directory has appeal. This way I could package.(u)mask packages on a global basis but still have the ability to fine tune each PC.
I already run my Gentoo based firewall as my ntpd server for the network. Then I run my behind the firewall server as a Gentoo rsync server, NFS shares server, Samba server. Since I have a server up 24/7 anyway, I could set up an NFS share for /network/portage.
This actually has the earmarks of something doable. Any dev care to comment? I'd be happy to refine it and put the enhancement request in.
I guess /etc/make.conf would need a new variable like NETWORK_PORTAGE="/etc/network/portage" where the default would be null.
_________________
People whom think M$ is mediocre, don't know the half of it. |
|
| Back to top |
|
|
Genone
Retired Dev
Joined: 14 Mar 2003
Posts: 9538
Location: beyond the rim
|
| Posted: Wed Sep 06, 2006 3:26 am Post subject: |
|
|
| dufeu wrote: | | I guess /etc/make.conf would need a new variable like NETWORK_PORTAGE="/etc/network/portage" where the default would be null. |
See here.
PS: please edit the topic so it makes sense again. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Gentoo Chat
