| View previous topic :: View next topic |
| Author |
Message |
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
 Posted: Sat Jan 31, 2015 3:00 pm Post subject: |
 |
|
| iamben wrote: | | You seem to be convinced that gentoo devs are out to ruin your life. I think they're doing a great job and I'm willing to give them the benefit of the doubt on changes like this. |
iamben ... oh, really, where do you get that impression? Does blacking your interlocutor work as a form of argumentation? Your giving the impression that I shouldn't *really* question the reasons provided for this change, I should take every change as de facto "good" because if not then I'm casting a dark cloud over the "great job" done.
| iamben wrote: | | The trick with package rebuilds is to let your CPU do most of the work... you don't have to interpret every package rebuild like a slap in the face from the maintainer. On my desktop install with 1400 packages, around 15 (fifteen!) needed a rebuild for some CPU flag change... not a big deal. |
That's not a "trick", its how package rebuilds happen ... and I'm quite aware of how amazing the CPU is in this regard. Yes, I don't have to "interpret" anything, but then I wasn't, though it sounds better for you that I was, as a "slap in the face" adds an extra level of excitement and drama to what was in reality a simple criticism (in fact, initially a question as to the reasoning involved). The fact that you have turned that into a drama, and cast any such questioning as an affront to "developers", simply attests to your not being able to reason as to it being justified ... or not ... but prefer to turn this against me, as I'm obviously "convinced" of things which any *reasonable* person would reject. Are you getting lessons from Lennart?
Also, with your "not a big deal" you act as though you are the only person with those 15 packages installed, but ok, we should rebuild on --changed-use and not question the reasoning involved.
best ... khay |
|
| Back to top |
|
 |
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3477
|
| Posted: Sat Jan 31, 2015 3:03 pm Post subject: |
|
|
| Actually I'm also curious about that design change. What is the problem it solves? |
|
| Back to top |
|
|
iamben
Apprentice
Joined: 10 May 2004
Posts: 278
|
| Posted: Sat Jan 31, 2015 3:07 pm Post subject: |
|
|
| Dudes, lighten up! |
|
| Back to top |
|
|
krinn
Watchman
Joined: 02 May 2003
Posts: 7470
|
| Posted: Sat Jan 31, 2015 5:46 pm Post subject: |
|
|
| iamben wrote: | | CFLAGS to USE flag mapping isn't nearly as straightforward as cpuinfo to USE flag mapping |
I don't see the difference between trusting what a user has put in his CFLAGS and what he have put in his CPU_FLAGS_X86...
Because you seems to assume CPU_FLAGS_X86 cannot be wrong, as it is fill by a little script that look at cpuinfo content... Sorry, its content is like all content from make.conf: user made, even if you tell user to use the script to fill it, it doesn't mean user cannot alter it. |
|
| Back to top |
|
|
iamben
Apprentice
Joined: 10 May 2004
Posts: 278
|
| Posted: Sat Jan 31, 2015 5:52 pm Post subject: |
|
|
| What I meant is that most people don't have specific -m<instruction> arguments in CFLAGS, most people just set a -march value. And more often than not, it's -march=native. Translating all the possible -march values is not as simple as a one-to-one cpuinfo flag to USE flag map. |
|
| Back to top |
|
|
mgorny
Developer
Joined: 27 Apr 2007
Posts: 83
|
| Posted: Sat Jan 31, 2015 6:01 pm Post subject: |
|
|
To answer a few of the concerns.
| Xywa wrote: | | I have used yours pythond script and I have differnet flags than before (eg. sse4_2 instead sse42 or few new eg. mmxext or aes ). What should I do? |
Well, the flags given by the script are or will be more correct :). Along with the changes we're planning to finally force globally consistent names of USE flags. For example, all packages will use sse4_2 (and not sse4.2, sse42, sse4).
| ct85711 wrote: | | Oh fun, another shift that most likely is going to be half pushed out, breaking stuff because it's not pushed fully out to everything, and properly supported... |
Blah, blah, blah. At the point there are three packages not using new flags. One is completely broken at the moment, so I couldn't really update it, the second one uses some fancy flags only in an ancient version that's going to be removed anyday soon, and the third one uses some fancy flags that we need to discuss how to handle. And it's rather unlikely you're using any of them :P.
| lost+found wrote: | | The script doesn't mention the `sse4' flag on a Core i3, although there are packages offering it... |
We're fixing those packages. There's no such thing as 'SSE4'. There's SSE4.1, SSE4.2 and SSE4a.
| eccerr0r wrote: | | Is it possible to now do a per-package CPU_FLAGS_X86? |
In package.use:
| Code: | | www-plugins/adobe-flash cpu_flags_x86_sse2 |
Unless Portage devs refuse to eventually merge the patch, portage-2.2.16 will also let you do:
| Code: | | www-plugins/adobe-flash CPU_FLAGS_X86: sse2 |
| khayyam wrote: | | hello ... I honestly don't see what benefit this change brings, I mean, does it matter if its in USE or some other var if they are all essentially useflags. I can see negatives in the forum of package rebuilds due to --changed-use || --newuse but what exactly is the problem its intended to solve? Is the USE namespace cluttered, and if so how does adding a sub-set improve this? |
Yes, the generic USE namespace is cluttered to the point that figuring out ffmpeg/libav USE flags takes much more time than necessary. When complete (yes, expect more rebuilds!) this will basically hide a number of irrelevant flags from users — like arm/mips/ppc optimization flags on x86 systems, and the other way around.
It also makes the flags 'stand out' from other USE flags, so users clearly know that 'avx2' is some CPU instruction set and not another fancy audio codec. In particular, when you see CPU_FLAGS_X86, you can use the script to get the values and forget about them until you change the CPU or new flags are added.
And yes, that's the major goal: making things more transparent. This also involves making all flags global so that packages don't invent their own versions you have to figure out separately. So transparent and consistent. Namespacing allows us to safely use easy flag names — the AES instruction set is named 'aes' in cpuinfo but we couldn't make it global 'aes' for obvious reasons. Now it can be just 'aes' since it's namespaced.
I don't understand why you're concerned about rebuilds. You are using Gentoo, after all. Rebuilds are the true fun of Gentoo! In fact, you should be running an --emptytree rebuild from time to time just for the fun of it (but seriously, to fix the results of 'unnecessary rebuilds' not being forced often enough).
| krinn wrote: | | It would had been an improvement if CPU_FLAGS_X86 was autobuild by parsing current CFLAGS |
I was considering this but it's entirely non-trivial. I'd probably have to spawn gcc to let it parse the flags (esp. -march=native) and give me a more final value. Then parse the command-line options there. For each -march=, I'd have to perform -march->instruction set mapping (it's hidden in C files of gcc), and apply -mfoo and -mno-foo explicitly. In the end, I'd have to perform another mapping from gcc instruction set names to gentoo repo USE flags and that shouldn't really be hardcoded inside Portage since it's specific to gentoo repo.
Of course, this brings the issue with people having -march= not entirely fit for their system. In this case, /proc/cpuinfo is more precise. In fact, maybe I should instead focus on improving the script to output best-fit CFLAGS as well :).
| limn wrote: | | I would have expected the news item to have the reason for the change. |
That's my fault. I was focused on keeping the news item short while keeping all suggestions and copy&paste snippets to make system update as short as possible. I didn't think about adding motivation or benefits :). That can be fixed of course.
Last edited by mgorny on Sat Jan 31, 2015 6:03 pm; edited 2 times in total |
|
| Back to top |
|
|
asturm
Developer
Joined: 05 Apr 2007
Posts: 9331
|
| Posted: Sat Jan 31, 2015 6:01 pm Post subject: |
|
|
Another storm in a teacup.
@mgorny: Thanks for the change, it is appreciated. And your script did reveal a few flags I hadn't set on some systems (probably because there was no consumer making use of them - yet). |
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Sat Jan 31, 2015 6:13 pm Post subject: |
|
|
| szatox wrote: | | Actually I'm also curious about that design change. What is the problem it solves? |
szatox ... here's what I've been able to ascertain: none (for 99.99999 use cases). There is no real "problem" for it to solve.
| iamben wrote: | | Dudes, lighten up! |
I'm perfectly light enough, thank you. If you weren't such a dabbler in the black arts of the smear and misdirection, then I'd probably just ignore that remark ... dude.
khay |
|
| Back to top |
|
|
iamben
Apprentice
Joined: 10 May 2004
Posts: 278
|
| Posted: Sat Jan 31, 2015 6:16 pm Post subject: |
|
|
| I'm not sure why you are being so hateful, I didn't mean to stir up such trouble. I'm sorry if I've offended you. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Sat Jan 31, 2015 6:25 pm Post subject: |
|
|
| krinn wrote: | | To disable sse in a package: CFLAGS="-nosse" USE="-sse" emerge package |
In contrast ot new solution, this breaks with default-enable sse.
Moreover, this is not what one (I?) is usually doing when emerging for a different ARCH: One does not set a single CPU-option but actually quite a lot - matching for the whole architecture. For instance, for emerging for a pentium3 from an athlon, I need now (skipping CFLAGS, and changes in VIDEO_CARDS and SOUND_CARDS):
| Quote: | | CPU_FLAGS_X86="mmx sse sse2" emerge ... |
while prevoiusly, I needed
| Quote: | | USE=''-3dnow -3nowext sse2 (oh, did I forget something... let's check what is currently on my machine)" emerge ... |
When emerging for a pentium3 from a core2, I need now
| Quote: | | CPU_FLAGS_X86="mmx sse sse2" emerge ... |
(yes, the same as before...) while previously, I needed
| Quote: | | USE="-sse3 -sse4.1 -sse4.2 -sse4_2 -fma3 -fma4 -fma -sse3 -ssse3 (oh, did I forget something... let's check what is currently on my machine; was something new added recently? damned... -aes-ni and maybe something else was forgotten)" emerge ... |
When build for athlon from i3, it is much worse...
So it is much more convenient than just a rename |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Sat Jan 31, 2015 6:34 pm Post subject: |
|
|
| iamben wrote: | | CFLAGS to USE flag mapping isn't nearly as straightforward as cpuinfo to USE flag mapping |
I was also thinking so, but it seems all flags have an analogue. Maybe some expert could write such a script - I would not have the knowledge to do so. |
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Sat Jan 31, 2015 6:42 pm Post subject: |
|
|
| iamben wrote: | | I'm not sure why you are being so hateful, I didn't mean to stir up such trouble. I'm sorry if I've offended you. |
iamben ... you're confusing honesty with ill will, there is a difference between criticism and animosity. If you hadn't started by personalising the discussion in making my question a "grumble", and then continued in that vein in an attempt to "ruin [my] life" (see how that works?) then I wouldn't have had to have taken you to task on it.
Your applying that same "black art" again above, because your implying that what was said could only come from hate. Its the emotive argument and it attempts to slide under the radar of reason because no one who's hateful can possibly be reasonable. It fails however because nothing I've said is hateful, its simply criticism backed up by reason.
best ... khay |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Sat Jan 31, 2015 6:55 pm Post subject: |
|
|
Since mgorny seems to read this thread, and he probably does not speak German, I would just like to report:
In the German Discussion Forum there is a "thank you" thread for the change. (I am not related with this thread, but just reporting). |
|
| Back to top |
|
|
krinn
Watchman
Joined: 02 May 2003
Posts: 7470
|
| Posted: Sat Jan 31, 2015 8:53 pm Post subject: |
|
|
| mgorny wrote: | | krinn wrote: | | It would had been an improvement if CPU_FLAGS_X86 was autobuild by parsing current CFLAGS |
I was considering this but it's entirely non-trivial. I'd probably have to spawn gcc to let it parse the flags (esp. -march=native) and give me a more final value. Then parse the command-line options there. For each -march=, I'd have to perform -march->instruction set mapping (it's hidden in C files of gcc), and apply -mfoo and -mno-foo explicitly. In the end, I'd have to perform another mapping from gcc instruction set names to gentoo repo USE flags and that shouldn't really be hardcoded inside Portage since it's specific to gentoo repo.
Of course, this brings the issue with people having -march= not entirely fit for their system. In this case, /proc/cpuinfo is more precise. In fact, maybe I should instead focus on improving the script to output best-fit CFLAGS as well  .
|
You have 3 different mapping because cpuinfo map some names, gcc map some other names and portage map some other names too.
First thing to do would be pickup either cpuinfo or gcc mapping and force gentoo to use one or the other and stop creating another standard to standardize the standards
gcc naming of flags are not reliable and may change with gcc versions, but it's also the same for cpuinfo, the flags output are just set thru quotes in comment (grep -n -A6 -B1 "If the comment" /usr/src/linux/arch/x86/include/asm/cpufeature.h ), so as it is "just" displaying info", nobody warrant X86_FEATURE_FPU won't be change to /* "coprocessor" OnBoard FPU */ without notice, and it depend on kernel versions this time ; not worst than gcc, but also not better.
I don't see what is really difficult that instead of mapping cpuinfo "pni" -> "gentoo eq" you parse gcc "-msse3" -> "gentoo eq".
And you don't have to parse user cflags to get the flags set, but parse the output of gcc with the cflags set by user (as gcc will validate them for you, and you don't care about what gcc version the user have then, something you will need to care with CPU_FLAGS_X86 as some flags might be valid for his cpu, valid for his kernel (so appears in /proc/cpuinfo), but not for his compiler version.
If the feature is there to ease user, it is just safer to parse gcc to seek cpu feature he wish and assume any feature not present are not need or wish.
Because it have little consequence (less optimize) if i forget to add "-msse" in my cflags and portage would build a package with a USE="-sse" than your X86_CPU_FEATURE assuming package should be build with USE="sse" while my cflags are set with "-mno-sse" on purpose just because my cpuinfo report my cpu could do "sse"
Please note that the "little consequence" is only for user not using -march=native in his cflags (with native gcc will gave you all cflags you could use, and not depending on the cpu, but on the cpu and gcc version in use) ; while the X86_CPU_FEATURE setting "sse" on a package while user have CFLAGS="-march=native -mno-sse" is a misery. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Sat Jan 31, 2015 9:11 pm Post subject: |
|
|
| krinn wrote: | | while the X86_CPU_FEATURE setting "sse" on a package while user have CFLAGS="-march=native -mno-sse" is a misery. |
Unfortunately, it is not so simple: For instance, you might want mplayer to include special "sse" assembler code if you additionally have USE=cpudetection, because then this code is only executed if the cpu detection says it is ok. On the other hand, you would be in trouble if gcc would "randomly" use sse somewhere. |
|
| Back to top |
|
|
krinn
Watchman
Joined: 02 May 2003
Posts: 7470
|
| Posted: Sat Jan 31, 2015 9:32 pm Post subject: |
|
|
| mv wrote: | | Unfortunately, it is not so simple: For instance, you might want mplayer to include special "sse" assembler code if you additionally have USE=cpudetection, because then this code is only executed if the cpu detection says it is ok. On the other hand, you would be in trouble if gcc would "randomly" use sse somewhere. |
Never said it was simple
But balancing pro & cons, i think gcc parsing is better:
gcc parsing method: CFLAGS="-mno-sse" -> CFLAGS="-mno-sse" USE="-sse" emerge mplayer
current: X86_CPU_FEATURE="sse" -> CFLAGS="-mno-sse" USE="sse" emerge mplayer ; it is just lame.
I know gcc is a bitch with mmx (i don't really know why), but it should do better job even if you don't dig into gcc files like mgorny said, as i assume too it would be complex digging. Because except mmx i don't know other flags gcc is getting crazy with. And that exception could be handle. |
|
| Back to top |
|
|
229566
Tux's lil' helper
Joined: 16 Aug 2010
Posts: 127
|
| Posted: Sat Jan 31, 2015 10:33 pm Post subject: |
|
|
| mv wrote: | When build for athlon from i3, it is much worse...
So it is much more convenient than just a rename |
Which is one more reason why portage should do it automagically. Even if that calls for a "autocpuflags" USE flag where portage will figure out everything your CPU supports and implicitly set those flags. Why? Because there has to be the ability to override it and configure as you wish. Eg. "autocpuflags" defines stuff automagically but one sets "-aes-ni" because they think it's backdoored (just an example, don't take this literally  ). And also I guess the same mechanism has to account for cross-compiling. I never did it so I don't know how that would work, but I'm assuming nonnegotiable automagic definition of CPU flags is not desired. |
|
| Back to top |
|
|
limn
l33t
Joined: 13 May 2005
Posts: 997
|
| Posted: Sat Jan 31, 2015 11:50 pm Post subject: |
|
|
@mgorny
In the best of all possible worlds, every news item would have:
What changed.
Why it changed.
What the user needs to do, or not.
If there is not enough space, then perhaps a link to a wiki page or the like.
Of course, in the best of all possible worlds, hard drives would never fail, etc. |
|
| Back to top |
|
|
Anon-E-moose
Watchman
Joined: 23 May 2008
Posts: 6202
Location: Dallas area
|
| Posted: Sun Feb 01, 2015 12:09 am Post subject: |
|
|
This seems more like the specious reasoning that the sysd fanbois use, "trust us, we know what's best" 
I've had no problem with looking at the possible USE flags for ffmpeg, or whatever then deciding if I need to add a flag to make.conf:USE or somewhere else and I doubt that I'm the only one capable of using more than two brain cells.
I mean seriously is this a really the best use of developers or are they trying to dumb it down to the lowest common denominator ie: windows for morons
Edit to add: I'm not trying to start anything but I'm tired of seeing the general users portrayed as being as dumb as a turnip.
_________________
UM780, 6.12 zen kernel, gcc 13, openrc, wayland |
|
| Back to top |
|
|
eccerr0r
Watchman
Joined: 01 Jul 2004
Posts: 9874
Location: almost Mile High in the USA
|
| Posted: Sun Feb 01, 2015 2:07 am Post subject: |
|
|
Only good reason for these ... virtual USE flag remapping... is to make it obvious to a new ARMEL user they should not be using the mmx USE flag...
*shrug*
No idea... Not sure it really changes much IMHO, just was a bit of a surprise for long time users I guess.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Sun Feb 01, 2015 8:23 am Post subject: |
|
|
| Anon-E-moose wrote: | | but I'm tired of seeing the general users portrayed as being as dumb as a turnip. |
I do not see how bringing some consistency and convenience to the processor-optimization flags portrayes the user as dump. It even cleaned up previous mistakes like using sse4.1 and sse4.2 together with sse4_1 and sse4_2 (BTW: Neither should be used for USE-flag-names according to pms) and even sse4 where the meaning of the latter was apparently unclear even to some developers.
That one could proceed further by setting CPU_FLAGS_X86 automatically in portage by some mechanism is without any discussion; however, cpuinfo2cpuflags-x86 is already useful enough for most users (I am very glad that this is now a package and not a one-shot usage as planned originally), and it is easy enough to write a wrapper setting CPU_FLAGS_X86 for any user.
I suggest that users who really think that it is sensible and doable to write a CFLAGS->CPU_FLAGS_X86 automatism should write a corresponding wrapper, preferrably in python, and publish it (e.g. on github): Once the code exists, I am sure it is only a small step to bring it to portage. However, as already mentioned, writing this thing requires quite some knowledge; I couldn't do it without a lot of reading, and I am convinced you have to dig through partially wrong or outdated information. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Sun Feb 01, 2015 8:31 am Post subject: |
|
|
Speaking about perhaps wrong and outdated information which users might have: I am already getting completely new information here.
| krinn wrote: | | I know gcc is a bitch with mmx (i don't really know why). |
What do you mean here? Does that mean that you do recommend to set CPU_FLAGS_X86="mmx" but nevertheless CFLAGS="-mno-mmx" on machines which do support mmx? |
|
| Back to top |
|
|
limn
l33t
Joined: 13 May 2005
Posts: 997
|
| Posted: Sun Feb 01, 2015 11:23 am Post subject: |
|
|
There may be a bit of a feeling of foisting about it.
| iamben wrote: | | Dudes, lighten up! |
may be appropriate.
My goal was to understand what was going on. Some folks like the idea,
others not. Then practically the whole of dense computing got hauled on.
Not all change is for the best. Not all forward movement is progress.
If it works, great. If it does not, try something else. |
|
| Back to top |
|
|
Roman_Gruber
Advocate
Joined: 03 Oct 2006
Posts: 3846
Location: Austro Bavaria
|
| Posted: Sun Feb 01, 2015 11:44 am Post subject: |
|
|
excellent script / eselect news.
though no idea why i did overlook the eselect news. i just checked and saw there was a new one...
Anything which makes more sense, gives a better overview is appreciated. |
|
| Back to top |
|
|
Anon-E-moose
Watchman
Joined: 23 May 2008
Posts: 6202
Location: Dallas area
|
| Posted: Sun Feb 01, 2015 11:52 am Post subject: |
|
|
| mv wrote: | | Anon-E-moose wrote: | | but I'm tired of seeing the general users portrayed as being as dumb as a turnip. |
I do not see how bringing some consistency and convenience to the processor-optimization flags portrayes the user as dump. |
| Quote: | | Yes, the generic USE namespace is cluttered to the point that figuring out ffmpeg/libav USE flags takes much more time than necessary. When complete (yes, expect more rebuilds!) this will basically hide a number of irrelevant flags from users — like arm/mips/ppc optimization flags on x86 systems, and the other way around. |
I didn't find it too much to figure it out. It was neither hard nor time consuming.
This whole hiding things, smacks of the specious reasoning that has gone into the whole sysd brouhaha, "we know what's best because you're just not smart enough".
And quite frankly once set the flags never needed to be looked at again, nor do I remember seeing a plethora of threads started saying it was just too difficult to figure out flags for ffmpeg/libav since those seem to be the onus for this whole change.
Just how difficult is it to do "cat /proc/cpuinfo" and look at the flags?
Consistency should have always been there and "ssse4" vs "sse4.1" vs "sse4_1", for example, was just developers throwing whatever sounded good to them without any checking on their part to find the correct flag or whether another dev was doing it differently.
Edit to add: when the script is run it show "fma3" as a flag, but that doesn't exist, there is "fma" which is fma3 and "fma4" so for consistency and correctness that should be fixed IMO.
Neither /proc/cpuinfo not gcc shows anything other than fma and fma4.
_________________
UM780, 6.12 zen kernel, gcc 13, openrc, wayland |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Portage & Programming
