| View previous topic :: View next topic |
| Author |
Message |
frankenputer
n00b
Joined: 09 Mar 2016
Posts: 26
|
 Posted: Wed Jul 13, 2016 8:00 pm Post subject: [HOWTO] Obtain some safe CPU flags |
 |
|
Updated the wiki to include the code below which allows the user to obtain some safe CPU flags.
Copy 'n paste ready.
| Code: | | gcc '-###' -march=native -x c - 2>&1 | gawk -lfilefuncs '{ gsub(/\"/,""); for (x=1; x < NF; x++) { if (match($x,/march|cache/)) { cflagz[$x]++; } else if (match($x,/mpopcnt|m3dnow|msse|mssse|maes/)) { x86[substr($x,3)]++; } else if (match($x,/mcx16|mabm|mlzcnt|msahf/)) { x86[substr($x,2)]++; } } } function concatMe(arr) { ret=""; for (x in arr) { str=(match(x,/cache/) ? "--param"" "x : x); ret=ret" "str; } return ret; } END { PROCINFO["sorted_in"]="@ind_str_asc"; cpuf="/proc/cpuinfo"; if (0 == stat(cpuf, buf)) { while (0 != (getline cur_line < cpuf)) { if (match(cur_line, "flags")) { split(cur_line, arr, " "); if (0 != arr_len=length(arr)) { for (x=1; x < arr_len; x++) { if (match(arr[x],/mmxext|3dnowext/)) { x86[arr[x]]++; } } } break; } } close(cpuf); } if (0 != length(x86)) { printf("CPU_FLAGS_X86=\"...%s\"\n",concatMe(x86)); } if (0 != length(cflagz)) { printf("CFLAGS=\"...%s\"\n",concatMe(cflagz)); } }' |
The script used to be ~ 5 lines long at the beginning.
Filename cpuorakle.awk
| Code: | #!/usr/bin/awk -f
@load "filefuncs";
{
gsub(/\"/,"");
for (x=1; x < NF; x++) {
if (match($x,/march|cache/)) {
cflagz[$x]++;
} else if (match($x,/mpopcnt|m3dnow|msse|mssse|maes/)) {
x86[substr($x,3)]++;
} else if (match($x,/mcx16|mabm|mlzcnt|msahf/)) {
x86[substr($x,2)]++;
}
}
}
function concatMe(arr) {
ret="";
for (x in arr) {
str=(match(x,/cache/) ? "--param"" "x : x);
ret=ret" "str;
}
return ret;
}
END {
PROCINFO["sorted_in"]="@ind_str_asc";
cpuf="/proc/cpuinfo";
if (0 == stat(cpuf, buf)) {
while (0 != (getline cur_line < cpuf)) {
if (match(cur_line, "flags")) {
split(cur_line, arr, " ");
if (0 != arr_len=length(arr)) {
for (x=1; x < arr_len; x++) {
if (match(arr[x],/mmxext|3dnowext/)) {
x86[arr[x]]++;
}
}
}
break;
}
}
close(cpuf);
}
if (0 != length(x86)) {
printf("CPU_FLAGS_X86=\"...%s\"\n",concatMe(x86));
}
if (0 != length(cflagz)) {
printf("CFLAGS=\"...%s\"\n",concatMe(cflagz));
}
} |
Execute it with:
| Code: | chmod +x cpuorakle.awk
gcc '-###' -march=native -x c - 2>&1 | ./cpuorakle.awk |
Sample output:
| Code: | CPU_FLAGS_X86="... 3dnow 3dnowext mabm mcx16 mlzcnt mmxext msahf popcnt sse sse2 sse3 sse4a"
CFLAGS="... -march=amdfam10 --param l1-cache-line-size=64 --param l1-cache-size=64 --param l2-cache-size=512" |
---
Synced the code with the upstream repo.
Last edited by frankenputer on Sun Jul 17, 2016 3:02 am; edited 8 times in total |
|
| Back to top |
|
 |
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
| Posted: Thu Jul 14, 2016 2:59 am Post subject: |
|
|
I have an ugly script that generates human-readable output for CPU_FLAGS_X86, may be of use to someone:
| Code: | #!/bin/sh
flags_path="$(portageq get_repo_path / gentoo)/profiles/desc/cpu_flags_x86.desc"
flags_regex='('$(sed -n '/^flags/ { s/^.*:\s//; y/ /|/; p; q }' /proc/cpuinfo)')'
awk -F' - ' '!/^#/ { printf "%-10s %s\n", $1, $2 }' "$flags_path" \
| egrep --color "(^|\[)$flags_regex\W" |
|
|
| Back to top |
|
|
frankenputer
n00b
Joined: 09 Mar 2016
Posts: 26
|
| Posted: Thu Jul 14, 2016 2:44 pm Post subject: |
|
|
| Ant P. wrote: | I have an ugly script that generates human-readable output for CPU_FLAGS_X86, may be of use to someone:
| Code: | #!/bin/sh
flags_path="$(portageq get_repo_path / gentoo)/profiles/desc/cpu_flags_x86.desc"
flags_regex='('$(sed -n '/^flags/ { s/^.*:\s//; y/ /|/; p; q }' /proc/cpuinfo)')'
awk -F' - ' '!/^#/ { printf "%-10s %s\n", $1, $2 }' "$flags_path" \
| egrep --color "(^|\[)$flags_regex\W" |
|
Thank you, added /proc/cpuinfo parsing. |
|
| Back to top |
|
|
Randy Andy
Veteran
Joined: 19 Jun 2007
Posts: 1152
Location: /dev/koelsch
|
| Posted: Thu Jul 14, 2016 5:11 pm Post subject: Safe CFLAGS and CPU Flags without reinventing the wheel |
|
|
Hi frankenputer,
your sample output contains lots of faults regarding the CPU_FLAGS_X86="" line.
It should only contain CPU Flags which also exist as USE-Flags.
But you don't have to fiddle this out by your own, if you would like to read the news message from:
| Quote: | | 2015-01-28-cpu_flags_x86-introduction |
and follow the instruction, except on little detail. 
Since yesterday the package has been renamed to:
| Quote: | | app-portage/cpuid2cpuflags |
Would be great if someone could release a new message regarding the renaming:
https://packages.gentoo.org/packages/app-portage/cpuid2cpuflags
and also correct the wiki asap:
https://wiki.gentoo.org/wiki/CPU_FLAGS_X86
Regarding dertermine safe CFLAGS:
Inspired by a lecture, I gave at the workshop here (sorry, only german language):
https://forums.gentoo.org/viewtopic-t-1017200-highlight-trolug.html
Sebastian Pipping wrote a nice python-script which does the job quite well.
The package is in the tree since almost a year and is called:
app-misc/resolve-march-native
After installation you can call it e.g. with:
| Code: | | resolve-march-native -a |
and take over the line to you CFLAGS="" line.
A hint to this from the
https://wiki.gentoo.org/wiki/Safe_CFLAGS would be also a great idea, cause it make things a lot easier for the user.
Best regards, Andy.
_________________
If you want to see a Distro done right, compile it yourself! |
|
| Back to top |
|
|
frankenputer
n00b
Joined: 09 Mar 2016
Posts: 26
|
| Posted: Thu Jul 14, 2016 7:16 pm Post subject: Re: Safe CFLAGS and CPU Flags without reinventing the wheel |
|
|
| Randy Andy wrote: | Hi frankenputer,
your sample output contains lots of faults regarding the CPU_FLAGS_X86="" line.
It should only contain CPU Flags which also exist as USE-Flags.
|
Hi Andy,
The Mihai's program detects avx512* flags which doesn't exists as USE flags at this point.
Including those existing cpu instructions that does not exists as CPU_FLAGS_X86 USE flags won't break the system.
In case they are added upstream, all users that have avx512* flags in their configs will be able to leverage them.
---
| Quote: | | But you don't have to fiddle this out by your own, if you would like to read the news message from: |
Running the Mihai's program:
| Code: | cpuinfo2cpuflags-x86
CPU_FLAGS_X86="3dnow 3dnowext mmx mmxext popcnt sse sse2 sse3 sse4a" |
My gawk kung fu:
| Code: | | CPU_FLAGS_X86="... 3dnow 3dnowext mabm mcx16 mlzcnt mmxext msahf popcnt sse sse2 sse3 sse4a " |
Compare both CPU_FLAGS_X86 lines.
mmx is included in vanilla make.conf
---
| Quote: | Sebastian Pipping wrote a nice python-script which does the job quite well.
The package is in the tree since almost a year and is called:
app-misc/resolve-march-native
After installation you can call it e.g. with:
Code:
resolve-march-native -a
and take over the line to you CFLAGS="" line. |
Running the program:
| Code: | resolve-march-native -a
-march=amdfam10 --param l1-cache-line-size=64 -O2 -pipe |
My gawk kung fu:
| Code: | | CFLAGS="... -march=amdfam10 --param l1-cache-line-size=64 --param l1-cache-size=64 --param l2-cache-size=512 " |
Compare both CFLAGS lines.
-O2 -pipe is included in vanilla make.conf
---
I'm not sure how stable avx* flags are, since I don't have intel cpu. This along with the following quote is the reason not to include them in first place.
| Quote: | | GCC depresses SSEx instructions when -mavx is used. Instead, it generates new AVX instructions or AVX equivalence for all SSEx instructions when needed. |
At the end of the day the user has choice, either mine gawk kung fu, Mihai's program or the Sebastian's one or the Ant P. one.
I appreciate negative and positive feedback, but can't stand next to "jump to conclusions" ones. |
|
| Back to top |
|
|
Randy Andy
Veteran
Joined: 19 Jun 2007
Posts: 1152
Location: /dev/koelsch
|
| Posted: Sun Jul 17, 2016 3:28 pm Post subject: Re: Safe CFLAGS and CPU Flags without reinventing the wheel |
|
|
Hi frankenputer,
| frankenputer wrote: |
The Mihai's program detects avx512* flags which doesn't exists as USE flags at this point.
Including those existing cpu instructions that does not exists as CPU_FLAGS_X86 USE flags won't break the system.
In case they are added upstream, all users that have avx512* flags in their configs will be able to leverage them.
|
Right, when your prediction once occurs.
---
| frankenputer wrote: |
Running the Mihai's program:
| Code: | cpuinfo2cpuflags-x86
CPU_FLAGS_X86="3dnow 3dnowext mmx mmxext popcnt sse sse2 sse3 sse4a" |
My gawk kung fu:
| Code: | | CPU_FLAGS_X86="... 3dnow 3dnowext mabm mcx16 mlzcnt mmxext msahf popcnt sse sse2 sse3 sse4a " |
Compare both CPU_FLAGS_X86 lines.
mmx is included in vanilla make.conf
|
Only cause mmx is named into the vanilla make.conf, does not guaranty that it would be thrown away by the average user.
You leave it out by your kung fu, which mean that packages which makes uses of it, doesn't compile this functionality into, although you set the superior grade mmext flag.
Much more packages makes usage of mmx compared to mmext, just compare the output of:
| Code: | | equery h cpu_flags_x86_mmx |
vs. | Code: | | equery h cpu_flags_x86_mmxext |
Similar for your mabm mcx16 mlzcnt msahf, these doen't exist as cpu_flags_x86 and probably never would.
Theses flags belongs to the CFLAGS, if there's a need activate (-m) or deactivate (-mno) it there.
---
| frankenputer wrote: |
Running the program:
| Code: | resolve-march-native -a
-march=amdfam10 --param l1-cache-line-size=64 -O2 -pipe |
My gawk kung fu:
| Code: | | CFLAGS="... -march=amdfam10 --param l1-cache-line-size=64 --param l1-cache-size=64 --param l2-cache-size=512 " |
Compare both CFLAGS lines.
-O2 -pipe is included in vanilla make.conf
|
Regarding resolve-march-native -a
The option -a is only for: | Quote: |
--add-recommended, -a
add recommended flags |
to give the user an idea what makes sense to add to his line.
Your claim here the 2 missing cache-size parameters, but since I found a package (openmpi), which doesn't compile successfully here on my westmere Xeon cpu if I use
the l1-cache-size= parameter, I'm not sure if leaving out the cache parameters is good or bad. How big would be the difference regarding performance? I never tested it.
In the sense of giving safe CFLAGS, it seems to be better to leave it out to me. To present safe CFLAGS is lastly the intention of this little helper script.
The power user (or worser the ricer) hast to decide by himself, how to optimize the settings stronger, but that's not what the title of such wikis reflect.
---
| frankenputer wrote: |
At the end of the day the user has choice, either mine gawk kung fu, Mihai's program or the Sebastian's one or the Ant P. one.
I appreciate negative and positive feedback, but can't stand next to "jump to conclusions" ones. |
Only to ask GCCs CPU recognition fails sometimes, there are lots of bug reports which shows that (references at the bottom).
That's why the Safe_CFLAGS Wiki and Sebastians program tries a different approach, compiling two files with different settings and compare it afterwards, to filter out wrongly recognized flags.
So far my feedback, but I agree, lastly the users decide what to use.
Regards, Andy.
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=39851
https://gcc.gnu.org/ml/gcc-help/2009-04/msg00301.html
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=43718
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=39851
https://gcc.gnu.org/ml/gcc-help/2009-04/msg00305.html
_________________
If you want to see a Distro done right, compile it yourself! |
|
| Back to top |
|
|
frankenputer
n00b
Joined: 09 Mar 2016
Posts: 26
|
| Posted: Sun Jul 17, 2016 5:15 pm Post subject: |
|
|
| Quote: | Only cause mmx is named into the vanilla make.conf, does not guaranty that it would be thrown away by the average user.
You leave it out by your kung fu, which mean that packages which makes uses of it, doesn't compile this functionality into, although you set the superior grade mmext flag. |
Learn what mmxext is.
Use the appropriate page to lookup such cpu instructions: https://packages.gentoo.org/useflags/cpu_flags_x86_mmxext
https://forums.gentoo.org/viewtopic-t-554289-start-0.html for the padlock cpu_flags_x86.
| Quote: | Similar for your mabm mcx16 mlzcnt msahf, these doen't exist as cpu_flags_x86 and probably never would.
Theses flags belongs to the CFLAGS, if there's a need activate (-m) or deactivate (-mno) it there.
Your claim here the 2 missing cache-size parameters, but since I found a package (openmpi), which doesn't compile successfully here on my westmere Xeon cpu if I use
the l1-cache-size= parameter, I'm not sure if leaving out the cache parameters is good or bad. How big would be the difference regarding performance? I never tested it.
In the sense of giving safe CFLAGS, it seems to be better to leave it out to me. To present safe CFLAGS is lastly the intention of this little helper script. |
Never say never, also google which cpu's are supporting avx512* and when they first appeared. Also it's bad idea to enable such CFLAGS globally, as it's bad idea to use USE flags globally instead per-package USE/CFLAG.
The "big" difference is that you've not used Google, also it tells me that you have not programmed in low level language.
Your two comments contradict with each other.
So you've found a bug regarding the cache size, can you be a nice person and report it to openmpi, so they can fix it ?
Same could be said for pipe, just because it ships in vanilla make.conf and is hard-coded in Sebastian's program doesn't make it "safe":
It tells the compiler to use pipes instead of temporary files during the different stages of compilation, which uses more memory. On systems with low memory, GCC might get killed. In those cases do not use this flag.
Also, no one needs hard-coded -O2 -pipe just to tell them that those flags have been "detected".
---
I'll stop right here and ask any moderator/administrator to lock this thread, before some **already angry** people start insulting me and embarrass themselves even further than they are currently. |
|
| Back to top |
|
|
Randy Andy
Veteran
Joined: 19 Jun 2007
Posts: 1152
Location: /dev/koelsch
|
| Posted: Mon Jul 18, 2016 8:57 am Post subject: |
|
|
| frankenputer wrote: |
Learn what mmxext is.
The "big" difference is that you've not used Google, also it tells me that you have not programmed in low level language.
I'll stop right here and ask any moderator/administrator to lock this thread, before some **already angry** people start insulting me and embarrass themselves even further than they are currently. |
Wow,
it seems that you know more about me, than myself.
Respect, never experienced that before here, maybe because you're relative new to this outstanding forum.
It looks like you're not really interested in technical based criticism and react very emotional when you get one.
Then you blame on me on a personal basis and would try to lock the thread - not bad guy, but to late.
No need to, cause I'm not longer willing to discuss/escalate it with you in that way!
Bye.
_________________
If you want to see a Distro done right, compile it yourself! |
|
| Back to top |
|
|
josephg
l33t
Joined: 10 Jan 2016
Posts: 783
Location: usually offline
|
| Posted: Tue Jan 09, 2018 8:15 pm Post subject: Re: [HOWTO] Obtain some safe CPU flags |
|
|
| frankenputer wrote: | | Updated the wiki to include the code below which allows the user to obtain some safe CPU flags. |
which wiki.. link please?
wiki mentons this tool, which gives me slightly different output
| Code: | $ cpuinfo2cpuflags-x86
CPU_FLAGS_X86="mmx mmxext sse sse2 sse3 ssse3" |
your "Copy 'n paste ready" script gives me
| Code: | CPU_FLAGS_X86="... mcx16 msahf sse sse2 sse3 ssse3"
CFLAGS="... -march=core2 --param l1-cache-line-size=64 --param l1-cache-size=32 --param l2-cache-size=2048" |
i see the thread is a bit old, and wonder if it is still correct. can i safely add the above generated output to /etc/portage/make.conf? |
|
| Back to top |
|
|
krinn
Watchman
Joined: 02 May 2003
Posts: 7470
|
| Posted: Wed Jan 10, 2018 2:41 pm Post subject: |
|
|
You don't need any tricky cflags, just use -march=native and your gcc will do that for you, all safely and nicely.
If really you have a reason to do so, like distcc, you should just copy the output of gcc using native.
| Code: | | gcc -march=native -### -E - $2 2>&1 | sed -r '/cc1/!d;s/(\")|(^.* - )//g' |
|
|
| Back to top |
|
|
josephg
l33t
Joined: 10 Jan 2016
Posts: 783
Location: usually offline
|
| Posted: Thu Jan 11, 2018 7:48 pm Post subject: |
|
|
| krinn wrote: | | You don't need any tricky cflags, just use -march=native and your gcc will do that for you, all safely and nicely. |
isn't native the default? why then so many experts have so much in their C*FLAGS as can be seen on so many threads? 
i want to make my system as efficient as can be, using less cpu/memory. i have these lines below. these no good?
| Code: | CFLAGS="-march=native -O2 --pipe -fomit-frame-pointer -ftree-vectorize"
CXXFLAGS="${CFLAGS}"
CPU_FLAGS_X86="mmx mmxext sse sse2 sse3 ssse3" |
i think -O2 and --pipe are also default. i don't know about other default flags. if they are all defaults, then i'd rather take them out, as you say.
Last edited by josephg on Thu Jan 11, 2018 10:19 pm; edited 1 time in total |
|
| Back to top |
|
|
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54624
Location: 56N 3W
|
| Posted: Thu Jan 11, 2018 8:01 pm Post subject: |
|
|
josephg,
-march=native cannot be the default. You might not have the same CPU as was used to build the stage3.
The default is -march unset, and -mtune=generic so the output code will run on any 64 bit AMD/Intel CPU.
-march=native won't work with distcc either. I really don't want amd64 code sent back to my Raspberry Pi :)
| Code: | $ gcc -march=native -### -E - $2 2>&1 | sed -r '/cc1/!d;s/(\")|(^.* - )//g'
-march=amdfam10 -mmmx -m3dnow -msse -msse2 -msse3 -mno-ssse3 -msse4a -mcx16 -msahf -mno-movbe -mno-aes -mno-sha -mno-pclmul -mpopcnt -mabm -mno-lwp -mno-fma -mno-fma4 -mno-xop -mno-bmi -mno-bmi2 -mno-tbm -mno-avx -mno-avx2 -mno-sse4.2 -mno-sse4.1 -mlzcnt -mno-rtm -mno-hle -mno-rdrnd -mno-f16c -mno-fsgsbase -mno-rdseed -mprfchw -mno-adx -mfxsr -mno-xsave -mno-xsaveopt -mno-avx512f -mno-avx512er -mno-avx512cd -mno-avx512pf -mno-prefetchwt1 -mno-clflushopt -mno-xsavec -mno-xsaves -mno-avx512dq -mno-avx512bw -mno-avx512vl -mno-avx512ifma -mno-avx512vbmi -mno-clwb -mno-mwaitx -mno-clzero -mno-pku --param l1-cache-size=64 --param l1-cache-line-size=64 --param l2-cache-size=512 -mtune=amdfam10 |
That's ugly. The no-* don't need to be listed in CFLAGS.
Its left as an exercise for the reader to improve that one liner to suppress the no-*
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
Dr.Willy
Guru
Joined: 15 Jul 2007
Posts: 547
Location: NRW, Germany
|
| Posted: Fri Jan 12, 2018 5:53 pm Post subject: |
|
|
| NeddySeagoon wrote: | That's ugly. The no-* don't need to be listed in CFLAGS.
Its left as an exercise for the reader to improve that one liner to suppress the no-* |
Now that you mention, I wrote that back when I tried distcc
| Code: | | gcc -v -E -x c -march=native -mtune=native - < /dev/null 2>&1 | sed -e '/cc1/!d; s:^.* - ::; s: -mno-\S\+::g' |
|
|
| Back to top |
|
|
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
| Posted: Sat Jan 13, 2018 3:45 am Post subject: |
|
|
It's no one-liner, but this is what I've gathered over the years (adding -E only just occurred to me though, thanks Dr. Willy).
| march-native.sh: | #!/bin/sh
# prints expanded value of -march=native (distcc-compatible) on stdout
# shellcheck disable=SC2046,SC2005
# (we want echo to wordsplit and collapse this to one line)
echo $(
gcc -v -E -march=native -x c /dev/null 2>&1 \
| grep -F -- '-march' \
| grep -Eo -- ' (-m|--param )\S+' \
| grep -Fv -- '-mno-'
) |
|
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Documentation, Tips & Tricks
