| View previous topic :: View next topic |
| Author |
Message |
Petross404
n00b
Joined: 27 Sep 2016
Posts: 55
|
 Posted: Thu Jul 18, 2019 2:12 pm Post subject: PGO hook for bashrc |
 |
|
I wrote a hook in bashrc for PGO to apply different cflags based on whether gcda files exist or not:
| Code: |
#!/bin/bash
PGO_DIR=/var/tmp/pgo
PROFILE_DIR="${PGO_DIR}/${CATEGORY}/${P}"
PROFILABLE=0
CFLAGS_PROFILE_GEN="-fprofile-generate=${PROFILE_DIR} -fprofile-arcs -fvpt"
CFLAGS_PROFILE_USE="-fprofile-use=${PROFILE_DIR} -fprofile-correction" #Multithreading is corrupting gcda files, so we "correct" them
LDFLAGS_PROFILE_GEN="-fprofile-arcs"
CFLAGS_CUSTOM=""
LDFLAGS_CUSTOM=""
if [ "${EBUILD_PHASE}" == "pre_src_unpack" ]
then
if type epatch_user >& /dev/null; then
cd "${S}"
epatch_user
fi
touch /tmp/profile #This is a global var to indicate if profiling is taking place
case "${CATEGORY}" in #Enable pgo for some categories
*app-editor* | *mail-client* | *media* | *www-client* )
#If the package isn't using debug cflags or a pgo USE flag then...
if [[ "$CFLAGS" != *"gdb"* || "${IUSE}" != "*pgo*" ]]; then
echo "1" > /tmp/profile
elog "Portage will try to profile " "${PN} ..."
#If this doesn't exist, then it's compiled for the fist time
if [ -d "${PROFILE_DIR}" ]
then
elog "Running with PGO for the first time. Use ${P} before you recompile it."
CFLAGS="${CFLAGS} ${CFLAGS_CUSTOM} ${CFLAGS_PROFILE_USE}"
LDFLAGS="${LDFLAGS} ${LDFLAGS_CUSTOM}"
else
elog "Portage will use the analysis that is stored at ${PROFILE_DIR}."
CFLAGS="${CFLAGS} ${CFLAGS_CUSTOM} ${CFLAGS_PROFILE_GEN}"
LDFLAGS="${LDFLAGS} ${LDFLAGS_CUSTOM} ${LDFLAGS_PROFILE_GEN}"
fi
CXXFLAGS="${CFLAGS}"
fi
if [[ "$IUSE" == "*pgo*" ]]; then
elog "PGO is in USE flags, no need to use this bashrc."
fi
esac
fi
....
if [ "${EBUILD_PHASE}" == "postinst" ]
then
#if PGO was applied, test if it succeeded and chown the PROFILE_DIR
read PROFILABLE < /tmp/profile
if [[ "$PROFILABLE" == 1 ]] ; then
for i in `equery f ${PN} | grep "bin/"`;
do
if [[ -z `strings $i | grep profiling` ]]; then
ewarn "This binary isn't making use of PGO while it's compiled with it:" $i
else
elog "This binary is making use of PGO:" $i
fi
done
fi
#Normal user should be able to "write" here the statistical analysis needed for PGO
# TODO: What about other users?
chown -R petros:petros "${PGO_DIR}"
rm /tmp/profile
#Ok, let's update the database
echo ":: Calling updatedb to update its database"
fi
|
Any comments and critic is welcome. |
|
| Back to top |
|
 |
ff11
l33t
Joined: 10 Mar 2014
Posts: 664
|
| Posted: Thu Jul 18, 2019 3:45 pm Post subject: |
|
|
Hi! You should make a parse on this thing first. One good parse and check is the site https://www.shellcheck.net
Or you could install the dev-util/shellcheck-bin and use the command (for the same result):
| Code: | | shellcheck --shell=bash script_file |
|
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 22730
|
| Posted: Fri Jul 19, 2019 1:07 am Post subject: Re: PGO hook for bashrc |
|
|
| Petross404 wrote: | | Code: | #!/bin/bash
PGO_DIR=/var/tmp/pgo
PROFILE_DIR="${PGO_DIR}/${CATEGORY}/${P}"
PROFILABLE=0
CFLAGS_CUSTOM=""
LDFLAGS_CUSTOM=""
if [ "${EBUILD_PHASE}" == "pre_src_unpack" ]
then
if type epatch_user >& /dev/null; then
cd "${S}"
epatch_user |
|
You should use pushd/popd here, or use a sub-shell, so that the working directory is not changed for subsequent steps. | Petross404 wrote: | | Code: | | touch /tmp/profile #This is a global var to indicate if profiling is taking place |
|
Do you have private /tmp for the build user? If not, then this is a bad place to save a flag. Further, looking at the later use, you only clean up if postinst runs. What happens if someone runs emerge --buildpkgonly, or if the build fails? What happens if someone uses emerge --jobs to run more than one package in parallel? | Petross404 wrote: | | Code: | | *app-editor* | *mail-client* | *media* | *www-client* ) |
|
You only need the glob for media. The others should be fine without a glob. | Petross404 wrote: | | Code: | #If this doesn't exist, then it's compiled for the fist time
if [ -d "${PROFILE_DIR}" ] |
|
This test looks backward. The test is inverted relative to the comment. | Petross404 wrote: | | Code: | | CXXFLAGS="${CFLAGS}" |
|
This discards any flags specific to CXXFLAGS, and will add any C-specific flags to CXXFLAGS. It would be better to maintain a variable containing the flags that must be added to both $CFLAGS and $CXXFLAGS, then append that variable to both of them.This is not a valid statement in the script. | Petross404 wrote: | | Code: | | if [[ -z `strings $i | grep profiling` ]]; then |
|
Missing quotes on $i. Also, this could have a false positive if the program includes a string that contains profiling, such as This program is a tool to to process profiling data. |
|
| Back to top |
|
|
Petross404
n00b
Joined: 27 Sep 2016
Posts: 55
|
| Posted: Fri Jul 19, 2019 9:27 am Post subject: |
|
|
@ff11 Thank you, it pointed out several warnings that I fixed.
@Hu
| Quote: |
You should use pushd/popd here, or use a sub-shell, so that the working directory is not changed for subsequent steps.
|
I don't quite understand what do you mean. Is this about
| Code: | PGO_DIR=/var/tmp/pgo #I think this should be quoted
PROFILE_DIR="${PGO_DIR}/${CATEGORY}/${P}" |
or about | Code: | cd "${S}"
epatch_user |
?
Also you were right about the if [[ -d ... ]]. The check and the comments were indeed inverted. Fixed now. The same applies for CXXFLAGS substitution from CFLAGS and this check
| Code: | | if [[ -z $(strings "${i}" | grep profiling) ]]; then |
although I don't have any other idea how to see if ${PN} was indeed compiled with profiling enabled. Maybe it's best to drop this entirely and this will also eliminate the need for a global var like PROFILABLE.
Which brings me to two last questions.
a) Say I need to change a global var inside unpack EBUILD_PHASE. I won't be able to read it's altered value in postinst EBUILD_PHASE. The only workaround seems to be a write operation on stdout or a file. How could I overcome this limitation?
b) Is this implementation sufficient?
| Code: | if [ "${EBUILD_PHASE}" == "unpack" ]
then
if type epatch_user >& /dev/null; then
cd "${S}"
epatch_user
fi
# TODO this is temporary!
elog "USE = ${USE} for ${P}"
elog "IUSE = ${IUSE} for ${P}"
if [[ "$CFLAGS" != *"gdb"* && "$CXXFLAGS" != *"gdb"* ]] && [[ "${IUSE}" != *"pgo"* && "${USE}" != *"pgo"* ]]
then
case "${CATEGORY}" in
app-editor | mail-client | *media* | www-client | sys-devel )
elog "Portage will try to build ${CATEGORY}/${PN} with PGO."
if [ -d "${PROFILE_DIR}" ]
then
elog "Portage will use the analysis that is stored at ${PROFILE_DIR}."
CFLAGS="${CFLAGS} ${CFLAGS_CUSTOM} ${CFLAGS_PROFILE_USE}"
CXXFLAGS="${CXXFLAGS} ${CXXFLAGS_CUSTOM} ${CXXFLAGS_PROFILE_USE}"
LDFLAGS="${LDFLAGS} ${LDFLAGS_CUSTOM}"
else
elog "Running with PGO for the first time. Use ${P} before you recompile it."
CFLAGS="${CFLAGS} ${CFLAGS_CUSTOM} ${CFLAGS_PROFILE_GEN}"
CXXFLAGS="${CXXFLAGS} ${CXXFLAGS_CUSTOM} ${CXXFLAGS_PROFILE_GEN}"
LDFLAGS="${LDFLAGS} ${LDFLAGS_CUSTOM} ${LDFLAGS_PROFILE_GEN}"
fi
;;
*)
elog "It seems portage won't compile ${CATEGORY}/${PN} with PGO."
;;
esac
#PGO might exist in IUSE but the user may want it or not. Print the appropriate message.
elif [[ "${IUSE}" == *"pgo"* ]]
then
local PGO_WARN_MSG="IUSE array of ${CATEGORY}/${PN} contains pgo,"
local PGO_HANDLE_MSG="It is best to leave ${CATEGORY}/${P} ebuild handle this."
if [[ "${USE}" == *"pgo"* ]]
then
ewarn "${PGO_WARN_MSG} and it's enabled."
ewarn "${PGO_HANDLE_MSG}"
elif [[ "${USE}" != *"pgo"* ]]
then
ewarn "${PGO_WARN_MSG} but the user disabled this USE flag."
ewarn "${PGO_HANDLE_MSG}"
fi
fi
fi
|
This produces :
| Code: | USE="pgo" ebuild /usr/portage/sys-devel/gcc/gcc-9.1.0-r1.ebuild unpack
* gcc-9.1.0.tar.xz BLAKE2B SHA512 size ;-) ... [ ok ]
* gcc-9.1.0-patches-1.1.tar.bz2 BLAKE2B SHA512 size ;-) ... [ ok ]
* USE = abi_x86_64 amd64 cxx elibc_glibc fortran go hardened kernel_linux lto multilib nls nptl openmp pgo pie sanitize ssp userland_GNU vtv for gcc-9.1.0
* IUSE = test vanilla +nls +nptl altivec debug +cxx +fortran doc hardened multilib objc pgo objc-gc libssp objc++ +openmp fixed-point go graphite +sanitize +vtv jit +pie +ssp +pch systemtap d lto for gcc-9.1.0
* IUSE array of sys-devel/gcc contains pgo, and it's enabled.
* It is best to leave sys-devel/gcc-9.1.0 ebuild handle this.
>>> Unpacking source...
|
and
| Code: |
USE="-pgo" ebuild /usr/portage/sys-devel/gcc/gcc-9.1.0-r1.ebuild unpack
* gcc-9.1.0.tar.xz BLAKE2B SHA512 size ;-) ... [ ok ]
* gcc-9.1.0-patches-1.1.tar.bz2 BLAKE2B SHA512 size ;-) ... [ ok ]
* USE = abi_x86_64 amd64 cxx elibc_glibc fortran go hardened kernel_linux lto multilib nls nptl openmp pie sanitize ssp userland_GNU vtv for gcc-9.1.0
* IUSE = test vanilla +nls +nptl altivec debug +cxx +fortran doc hardened multilib objc pgo objc-gc libssp objc++ +openmp fixed-point go graphite +sanitize +vtv jit +pie +ssp +pch systemtap d lto for gcc-9.1.0
* IUSE array of sys-devel/gcc contains pgo but the user disabled this USE flag.
* It's best to leave sys-devel/gcc-9.1.0 ebuild handle this.
>>> Unpacking source...
|
I think that all cases have been covered!
Thank you for your precious hints and help. |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 22730
|
| Posted: Sat Jul 20, 2019 1:25 am Post subject: |
|
|
| Petross404 wrote: | @Hu | Hu wrote: | You should use pushd/popd here, or use a sub-shell, so that the working directory is not changed for subsequent steps.
|
I don't quite understand what do you mean. Is this about | Code: | PGO_DIR=/var/tmp/pgo #I think this should be quoted
PROFILE_DIR="${PGO_DIR}/${CATEGORY}/${P}" |
or about | Code: | cd "${S}"
epatch_user |
? |
The latter. | Petross404 wrote: | | a) Say I need to change a global var inside unpack EBUILD_PHASE. I won't be able to read it's altered value in postinst EBUILD_PHASE. The only workaround seems to be a write operation on stdout or a file. How could I overcome this limitation? |
I am not aware of a good general purpose solution for this. Remember that writing a file outside the work area is not guaranteed to work, because you cannot assume that the postinst phase will run in the same filesystem as the build, nor that the administrator will not have done general cleanup between those phases. | Petross404 wrote: |
b) Is this implementation sufficient?
| Code: | cd "${S}"
epatch_user |
|
Use instead: | Code: | | pushd "$S" && epatch_user && popd |
The rest of it looks fine to me. |
|
| Back to top |
|
|
Petross404
n00b
Joined: 27 Sep 2016
Posts: 55
|
| Posted: Sat Jul 20, 2019 11:44 am Post subject: |
|
|
| Thank you very much. Appreciated. |
|
| Back to top |
|
|
Gatak
Apprentice
Joined: 04 Jan 2004
Posts: 174
|
| Posted: Mon Feb 03, 2020 9:19 am Post subject: |
|
|
I got inspired of this thread and thought I could make something similar, but a little more generic and simple.
| Code: | bin/bash
if [ ! -z "$pgo" ]
then
PGO_DIR="/var/tmp/pgo"
PROFILE_DIR="${PGO_DIR}/${CATEGORY}/${P}"
if [ $pgo -eq 1 ]
then
echo "Enabling profile-generation "
COMMON_FLAGS="-O3 -march=native -pipe -fprofile-generate=${PROFILE_DIR}"
CFLAGS="${COMMON_FLAGS}"
CXXFLAGS="${COMMON_FLAGS}"
FCFLAGS="${COMMON_FLAGS}"
FFLAGS="${COMMON_FLAGS}"
LDFLAGS="${LDFLAGS} -fprofile-generate=${PROFILE_DIR}"
# create a profile directory
test -d "${PROFILE_DIR}" || mkdir -p "${PROFILE_DIR}"
chmod 0777 "${PROFILE_DIR}"
rm "${PROFILE_DIR}/*" >/dev/null
elif [ $pgo -eq 2 ]
then
# test -d "${PROFILE_DIR}" || echo "No existing existing profiler data found.
rm "${PROFILE_DIR}/*conftest.gcda"
echo "Using previously created profiled data "
COMMON_FLAGS="-O3 -march=native -pipe -fprofile-use=${PROFILE_DIR} -flto"
CFLAGS="${COMMON_FLAGS}"
CXXFLAGS="${COMMON_FLAGS}"
FCFLAGS="${COMMON_FLAGS}"
FFLAGS="${COMMON_FLAGS}"
LDFLAGS="${LDFLAGS} -fprofile-use=${PROFILE_DIR}"
fi
fi
|
The script is simple. To enable pgo profile run do "pgo=1 emerge some/app". Then you run your app for a while to gather relevant profile data. Next you run "pgo=2 emerge some/app" to re-compile using the profiled data.
I have one issue that the configure phase of the ebuild leaves "conftest.cgda". This must be removed before running "pgo=2". Any input on how to solve this in a nicer way?
Also, sometimes the profile data has wrong permissions so that portage can't access it during pgo=2.
It would also be nice to preserve any cflags set by make.conf and package.env. |
|
| Back to top |
|
|
Rutcha
n00b
Joined: 13 Dec 2019
Posts: 7
Location: Brasil, Latin America
|
| Posted: Sun Jan 09, 2022 12:43 pm Post subject: following instructions no longer works |
|
|
I appreciate your tips and I followed them, but
as of today, I can't get {PN} or {PF} to mean anything. The get ignored, my directory goes like "/var/tmp/pgo/${PN}" or ${PF} or ${CATEGORY}/${P};
none work, they end up only '/var/tmp/pgo' |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 22730
|
|
| Back to top |
|
|
|
Portage & Programming
