Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
PGO hook for bashrc
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
Petross404
n00b
n00b


Joined: 27 Sep 2016
Posts: 55

PostPosted: Thu Jul 18, 2019 2:12 pm    Post subject: PGO hook for bashrc Reply with quote

I wrote a hook in bashrc for PGO to apply different cflags based on whether gcda files exist or not:

Code:

#!/bin/bash
PGO_DIR=/var/tmp/pgo
PROFILE_DIR="${PGO_DIR}/${CATEGORY}/${P}"
PROFILABLE=0
CFLAGS_PROFILE_GEN="-fprofile-generate=${PROFILE_DIR} -fprofile-arcs -fvpt"
CFLAGS_PROFILE_USE="-fprofile-use=${PROFILE_DIR} -fprofile-correction" #Multithreading is corrupting gcda files, so we "correct" them
LDFLAGS_PROFILE_GEN="-fprofile-arcs"

CFLAGS_CUSTOM=""
LDFLAGS_CUSTOM=""

if [ "${EBUILD_PHASE}" == "pre_src_unpack" ]
then
   if type epatch_user >& /dev/null; then
      cd "${S}"
      epatch_user
   fi

   touch /tmp/profile        #This is a global var to indicate if profiling is taking place
   case "${CATEGORY}" in     #Enable pgo for some categories
      *app-editor* | *mail-client* | *media* | *www-client* )

                #If the package isn't using debug cflags or a pgo USE flag then...
      if [[ "$CFLAGS" != *"gdb"* || "${IUSE}" != "*pgo*" ]]; then
         echo "1" > /tmp/profile
         elog "Portage will try to profile " "${PN} ..."

                        #If this doesn't exist, then it's compiled for the fist time
         if [ -d "${PROFILE_DIR}" ]
         then
            elog "Running with PGO for the first time. Use ${P} before you recompile it."
              CFLAGS="${CFLAGS} ${CFLAGS_CUSTOM} ${CFLAGS_PROFILE_USE}"
            LDFLAGS="${LDFLAGS} ${LDFLAGS_CUSTOM}"
         else
            elog "Portage will use the analysis that is stored at ${PROFILE_DIR}."
             CFLAGS="${CFLAGS} ${CFLAGS_CUSTOM} ${CFLAGS_PROFILE_GEN}"
            LDFLAGS="${LDFLAGS} ${LDFLAGS_CUSTOM} ${LDFLAGS_PROFILE_GEN}"
         fi
         CXXFLAGS="${CFLAGS}"
      fi
      if [[ "$IUSE" == "*pgo*" ]]; then
         elog "PGO is in USE flags, no need to use this bashrc."
      fi
   esac
fi

....

if [ "${EBUILD_PHASE}" == "postinst" ]
then
   #if PGO was applied, test if it succeeded and chown the PROFILE_DIR
   read PROFILABLE < /tmp/profile
   if [[ "$PROFILABLE" == 1 ]]  ; then
      for i in `equery f ${PN} | grep "bin/"`;
      do
         if [[ -z `strings $i | grep profiling` ]]; then
            ewarn "This binary isn't making use of PGO while it's compiled with it:" $i
         else
            elog  "This binary is making use of PGO:" $i
         fi
      done
        fi

        #Normal user should be able to "write" here the statistical analysis needed for PGO
        # TODO: What about other users?
   chown -R petros:petros "${PGO_DIR}"
   rm /tmp/profile
   
   #Ok, let's update the database
   echo ":: Calling updatedb to update its database"
fi


Any comments and critic is welcome.
Back to top
View user's profile Send private message
ff11
l33t
l33t


Joined: 10 Mar 2014
Posts: 664

PostPosted: Thu Jul 18, 2019 3:45 pm    Post subject: Reply with quote

Hi! You should make a parse on this thing first. One good parse and check is the site https://www.shellcheck.net
Or you could install the dev-util/shellcheck-bin and use the command (for the same result):
Code:
shellcheck --shell=bash script_file
Back to top
View user's profile Send private message
Hu
Administrator
Administrator


Joined: 06 Mar 2007
Posts: 22771

PostPosted: Fri Jul 19, 2019 1:07 am    Post subject: Re: PGO hook for bashrc Reply with quote

Petross404 wrote:
Code:
#!/bin/bash
PGO_DIR=/var/tmp/pgo
PROFILE_DIR="${PGO_DIR}/${CATEGORY}/${P}"
PROFILABLE=0

CFLAGS_CUSTOM=""
LDFLAGS_CUSTOM=""

if [ "${EBUILD_PHASE}" == "pre_src_unpack" ]
then
   if type epatch_user >& /dev/null; then
      cd "${S}"
      epatch_user
You should use pushd/popd here, or use a sub-shell, so that the working directory is not changed for subsequent steps.
Petross404 wrote:
Code:
   touch /tmp/profile        #This is a global var to indicate if profiling is taking place
Do you have private /tmp for the build user? If not, then this is a bad place to save a flag. Further, looking at the later use, you only clean up if postinst runs. What happens if someone runs emerge --buildpkgonly, or if the build fails? What happens if someone uses emerge --jobs to run more than one package in parallel?
Petross404 wrote:
Code:
      *app-editor* | *mail-client* | *media* | *www-client* )
You only need the glob for media. The others should be fine without a glob.
Petross404 wrote:
Code:
                        #If this doesn't exist, then it's compiled for the fist time
         if [ -d "${PROFILE_DIR}" ]
This test looks backward. The test is inverted relative to the comment.
Petross404 wrote:
Code:
         CXXFLAGS="${CFLAGS}"
This discards any flags specific to CXXFLAGS, and will add any C-specific flags to CXXFLAGS. It would be better to maintain a variable containing the flags that must be added to both $CFLAGS and $CXXFLAGS, then append that variable to both of them.
Petross404 wrote:
Code:
....
This is not a valid statement in the script.
Petross404 wrote:
Code:
         if [[ -z `strings $i | grep profiling` ]]; then
Missing quotes on $i. Also, this could have a false positive if the program includes a string that contains profiling, such as This program is a tool to to process profiling data.
Back to top
View user's profile Send private message
Petross404
n00b
n00b


Joined: 27 Sep 2016
Posts: 55

PostPosted: Fri Jul 19, 2019 9:27 am    Post subject: Reply with quote

@ff11 Thank you, it pointed out several warnings that I fixed.

@Hu
Quote:

You should use pushd/popd here, or use a sub-shell, so that the working directory is not changed for subsequent steps.

I don't quite understand what do you mean. Is this about
Code:
PGO_DIR=/var/tmp/pgo #I think this should be quoted
PROFILE_DIR="${PGO_DIR}/${CATEGORY}/${P}"


or about
Code:
cd "${S}"
      epatch_user
?

Also you were right about the if [[ -d ... ]]. The check and the comments were indeed inverted. Fixed now. The same applies for CXXFLAGS substitution from CFLAGS and this check
Code:
if [[ -z $(strings "${i}" | grep profiling) ]]; then

although I don't have any other idea how to see if ${PN} was indeed compiled with profiling enabled. Maybe it's best to drop this entirely and this will also eliminate the need for a global var like PROFILABLE.

Which brings me to two last questions.

a) Say I need to change a global var inside unpack EBUILD_PHASE. I won't be able to read it's altered value in postinst EBUILD_PHASE. The only workaround seems to be a write operation on stdout or a file. How could I overcome this limitation?

b) Is this implementation sufficient?
Code:
if [ "${EBUILD_PHASE}" == "unpack" ]
then
   if type epatch_user >& /dev/null; then
      cd "${S}"
      epatch_user
   fi

   # TODO this is temporary!
   elog "USE = ${USE} for ${P}"
   elog "IUSE = ${IUSE} for ${P}"

   if [[ "$CFLAGS" != *"gdb"* && "$CXXFLAGS" != *"gdb"* ]] && [[ "${IUSE}" != *"pgo"* && "${USE}" != *"pgo"* ]]
   then
      case "${CATEGORY}" in
         app-editor | mail-client | *media* | www-client | sys-devel )
         
            elog "Portage will try to build ${CATEGORY}/${PN} with PGO."

            if [ -d "${PROFILE_DIR}" ]
            then
               elog "Portage will use the analysis that is stored at ${PROFILE_DIR}."
               CFLAGS="${CFLAGS} ${CFLAGS_CUSTOM} ${CFLAGS_PROFILE_USE}"
               CXXFLAGS="${CXXFLAGS} ${CXXFLAGS_CUSTOM} ${CXXFLAGS_PROFILE_USE}"
               LDFLAGS="${LDFLAGS} ${LDFLAGS_CUSTOM}"
            else
               elog "Running with PGO for the first time. Use ${P} before you recompile it."
               CFLAGS="${CFLAGS} ${CFLAGS_CUSTOM} ${CFLAGS_PROFILE_GEN}"
               CXXFLAGS="${CXXFLAGS} ${CXXFLAGS_CUSTOM} ${CXXFLAGS_PROFILE_GEN}"
               LDFLAGS="${LDFLAGS} ${LDFLAGS_CUSTOM} ${LDFLAGS_PROFILE_GEN}"
            fi
            ;;
         *)
            elog "It seems portage won't compile ${CATEGORY}/${PN} with PGO."
            ;;
      esac
   #PGO might exist in IUSE but the user may want it or not. Print the appropriate message.
   elif [[ "${IUSE}" == *"pgo"* ]]
   then
      local PGO_WARN_MSG="IUSE array of ${CATEGORY}/${PN} contains pgo,"
      local PGO_HANDLE_MSG="It is best to leave ${CATEGORY}/${P} ebuild handle this."

      if [[ "${USE}" == *"pgo"* ]]
      then
         ewarn "${PGO_WARN_MSG} and it's enabled."
         ewarn "${PGO_HANDLE_MSG}"
      elif [[ "${USE}" != *"pgo"* ]]
      then
         ewarn "${PGO_WARN_MSG} but the user disabled this USE flag."
         ewarn "${PGO_HANDLE_MSG}"
      fi
   fi
fi

This produces :
Code:
USE="pgo" ebuild /usr/portage/sys-devel/gcc/gcc-9.1.0-r1.ebuild unpack
 * gcc-9.1.0.tar.xz BLAKE2B SHA512 size ;-) ...                                             [ ok ]
 * gcc-9.1.0-patches-1.1.tar.bz2 BLAKE2B SHA512 size ;-) ...                         [ ok ]
 * USE = abi_x86_64 amd64 cxx elibc_glibc fortran go hardened kernel_linux lto multilib nls nptl openmp pgo pie sanitize ssp userland_GNU vtv for gcc-9.1.0
 * IUSE = test vanilla +nls +nptl altivec debug +cxx +fortran doc hardened multilib objc pgo objc-gc libssp objc++ +openmp fixed-point go graphite +sanitize +vtv jit +pie +ssp +pch systemtap d lto for gcc-9.1.0
 * IUSE array of sys-devel/gcc contains pgo, and it's enabled.
 * It is best to leave sys-devel/gcc-9.1.0 ebuild handle this.
>>> Unpacking source...

and
Code:

USE="-pgo" ebuild /usr/portage/sys-devel/gcc/gcc-9.1.0-r1.ebuild unpack
 * gcc-9.1.0.tar.xz BLAKE2B SHA512 size ;-) ...                                                 [ ok ]
 * gcc-9.1.0-patches-1.1.tar.bz2 BLAKE2B SHA512 size ;-) ...                             [ ok ]
 * USE = abi_x86_64 amd64 cxx elibc_glibc fortran go hardened kernel_linux lto multilib nls nptl openmp pie sanitize ssp userland_GNU vtv for gcc-9.1.0
 * IUSE = test vanilla +nls +nptl altivec debug +cxx +fortran doc hardened multilib objc pgo objc-gc libssp objc++ +openmp fixed-point go graphite +sanitize +vtv jit +pie +ssp +pch systemtap d lto for gcc-9.1.0
 * IUSE array of sys-devel/gcc contains pgo but the user disabled this USE flag.
 * It's best to leave sys-devel/gcc-9.1.0 ebuild handle this.
>>> Unpacking source...

I think that all cases have been covered!

Thank you for your precious hints and help.
Back to top
View user's profile Send private message
Hu
Administrator
Administrator


Joined: 06 Mar 2007
Posts: 22771

PostPosted: Sat Jul 20, 2019 1:25 am    Post subject: Reply with quote

Petross404 wrote:
@Hu
Hu wrote:
You should use pushd/popd here, or use a sub-shell, so that the working directory is not changed for subsequent steps.
I don't quite understand what do you mean. Is this about
Code:
PGO_DIR=/var/tmp/pgo #I think this should be quoted
PROFILE_DIR="${PGO_DIR}/${CATEGORY}/${P}"
or about
Code:
cd "${S}"
      epatch_user
?
The latter.
Petross404 wrote:
a) Say I need to change a global var inside unpack EBUILD_PHASE. I won't be able to read it's altered value in postinst EBUILD_PHASE. The only workaround seems to be a write operation on stdout or a file. How could I overcome this limitation?
I am not aware of a good general purpose solution for this. Remember that writing a file outside the work area is not guaranteed to work, because you cannot assume that the postinst phase will run in the same filesystem as the build, nor that the administrator will not have done general cleanup between those phases.
Petross404 wrote:

b) Is this implementation sufficient?
Code:
      cd "${S}"
      epatch_user
Use instead:
Code:
pushd "$S" && epatch_user && popd
The rest of it looks fine to me.
Back to top
View user's profile Send private message
Petross404
n00b
n00b


Joined: 27 Sep 2016
Posts: 55

PostPosted: Sat Jul 20, 2019 11:44 am    Post subject: Reply with quote

Thank you very much. Appreciated.
Back to top
View user's profile Send private message
Gatak
Apprentice
Apprentice


Joined: 04 Jan 2004
Posts: 174

PostPosted: Mon Feb 03, 2020 9:19 am    Post subject: Reply with quote

I got inspired of this thread and thought I could make something similar, but a little more generic and simple.

Code:
bin/bash

if  [ ! -z "$pgo" ]
then
   PGO_DIR="/var/tmp/pgo"
   PROFILE_DIR="${PGO_DIR}/${CATEGORY}/${P}"
   if [ $pgo -eq 1 ]
   then
      echo "Enabling profile-generation "
      COMMON_FLAGS="-O3 -march=native -pipe -fprofile-generate=${PROFILE_DIR}"
      CFLAGS="${COMMON_FLAGS}"
      CXXFLAGS="${COMMON_FLAGS}"
      FCFLAGS="${COMMON_FLAGS}"
      FFLAGS="${COMMON_FLAGS}"
      LDFLAGS="${LDFLAGS} -fprofile-generate=${PROFILE_DIR}"
      # create a profile directory
      test -d "${PROFILE_DIR}" || mkdir -p "${PROFILE_DIR}"
      chmod 0777 "${PROFILE_DIR}"
      rm "${PROFILE_DIR}/*" >/dev/null
   elif  [ $pgo -eq 2 ]
   then
#      test -d "${PROFILE_DIR}" || echo "No existing existing profiler data found.
      rm "${PROFILE_DIR}/*conftest.gcda"
      echo "Using previously created profiled data "
      COMMON_FLAGS="-O3 -march=native -pipe -fprofile-use=${PROFILE_DIR} -flto"
      CFLAGS="${COMMON_FLAGS}"
      CXXFLAGS="${COMMON_FLAGS}"
      FCFLAGS="${COMMON_FLAGS}"
      FFLAGS="${COMMON_FLAGS}"
      LDFLAGS="${LDFLAGS} -fprofile-use=${PROFILE_DIR}"

   fi
fi


The script is simple. To enable pgo profile run do "pgo=1 emerge some/app". Then you run your app for a while to gather relevant profile data. Next you run "pgo=2 emerge some/app" to re-compile using the profiled data.

I have one issue that the configure phase of the ebuild leaves "conftest.cgda". This must be removed before running "pgo=2". Any input on how to solve this in a nicer way?

Also, sometimes the profile data has wrong permissions so that portage can't access it during pgo=2.

It would also be nice to preserve any cflags set by make.conf and package.env.
Back to top
View user's profile Send private message
Rutcha
n00b
n00b


Joined: 13 Dec 2019
Posts: 7
Location: Brasil, Latin America

PostPosted: Sun Jan 09, 2022 12:43 pm    Post subject: following instructions no longer works Reply with quote

I appreciate your tips and I followed them, but
as of today, I can't get {PN} or {PF} to mean anything. The get ignored, my directory goes like "/var/tmp/pgo/${PN}" or ${PF} or ${CATEGORY}/${P};

none work, they end up only '/var/tmp/pgo'
Back to top
View user's profile Send private message
Hu
Administrator
Administrator


Joined: 06 Mar 2007
Posts: 22771

PostPosted: Sun Jan 09, 2022 5:50 pm    Post subject: Reply with quote

For future readers, Rutcha also posted {PN} variable no more and 'Ebuild variable' vanishes during normal portage emerge about this same subject. In the free thread, it is made clear that the variables are being used in make.conf, where their usage is invalid and has never worked. The instructions here did not rot over time.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum