| View previous topic :: View next topic |
| Author |
Message |
0x4a47
n00b
Joined: 24 Mar 2004
Posts: 18
|
 Posted: Fri Nov 20, 2009 1:18 pm Post subject: |
 |
|
| ppurka wrote: |
Very good! I think I don't have any fetch restricted files in my system which is why I never ran into this bug (I recently did a quietemerge -e world). I will update the script soon to catch such events.
|
thanks very much! another fetch restricted package i have e.g. is truecrypt if you need more test cases 
| Quote: | | That is something I was unaware of. The TMPFILE I create is created with the id of root and uses default permissions which gives only read attribute to other users. If you have another malicious user with root privileges then it is really unfortunate. I am against changing this to mktemp since it will create a bunch of files (over time) in /tmp which I want to avoid. Secondly there is also the case that the users will have to remember to read my output carefully everytime. |
this has nothing to do with other root users but normal users doing mischief. i know that it is unlikey for such an attack for standard setups but imagine a server where other people have access, but not root rights. a malicous user can point a symlink from /tmp/output.log to /etc/shadow or some other only by root accessible files. if root runs quietemerge, it will overwrite /etc/shadow and render the system unusable, so it mainly is a denial of service attack. if a user is able to (somehow) define the output of emerge or quietemerge it might be possible to create another user entry in /etc/shadow for that example. you can test this, but backup your files before
another thing which makes the attack more hypothetical is that if the /tmp/output.log already exists, a normal user can't overwrite it, because you created it with correct permissions.
| Quote: |
What I will do though is delete the file just before redirecting to it. That should mitigate any security concerns.
[...]
I guess I can use mktemp for some other tmpfiles in the script. That I will do. |
i don't think that your solution to delete the file everytime is a good one because it gives a potential attacker more opportunity to attack (although again unlikely, i know). what you could do (also mentioned in link i posted) is check whether the file exists and is a sym- or hardlink and if it is ok, then write to it. |
|
| Back to top |
|
 |
ppurka
Advocate
Joined: 26 Dec 2004
Posts: 3256
|
| Posted: Sat Nov 21, 2009 5:48 am Post subject: |
|
|
Update to version 20091121. Changelog- Bug fix: Recognize failure due to fetch restriction
- Bug fix: Check temporary files and delete them for security sake. (Thanks to 0x4a47 for pointing out the security risks).
I have also changed the names of the temporary files so that they contain the name of the script.
- Bug fix: If emerge fails on a package very early in the emerge process, the script fails to even recognize that such a package was being emerged. Changed the sleep to "sleep 1" in the first while loop.
@0x4a47: I am simply deleting the temporary files on a fresh run of the script. Even if they were symlinks or hard links, I don't care,- those will be wiped out. This helps me in two ways:
1. I don't have to care about earlier files
2. I can avoid mktemp and the consequent tons of temporary files in /tmp (accumulated over time) if the script is exited suddenly. This is taken care of to some extent at present if the files are not arbitrarily named as mktemp would do.
_________________
emerge --quiet redefined | E17 vids: I, II | Now using kde5 | e is unstable :-/ |
|
| Back to top |
|
|
Gibbo_07
n00b
Joined: 15 Nov 2009
Posts: 38
Location: Brisbane, AU
|
| Posted: Mon Nov 30, 2009 7:56 am Post subject: |
|
|
Great little tool you have here ppurka (&co)
Question however, I have been using a (2.5gb) tmpfs for /var/tmp/portage mounted at boot which this way I need to remember to unmount before merging kde, openoffice etc. With your tool here I decided to use the .config to mount my tmpfs and assumed it would unmount the tmpfs when done but apparently not? My ideal use of this would be for it to mount the tmpfs on start and unmount when finished.
A feature suggestion perhaps would be to add a switch to disable the mounting of tmpfs when not wanted on the fly per emerge? Or if your preference is to leave the tmpfs mounted afterwards by default, add switch to unmount when done. Either way would be great
Thanks! |
|
| Back to top |
|
|
ppurka
Advocate
Joined: 26 Dec 2004
Posts: 3256
|
| Posted: Mon Nov 30, 2009 3:15 pm Post subject: |
|
|
| Gibbo_07 wrote: | Great little tool you have here ppurka (&co)
Question however, I have been using a (2.5gb) tmpfs for /var/tmp/portage mounted at boot which this way I need to remember to unmount before merging kde, openoffice etc. With your tool here I decided to use the .config to mount my tmpfs and assumed it would unmount the tmpfs when done but apparently not? My ideal use of this would be for it to mount the tmpfs on start and unmount when finished. |
That is not true. The script should unmount tmpfs when it is done. The only case when it doesn't unmount tmpfs is when there has been some error in emerge. The reason for not unmounting in that case is so that you can collect error information, etc and investigate further and/or submit a bug report.
Even if you have mounted tmpfs "manually", but have MOUNT_TMPFS=0, the script will not unmount tmpfs. If you have MOUNT_TMPFS=1 in this case, then the script will tell you " * tmpfs is already mounted" just before the emerge starts, and it will unmount the tmpfs after the emerge is successful.
| Quote: | A feature suggestion perhaps would be to add a switch to disable the mounting of tmpfs when not wanted on the fly per emerge? Or if your preference is to leave the tmpfs mounted afterwards by default, add switch to unmount when done. Either way would be great
Thanks! |
Actually the opposite is possible right now If you have not modified MOUNT_TMPFS in the config file (~/.config/quietemerge.config) then you can temporarily enable mounting of tmpfs by running quietemerge as | Code: | | MOUNT_TMPFS=1 quietemerge <rest of the arguments> |
EDIT: I hope you now have an understanding of how the script handles mounting/unmounting of tmpfs. If you are still having this problem could you identify some package which takes a few seconds to emerge (such as eselect) and run the following command: | Code: | | bash -x quietemerge eselect |
and then copy all the output in the terminal and provide me the output? Also, could you provide me the output of while the emerge is running!
_________________
emerge --quiet redefined | E17 vids: I, II | Now using kde5 | e is unstable :-/ |
|
| Back to top |
|
|
Gibbo_07
n00b
Joined: 15 Nov 2009
Posts: 38
Location: Brisbane, AU
|
| Posted: Mon Nov 30, 2009 6:15 pm Post subject: |
|
|
| ppurka wrote: |
That is not true. The script should unmount tmpfs when it is done. The only case when it doesn't unmount tmpfs is when there has been some error in emerge. The reason for not unmounting in that case is so that you can collect error information, etc and investigate further and/or submit a bug report.
Even if you have mounted tmpfs "manually", but have MOUNT_TMPFS=0, the script will not unmount tmpfs. If you have MOUNT_TMPFS=1 in this case, then the script will tell you " * tmpfs is already mounted" just before the emerge starts, and it will unmount the tmpfs after the emerge is successful. |
That would make sense, the confusion lies in that this is not the behaviour I have observed I have disabled my manual fstab entry for the portage tmpfs and instead have set the MOUNT_TMPFS=1. I have merged many packages (without error) and have always had the tmpfs stay mounted afterwards - I thought it was a "feature" hah. I umount /var/tmp/portage after each merge, watch it be remounted and used with your script, but never unmounted by itself.
| Quote: | Actually the opposite is possible right now If you have not modified MOUNT_TMPFS in the config file (~/.config/quietemerge.config) then you can temporarily enable mounting of tmpfs by running quietemerge as | Code: | | MOUNT_TMPFS=1 quietemerge <rest of the arguments> |
|
Thanks makes sense now, I assume it is also possible to | Code: | | MOUNT_TMPFS=0 quietemerge <x> |
correct? Just what I need.
| Quote: | EDIT: I hope you now have an understanding of how the script handles mounting/unmounting of tmpfs. If you are still having this problem could you identify some package which takes a few seconds to emerge (such as eselect) and run the following command: | Code: | | bash -x quietemerge eselect |
and then copy all the output in the terminal and provide me the output? Also, could you provide me the output of while the emerge is running! |
So yes I am still having the problem so here's the info (using your example to keep it simple)
| Code: |
gentoobox gibbo # bash -x quietemerge eselect
+ blue='\x1b[1;34m'
+ bold='\x1b[1m'
+ cyan='\x1b[1;36m'
+ green='\x1b[1;32m'
+ normal='\x1b[0m'
+ pink='\x1b[1;35m'
+ red='\x1b[1;31m'
+ reverse='\x1b[7m'
+ underline='\x1b[4m'
+ yellow='\x1b[1;33m'
+ config_dir=/root/.config
+ config_file=quietemerge.config
+ [[ -z eselect ]]
+ [[ -z eselect ]]
+ [[ eselect = -h* ]]
+ [[ eselect = --h* ]]
+ [[ -f /root/.config/quietemerge.config ]]
+ . /root/.config/quietemerge.config
++ MOUNT_TMPFS=1
++ TMPFS_SIZE=60%
++ TXT_UPDATER=etc-update
++ GUI_UPDATER=etc-update
++ SHOW_MERGE_TIME=1
+ TMPFILE=/tmp/quietemerge.output.log
+ tmpE=/tmp/quietemerge.tmpE.0
+ tmpT=/tmp/quietemerge.tmpT
+ : 5
+ : 1
+ : 1
+ _num=0
+ cmdline=
+ e_opt_ask=0
+ e_opt_pretend=0
+ e_opt_verbose=
+ eta=0
+ exit_status=(0 0)
+ my_len=19
+ PID=
+ pkg=
+ trap 'exit_prog 1' 1 2 5 15
+ which genlop
+ [[ -z eselect ]]
+ case "$1" in
+ cmdline=' eselect'
+ shift
+ [[ -z '' ]]
+ secure_file /tmp/quietemerge.tmpE.0
+ [[ -e /tmp/quietemerge.tmpE.0 ]]
+ centered_output dash ' Pretended emerge '
++ tput cols
+ local cols=104
++ echo ' Pretended emerge '
++ sed -e 's#\\x1[bB]\[[^m]\+m##g'
+ local 'len= Pretended emerge '
+ len=18
+ [[ 18 -ge 104 ]]
+ printf -vBLANK %104s ''
+ local D_ASH=--------------------------------------------------------------------------------------------------------
+ local begin=43
+ local end=43
+ [[ -z Pretended emerge ]]
+ [[ dash = \d\a\s\h ]]
+ echo -e '-------------------------------------------\x1b[7m Pretended emerge \x1b[0m-------------------------------------------'
------------------------------------------- Pretended emerge -------------------------------------------
+ emerge --color=y -p eselect
+ tee -i /tmp/quietemerge.tmpE.0
These are the packages that would be merged, in order:
Calculating dependencies... done!
[ebuild R ] app-admin/eselect-1.2.8
+ status=0
+ sed -i -e 's/[[^m]*m//g' /tmp/quietemerge.tmpE.0
+ [[ 0 -eq 0 ]]
+ echo
+ infor 'Estimating total update time ...'
++ tput el
+ echo -ne ' \x1b[1;33m*\x1b[0m Estimating total update time ...\r'
++ cat /tmp/quietemerge.tmpE.0me ...
++ genlop -npq
++ sed -n -e '/Estimated update time/s/^.*: \(.*\)$/\1/p'
+ centered_output blank ' Total ETA: less than a minute. '
++ tput cols
+ local cols=104
++ echo ' Total ETA: less than a minute. '
++ sed -e 's#\\x1[bB]\[[^m]\+m##g'
+ local 'len= Total ETA: less than a minute. '
+ len=32
+ [[ 32 -ge 104 ]]
+ printf -vBLANK %104s ''
+ local D_ASH=--------------------------------------------------------------------------------------------------------
+ local begin=36
+ local end=36
+ [[ -z Total ETA: less than a minute. ]]
+ [[ blank = \d\a\s\h ]]
+ echo -e ' \x1b[7m Total ETA: less than a minute. \x1b[0m '
Total ETA: less than a minute.
+ echo
+ [[ 0 -eq 1 ]]
+ [[ 0 -eq 0 ]]
+ LC_ALL=C
+ grep -q '^\[ebuild[ ]*I' /tmp/quietemerge.tmpE.0
+ LC_ALL=C
+ grep -q '^\[blocks[ ]*B' /tmp/quietemerge.tmpE.0
+ LC_ALL=C
+ grep -q '^\[blocks[ ]*b' /tmp/quietemerge.tmpE.0
+ LC_ALL=C
+ grep -qE '^\[ebuild[^]]+F' /tmp/quietemerge.tmpE.0
+ [[ 0 -ne 0 ]]
+ unset status
+ sed -i -e '/^\[ebuild/!d' /tmp/quietemerge.tmpE.0
+ fix_edo
++ portageq envvar EMERGE_DEFAULT_OPTS
+ local edo=
+ local new_edo
+ [[ -n '' ]]
+ return
+ [[ 0 -eq 1 ]]
++ wc -l /tmp/quietemerge.tmpE.0
+ _total_num='1 /tmp/quietemerge.tmpE.0'
+ _total_num=1
+ mount_tmpfs
+ [[ 1 -ne 1 ]]
++ portageq envvar PORTAGE_TMPDIR
+ local var_tmp=/var/tmp/portage
+ mount
+ grep -q -s -w 'tmpfs on /var/tmp/portage'
+ [[ -n 60% ]]
+ mount -t tmpfs -o size=60% tmpfs /var/tmp/portage
+ [[ 0 -ne 0 ]]
+ [[ 1 -eq 1 ]]
+ secure_file /tmp/quietemerge.tmpT
+ [[ -e /tmp/quietemerge.tmpT ]]
+ secure_file /tmp/quietemerge.output.log
+ [[ -e /tmp/quietemerge.output.log ]]
+ rm -f /tmp/quietemerge.output.log
+ PID=24891
+ echo
+ emerge -q eselect
+ [[ '' = *--resume* ]]
+ centered_output dash ' Starting emerge '
++ tput cols
+ local cols=104
++ echo ' Starting emerge '
++ sed -e 's#\\x1[bB]\[[^m]\+m##g'
+ local 'len= Starting emerge '
+ len=17
+ [[ 17 -ge 104 ]]
+ printf -vBLANK %104s ''
+ local D_ASH=--------------------------------------------------------------------------------------------------------
+ local begin=44
+ local end=43
+ [[ -z Starting emerge ]]
+ [[ dash = \d\a\s\h ]]
+ echo -e '--------------------------------------------\x1b[7m Starting emerge \x1b[0m-------------------------------------------'
-------------------------------------------- Starting emerge -------------------------------------------
+ echo
+ ps -p 24891
+ LC_ALL=C
+ grep -qsE -m1 '(Emerging|Installing)' /tmp/quietemerge.output.log
+ sleep 1
+ ps -p 24891
+ LC_ALL=C
+ grep -qsE -m1 '(Emerging|Installing)' /tmp/quietemerge.output.log
+ sleep 1
+ ps -p 24891
+ LC_ALL=C
+ grep -qsE -m1 '(Emerging|Installing)' /tmp/quietemerge.output.log
+ ps -p 24891
++ tac /tmp/quietemerge.output.log
++ LC_ALL=C
++ grep -E -m1 '(Emerging|Installing)'
+ tmp='>>> Emerging (1 of 1) app-admin/eselect-1.2.8'
++ sed -n -e 's/.*) \([^ ]*\)[ ]\?.*$/\1/p'
++ echo '>>> Emerging (1 of 1) app-admin/eselect-1.2.8'
+ pkg_tmp=app-admin/eselect-1.2.8
+ [[ app-admin/eselect-1.2.8 != '' ]]
+ [[ -n '' ]]
+ pkg=app-admin/eselect-1.2.8
+ len=23
+ update_term_title
+ [[ -n :0 ]]
+ [[ -f /tmp/quietemerge.tmpE.0 ]]
++ wc -l /tmp/quietemerge.tmpE.0
+ _num='1 /tmp/quietemerge.tmpE.0'
+ _num=1
++ cat /tmp/quietemerge.tmpE.0
++ genlop -npq
++ sed -n -e '/Estimated update time/s/^.*: \(.*\)$/\1/p'
+ eta='less than a minute.'
+ echo -ne '\x1b]0;1/1: eselect-1.2.8, Total ETA: less than a minute.\007'
++ tput cols
+ cols=104
+ printf -vBLANK %104s ''
++ genlop -npq
++ sed -n -e '/Estimated update time/s/^.*: \(.*\)$/\1/p'
++ sed -n -e '/app-admin\/eselect-1.2.8/p' /tmp/quietemerge.tmpE.0
+ pkg_tmp='less than a minute.'
+ total_len=70
+ [[ >>> Emerging (1 of 1) app-admin/eselect-1.2.8 = *Emerging* ]]
+ [[ 70 -lt 104 ]]
++ sed -n '/ETA: /s/^[ ]*ETA:[ ]*//p'
++ genlop -nqc
+ tmp=
+ infolr infor 'Emerging \x1b[1;36mapp-admin/eselect-1.2.8\x1b[0m. ETA:' ''
+ [[ -n '' ]]
++ tput el
+ infor 'Emerging \x1b[1;36mapp-admin/eselect-1.2.8\x1b[0m. ETA:'
++ tput el
+ echo -ne ' \x1b[1;33m*\x1b[0m Emerging \x1b[1;36mapp-admin/eselect-1.2.8\x1b[0m. ETA:\r'
+ returnging app-admin/eselect-1.2.8. ETA:
+ [[ -n :0 ]]
+ echo -ne '\x1b]0;1/1: eselect-1.2.8, Total ETA: less than a minute.\007'
+ sleep 5
+ ps -p 24891
++ LC_ALL=C
++ grep -E -m1 '(Emerging|Installing)'
++ tac /tmp/quietemerge.output.log
+ tmp='>>> Installing (1 of 1) app-admin/eselect-1.2.8'
++ echo '>>> Installing (1 of 1) app-admin/eselect-1.2.8'
++ sed -n -e 's/.*) \([^ ]*\)[ ]\?.*$/\1/p'
+ pkg_tmp=app-admin/eselect-1.2.8
+ [[ app-admin/eselect-1.2.8 != \a\p\p\-\a\d\m\i\n\/\e\s\e\l\e\c\t\-\1\.\2\.\8 ]]
+ [[ >>> Installing (1 of 1) app-admin/eselect-1.2.8 = *Emerging* ]]
++ tput el
+ infor 'Installing \x1b[1;36mapp-admin/eselect-1.2.8\x1b[0m.'
++ tput el
+ echo -ne ' \x1b[1;33m*\x1b[0m Installing \x1b[1;36mapp-admin/eselect-1.2.8\x1b[0m.\r'
+ [[ -n :0 ]]g app-admin/eselect-1.2.8.
+ echo -ne '\x1b]0;1/1: eselect-1.2.8, Total ETA: less than a minute.\007'
+ sleep 5
+ ps -p 24891
+ LC_ALL=C
+ grep -qi 'error.*app-admin/eselect-1.2.8 fail' /tmp/quietemerge.output.log
+ LC_ALL=C
+ grep -q 'ACCESS VIOLATION SUMMARY' /tmp/quietemerge.output.log
+ tail -n 25 /tmp/quietemerge.output.log
+ LC_ALL=C
+ grep -qiE '^\!\!\!.*(abort|error)'
+ tail -n 25 /tmp/quietemerge.output.log
+ LC_ALL=C
+ grep -qiE 'fetch failed.*app-admin/eselect-1.2.8'
+ [[ 0 -ne 0 ]]
+ [[ -n 23 ]]
++ merge_time =app-admin/eselect-1.2.8
++ [[ 1 -eq 1 ]]
++ tail -n 40 /var/log/emerge.log
++ genlop -f /tmp/quietemerge.tmpT -nt =app-admin/eselect-1.2.8
++ tail -n 1
++ sed -n '/merge time:/s/.*: \(.*\)$/\1/p'
+ infolr info 'Install \x1b[1;36mapp-admin/eselect-1.2.8\x1b[0m.' '\x1b[1;32m6 seconds.\x1b[0m'
+ [[ -n \x1b[1;32m6 seconds.\x1b[0m ]]
++ echo 'Install \x1b[1;36mapp-admin/eselect-1.2.8\x1b[0m.'
++ sed -e 's#\\x1[bB]\[[^m]\+m##g'
+ local 'left=Install app-admin/eselect-1.2.8.'
++ echo '\x1b[1;32m6 seconds.\x1b[0m'
++ sed -e 's#\\x1[bB]\[[^m]\+m##g'
+ local 'right=6 seconds.'
+ local middle=57
+ [[ 57 -le 0 ]]
+ info 'Install \x1b[1;36mapp-admin/eselect-1.2.8\x1b[0m. \x1b[1;32m6 seconds.\x1b[0m'
+ echo -e ' \x1b[1;33m*\x1b[0m Install \x1b[1;36mapp-admin/eselect-1.2.8\x1b[0m. \x1b[1;32m6 seconds.\x1b[0m'
* Install app-admin/eselect-1.2.8. 6 seconds.
+ tail /tmp/quietemerge.output.log
+ LC_ALL=C
+ grep -qi preserved
+ [[ -n etc-update ]]
+ tail /tmp/quietemerge.output.log
+ LC_ALL=C
+ grep -qEi -m1 'config.*updat'
+ tail /tmp/quietemerge.output.log
+ LC_ALL=C
+ grep -qEi -m1 'config.*updat'
+ exit_prog 0
+ [[ -f /tmp/quietemerge.tmpE.0 ]]
+ rm -f /tmp/quietemerge.tmpE.0
+ [[ -f /tmp/quietemerge.tmpT ]]
+ rm -f /tmp/quietemerge.tmpT
+ echo
+ [[ -n 24891 ]]
+ ps -p 24891
+ [[ 0 -eq 0 ]]
+ unmount_tmpfs
+ [[ 1 -ne 1 ]]
++ portageq envvar PORTAGE_TMPDIR
+ local var_tmp=/var/tmp/portage
+ [[ 0 -eq 0 ]]
+ mount
+ grep -q -s -w 'tmpfs on /var/tmp/portage'
+ [[ 0 -ne 0 ]]
+ echo
+ exit 0
[b]strait after[/b]
gentoobox gibbo # ls -l /var/tmp/portage
total 0
|
During the emerge (not that it matters when it's not being unmounted  )
| Code: |
gentoobox / # mount | grep tmpfs
rc-svcdir on /lib64/rc/init.d type tmpfs (rw,nosuid,nodev,noexec,relatime,size=1024k,mode=755)
udev on /dev type tmpfs (rw,nosuid,relatime,size=10240k,mode=755)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /mnt/rwstorage/var/tmp/portage type tmpfs (rw,size=60%)
|
Is definitely not being auto unmounted for me. Using the latest version (21112009). Thanks for your time ofcourse.
Edit: perhaps useful, I have /var on a separate partition with symlinks as per the gentoo guide... if this changes anything.... |
|
| Back to top |
|
|
ppurka
Advocate
Joined: 26 Dec 2004
Posts: 3256
|
| Posted: Mon Nov 30, 2009 9:07 pm Post subject: |
|
|
| Gibbo_07 wrote: | Thanks makes sense now, I assume it is also possible to | Code: | | MOUNT_TMPFS=0 quietemerge <x> |
correct? Just what I need. |
Unfortunately no 
This is because I kept the same variables in the config file. It occurred to me a few weeks after I released the script that I should have kept the names of the variables in the config file a bit different, so that command line environment variables would be always be given preference.
I never fixed the config file since no one asked for this feature | Quote: | | Code: |
gentoobox / # mount | grep tmpfs
rc-svcdir on /lib64/rc/init.d type tmpfs (rw,nosuid,nodev,noexec,relatime,size=1024k,mode=755)
udev on /dev type tmpfs (rw,nosuid,relatime,size=10240k,mode=755)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /mnt/rwstorage/var/tmp/portage type tmpfs (rw,size=60%)
|
Is definitely not being auto unmounted for me. Using the latest version (21112009). Thanks for your time ofcourse.
Edit: perhaps useful, I have /var on a separate partition with symlinks as per the gentoo guide... if this changes anything.... |
Exactly! You see, the tmpfs is being mounted on /mnt/rwstorage/var/tmp/portage. Now, if you want your /var/tmp directory to be in some other directory, you should set the variable PORTAGE_TMPDIR in your /etc/make.conf. If you search my script for this variable, you will see that tmpfs is mounted/unmounted depending on the value of this variable. This is important since I definitely do not want to mount some other tmpfs directory. Someone may have multiple tmpfs mounted simultaneously.
To set your PORTAGE_TMPDIR, you should note that for regular portage, it is /var/tmp. For other directory, give the full directory name except for the last directory "portage". You can run | Code: | | portageq envvar PORTAGE_TMPDIR |
to determine what is the current value.
_________________
emerge --quiet redefined | E17 vids: I, II | Now using kde5 | e is unstable :-/ |
|
| Back to top |
|
|
Gibbo_07
n00b
Joined: 15 Nov 2009
Posts: 38
Location: Brisbane, AU
|
| Posted: Tue Dec 01, 2009 3:15 am Post subject: |
|
|
| ppurka wrote: | Unfortunately no
This is because I kept the same variables in the config file. It occurred to me a few weeks after I released the script that I should have kept the names of the variables in the config file a bit different, so that command line environment variables would be always be given preference.
I never fixed the config file since no one asked for this feature |
Pretty please? :d Or how possible would it be to have another sub option (eg. ASKTMPFSMOUNT) in the config file whereby if TMPFS is to be used then makes the script ask for y or n each time (just after the emerge --ask prompt) you perform an emerge whether to use the tmpfs or not? I wouldn't mind having to press 'y/n' again - and would actually make the user think hang on will this emerge fit in my tmpfs? Seems classier than hacking up the cfg or exporting variables each time need to update kde/openoffice etc. I would code it myself if I had the skills
| Quote: | Now, if you want your /var/tmp directory to be in some other directory, you should set the variable PORTAGE_TMPDIR in your /etc/make.conf. If you search my script for this variable, you will see that tmpfs is mounted/unmounted depending on the value of this variable. This is important since I definitely do not want to mount some other tmpfs directory. Someone may have multiple tmpfs mounted simultaneously.
To set your PORTAGE_TMPDIR, you should note that for regular portage, it is /var/tmp. For other directory, give the full directory name except for the last directory "portage". You can run | Code: | | portageq envvar PORTAGE_TMPDIR |
to determine what is the current value. |
Thanks that was the trick. I do find it strange that the script can mount it using the symlink but not unmount it in the same fashion? If I do | Code: | | umount /var/tmp/portage |
it will unmount via the symlink - so is this a feature or? I don't see how the wrong tmpfs can be mounted/unmounted? for all purposes /var/tmp/portage points to my /mnt/rwstorage/var/tmp/portage - other programs have to follow this path in the background I just have a hard time understanding why this script could mount but not unmount without specifically defining it
Well I had a bad experience the first time calling quietemerge after your advice with the path changed. One of those 'what the bleep is happening here' things that don't really make sense. To try explain, on near fresh boot I was testing the mount/unmount so called | Code: | | quietemerge -av k3b |
- it showed only the header and text saying calculating dependancies - then it crashed back to the prompt without actually bringing up the package name showing my use flags etc nor did it ask the yes|no just strait back to bash prompt. Happened 3 times in a row, and THEN it worked... however then half way through emerging the pkg my whole KDE crashed back to entranced!! ok strange... logged back in tried again, same thing! so I decided to alt+f1 to another terminal and check my logs. Used cat /var/log/messages and it was 'corrupt' as in random wingding style characters everywhere. Then my whole bash prompt was doing the same thing! Rebooted and now all seems to work fine. Just weird... not sure if it was the script or emerge or what but something funky went down that's for sure and there isn't anything useful in my logs about it that I can find. Will update if experience it again. |
|
| Back to top |
|
|
ppurka
Advocate
Joined: 26 Dec 2004
Posts: 3256
|
| Posted: Tue Dec 01, 2009 4:05 am Post subject: |
|
|
| Gibbo_07 wrote: |
Thanks that was the trick. I do find it strange that the script can mount it using the symlink but not unmount it in the same fashion? If I do | Code: | | umount /var/tmp/portage |
it will unmount via the symlink - so is this a feature or? I don't see how the wrong tmpfs can be mounted/unmounted? for all purposes /var/tmp/portage points to my /mnt/rwstorage/var/tmp/portage - other programs have to follow this path in the background I just have a hard time understanding why this script could mount but not unmount without specifically defining it |
You see.. you are assuming that you have a mounted tmpfs in /var/tmp/portage. The script doesn't know this. So, the script has to look at the output of mount. And there it sees:
| Code: | | tmpfs on /mnt/rwstorage/var/tmp/portage type tmpfs (rw,size=60%) |
This does not correspond to /var/tmp/portage. The script has to check this, otherwise umount will fail with a nasty error.
| Quote: | Well I had a bad experience the first time calling quietemerge after your advice with the path changed. One of those 'what the bleep is happening here' things that don't really make sense. To try explain, on near fresh boot I was testing the mount/unmount so called | Code: | | quietemerge -av k3b |
- it showed only the header and text saying calculating dependancies - then it crashed back to the prompt without actually bringing up the package name showing my use flags etc nor did it ask the yes|no just strait back to bash prompt. Happened 3 times in a row, and THEN it worked... however then half way through emerging the pkg my whole KDE crashed back to entranced!! ok strange... logged back in tried again, same thing! so I decided to alt+f1 to another terminal and check my logs. Used cat /var/log/messages and it was 'corrupt' as in random wingding style characters everywhere. Then my whole bash prompt was doing the same thing! Rebooted and now all seems to work fine. Just weird... not sure if it was the script or emerge or what but something funky went down that's for sure and there isn't anything useful in my logs about it that I can find. Will update if experience it again. |
I don't know about this. The script runs only emerge, grep, sed, while, tail, mount, umount, echo. None of these commands should interfere with your graphical environment nor should they introduce anything in your system log files. And after the security concerns raised by 0x4a47, I believe I have fixed the last security loophole.
You should investigate what really happened there. I suspect you have a troublesome RAM, or did you run out of hard disk space?
_________________
emerge --quiet redefined | E17 vids: I, II | Now using kde5 | e is unstable :-/ |
|
| Back to top |
|
|
Gibbo_07
n00b
Joined: 15 Nov 2009
Posts: 38
Location: Brisbane, AU
|
| Posted: Tue Dec 01, 2009 7:29 am Post subject: |
|
|
Thanks for the explanation ppurka, makes sense now.
As for the corruption before well no sign of it since so far albeit I havn't merged much since either but your right it's most likely nothing to do with the script, and even worse for me that I am 99% sure the ram is fine and had plenty of both hdd and tmpfs space left during. It definitely had something to do with the merge process itself, crashed during the build/make at the same spot twice in a row. Anyway dribbling on but thanks for the insight think I have to let this anomoly slide into the unsolved category.
Carry on
edit: I am wondering how compatible with portage-2.2 this will be? do you plan to implement support for the new features like sets and concurrent merges etc? cheers |
|
| Back to top |
|
|
ppurka
Advocate
Joined: 26 Dec 2004
Posts: 3256
|
| Posted: Tue Dec 01, 2009 2:33 pm Post subject: |
|
|
| Gibbo_07 wrote: | Thanks for the explanation ppurka, makes sense now.
As for the corruption before well no sign of it since so far albeit I havn't merged much since either but your right it's most likely nothing to do with the script, and even worse for me that I am 99% sure the ram is fine and had plenty of both hdd and tmpfs space left during. It definitely had something to do with the merge process itself, crashed during the build/make at the same spot twice in a row. Anyway dribbling on but thanks for the insight think I have to let this anomoly slide into the unsolved category.
Carry on
edit: I am wondering how compatible with portage-2.2 this will be? do you plan to implement support for the new features like sets and concurrent merges etc? cheers |
I am using portage-2.2 for several months. Sets are already supported (no special effort is needed on my part). --jobs is not supported since I have no idea how to display two simultaneous merges.
_________________
emerge --quiet redefined | E17 vids: I, II | Now using kde5 | e is unstable :-/ |
|
| Back to top |
|
|
ppurka
Advocate
Joined: 26 Dec 2004
Posts: 3256
|
| Posted: Thu Dec 03, 2009 3:23 am Post subject: |
|
|
Update to Version 20091202. Changelog:
* Only one new feature: Allow command line environment variables to override config file variables.
In order to temporarily override the settings present in the config file, prepend the variable with an underscore _ and provide it as an environment variable.
An example where I want to override the config file setting temporarily, and/or provide a temporary setting: | Code: | | _DELAY=2 _MOUNT_TMPFS=1 quietemerge -a eselect |
It is not necessary to update to this version if you do not want to temporarily override the config file variables. There are no other fixes or features. Also, I decided to change the script instead of the config file. Seemed simpler to handle.
_________________
emerge --quiet redefined | E17 vids: I, II | Now using kde5 | e is unstable :-/ |
|
| Back to top |
|
|
doubleagent
Guru
Joined: 15 Apr 2005
Posts: 444
Location: 127.0.0.1
|
| Posted: Mon Dec 14, 2009 6:59 am Post subject: |
|
|
Just noticed this. Nice work. 
_________________
| shickapooka wrote: | | i think they programmed [otw] based on a right-wing jewish-nigger-nazi, his gay, retarded, left-wing love slave with webbed feet, and their three headed cat that poops uncontrollably. the cat is also an apple fanboy |
|
|
| Back to top |
|
|
ppurka
Advocate
Joined: 26 Dec 2004
Posts: 3256
|
| Posted: Mon Dec 14, 2009 7:00 am Post subject: |
|
|
| doubleagent wrote: | | Just noticed this. Nice work. |
Thank you!
_________________
emerge --quiet redefined | E17 vids: I, II | Now using kde5 | e is unstable :-/ |
|
| Back to top |
|
|
ppurka
Advocate
Joined: 26 Dec 2004
Posts: 3256
|
| Posted: Wed Dec 23, 2009 3:29 am Post subject: |
|
|
Update to version 20091222. Changelog:- Bug fix: Catching ACCESS VIOLATION should be less buggy now.
- Genlop should be slightly faster now.
- Change output during next iteration if terminal size changes during compile.
Happy holidays everyone!
_________________
emerge --quiet redefined | E17 vids: I, II | Now using kde5 | e is unstable :-/ |
|
| Back to top |
|
|
devsk
Advocate
Joined: 24 Oct 2003
Posts: 3003
Location: Bay Area, CA
|
| Posted: Wed Dec 30, 2009 10:23 am Post subject: |
|
|
| May I ask what font are you using in the terminal in that screenshot? |
|
| Back to top |
|
|
ppurka
Advocate
Joined: 26 Dec 2004
Posts: 3256
|
| Posted: Wed Dec 30, 2009 4:35 pm Post subject: |
|
|
| devsk wrote: | | May I ask what font are you using in the terminal in that screenshot? |
It is monaco. There is a modified "linuxified" version on the net.
_________________
emerge --quiet redefined | E17 vids: I, II | Now using kde5 | e is unstable :-/ |
|
| Back to top |
|
|
devsk
Advocate
Joined: 24 Oct 2003
Posts: 3003
Location: Bay Area, CA
|
| Posted: Fri Jan 01, 2010 9:36 am Post subject: |
|
|
| ppurka wrote: | | --jobs is not supported since I have no idea how to display two simultaneous merges. |
I was hoping you won't say that... I have a i7920 with HT ON, so --jobs is a must for large updates. There must be something we can do... |
|
| Back to top |
|
|
ppurka
Advocate
Joined: 26 Dec 2004
Posts: 3256
|
| Posted: Fri Jan 01, 2010 5:15 pm Post subject: |
|
|
| devsk wrote: | | ppurka wrote: | | --jobs is not supported since I have no idea how to display two simultaneous merges. |
I was hoping you won't say that... I have a i7920 with HT ON, so --jobs is a must for large updates. There must be something we can do... |
Frankly speaking, I have no idea how to display simultaneous merges. I would have liked to do it too.
Think about it: I will have to detect every $DELAY seconds how many concurrent emerge processes are running:
1. genlop will not give a correct output since genlop doesn't detect emerges while it is in "installing" phase
2. the emerge output is also not that suitable.
Then I have to update that many lines in the terminal. tput might help in the latter, but overall it is too much of a hassle
_________________
emerge --quiet redefined | E17 vids: I, II | Now using kde5 | e is unstable :-/ |
|
| Back to top |
|
|
ppurka
Advocate
Joined: 26 Dec 2004
Posts: 3256
|
| Posted: Wed Jan 20, 2010 3:55 am Post subject: |
|
|
Update to version 20100119. Changelog:- Support for simultaneous multiple quietemerge instances.
This is not the support for --jobs. It is the support for running more than one quietemerge simultaneously. This was strictly speaking not possible earlier since we need to ensure that different log and temporary files are used, and tmpfs (un)mounting works (i.e. one running instance does not by mistake unmount tmpfs, while the other is running).
This also led me to uncover more bugs in genlop  So, be ready to see completely irrational genlop -c output if multiple quietemerge instances are running. However the emerges will still work just fine and I believe the script output will remain more or less sane. What may not be sane is the "ETA" 
_________________
emerge --quiet redefined | E17 vids: I, II | Now using kde5 | e is unstable :-/ |
|
| Back to top |
|
|
Mekoryuk
Apprentice
Joined: 17 Sep 2003
Posts: 174
|
| Posted: Tue Feb 02, 2010 8:54 pm Post subject: |
|
|
| I'm a little confused as to what constitutes proper behavior for the script. Since day one I've only ever seen "ETA" give an actual value on a handful of packages; most of the time ETA remains blank, and only ever mentions something when the build is almost finished ("any time now"). I've experimented with various values in the config file, but the result remains the same. I suppose it's because that genlop is attempting to calculate time remaining concurrently with the emerge, and that in most cases the emerge will finish long before genlop has a chance to figure it out, but even for large packages where one might expect genlop to figure it out, ETA still remains blank. Is this normal behavior? |
|
| Back to top |
|
|
ppurka
Advocate
Joined: 26 Dec 2004
Posts: 3256
|
| Posted: Tue Feb 02, 2010 10:02 pm Post subject: |
|
|
| Mekoryuk wrote: | | I'm a little confused as to what constitutes proper behavior for the script. Since day one I've only ever seen "ETA" give an actual value on a handful of packages; most of the time ETA remains blank, and only ever mentions something when the build is almost finished ("any time now"). I've experimented with various values in the config file, but the result remains the same. I suppose it's because that genlop is attempting to calculate time remaining concurrently with the emerge, and that in most cases the emerge will finish long before genlop has a chance to figure it out, but even for large packages where one might expect genlop to figure it out, ETA still remains blank. Is this normal behavior? |
No, this is not normal behaviour.
The cases where you will see a decent ETA is when the package has a good compile time. If the compile time is almost negligible then ETA will remain blank. Cases where you may get blank ETA:
1. package is small, or package compiles very quickly.
2. package is large, but package has no compile time. The only thing that takes time is the unpacking of the package (For example if you try to emerge openoffice-bin).
3. DELAY is set very large in the config script, so genlop output is not updated frequently enough.
4. /var/log/emerge.log is very large. So, genlop takes a significant amount of time to update, and by then the package has already compiled.
_________________
emerge --quiet redefined | E17 vids: I, II | Now using kde5 | e is unstable :-/ |
|
| Back to top |
|
|
gerard27
Advocate
Joined: 04 Jan 2004
Posts: 2377
Location: Netherlands
|
| Posted: Sun Apr 04, 2010 10:42 pm Post subject: |
|
|
Thanks a lot ppurka.
Gerard.
_________________
To install Gentoo I use sysrescuecd.Based on Gentoo,has firefox to browse Gentoo docs and mc to browse (and edit) files.
The same disk can be used for 32 and 64 bit installs.
You can follow the Handbook verbatim.
http://www.sysresccd.org/Download |
|
| Back to top |
|
|
ppurka
Advocate
Joined: 26 Dec 2004
Posts: 3256
|
| Posted: Mon Apr 05, 2010 3:06 am Post subject: |
|
|
| gerard82 wrote: | Thanks a lot ppurka.
Gerard. |
I am glad it is of help
_________________
emerge --quiet redefined | E17 vids: I, II | Now using kde5 | e is unstable :-/ |
|
| Back to top |
|
|
JustJoe
Tux's lil' helper
Joined: 30 Jul 2005
Posts: 80
|
| Posted: Tue Apr 06, 2010 6:35 pm Post subject: |
|
|
ppurka,
Great script, it looks awesome.
Maybe you can implement cpu-scaling:
before merge
# the next line would change the cpu-governour, if available, to the highest frequency
[ -f /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor ] && echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
[ -f /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor ] && echo performance > /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor
after merge
# and set the scheduler back to "ondemand"
[ -f /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor ] && echo ondemand > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
[ -f /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor ] && echo ondemand > /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor
You probably need a routine to check how many cores there are on the system.
These lines are taken from the tmerge-script @ http://en.gentoo-wiki.com/wiki/Speeding_up_portage_with_tmpfs
It sure does make a difference in merge time: https://forums.gentoo.org/viewtopic-p-6233551.html#6233551 |
|
| Back to top |
|
|
ppurka
Advocate
Joined: 26 Dec 2004
Posts: 3256
|
| Posted: Tue Apr 06, 2010 7:08 pm Post subject: |
|
|
| JustJoe wrote: | ppurka,
Great script, it looks awesome.
Maybe you can implement cpu-scaling:
|
Hi, your proposal seems interesting. However, I don't think I will implement cpu-scaling. People have many different setups. Some people control the scaling using cpufreq, some people use their DE's and so on. It is very easy to go wrong when you mess with others' cpu-scaling.
These type of things are best left to the daemon/DE/manual control as the individual wants.
_________________
emerge --quiet redefined | E17 vids: I, II | Now using kde5 | e is unstable :-/ |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Documentation, Tips & Tricks
