| View previous topic :: View next topic |
| Author |
Message |
inbreed
n00b
Joined: 29 Nov 2003
Posts: 22
Location: Germany
|
 Posted: Sun Nov 07, 2004 6:58 pm Post subject: lightspeed emerging without any tools.. |
 |
|
Hi,
i hope nobody else did this post yet, and that my search-lazyness will soon be gone.. 
On my way wrtiting some ramdisk-scripts i noticed, that i was able to emerge xorg-x11 within 20 minutes in 1200 mhz only! without any tools or special-USE-Flags.
later on i found ot, that all emerges are flying fast!
how?
all you need is tmpfs support to mount a tmpfs into your /var/tmp/portage.
why?
the way ram -> cpu ->ram -> hdd seems to be faster than
hdd -> ram -> cpu -> ram -> hdd -> hdd..
so what, damnt it!
if you got a ot of ram (e.G. 1 GB), try: | Code: | | mount -t tmpfs tmpfs /var/tmp/portage |
to create a RAM/2 ramdisk ( the ram will only be used if necessary.
if you want to emerge xorg, you'D need about 850mb of free ram, so mount like this:
| Code: | | mount -t tmpfs tmpfs -o size=850000000 /var/tmp/portage |
after emerging you can do a manually autoclean :
| Code: | | umount /var/tmp/portage |
please tell me wether it works fine for you or not! |
|
| Back to top |
|
 |
Vanquirius
Retired Dev
Joined: 14 Jun 2002
Posts: 1297
Location: Ethereal plains
|
| Posted: Sun Nov 07, 2004 10:25 pm Post subject: |
|
|
I don't have 1Gb of RAM to test large compiles, but I tested emerging smaller programs with 100Mb of RAM for /var/tmp/portage and it worked OK. I haven't timed it, but I don't think it had such a performance increase.
_________________
Hello. |
|
| Back to top |
|
|
Forse
Apprentice
Joined: 26 Dec 2002
Posts: 260
Location: /dev/random
|
| Posted: Mon Nov 08, 2004 12:05 am Post subject: |
|
|
I don't think it's such a good idea...tmpfs is shared memory and there for will be also placed in swap...so you're back on hdd -_-
_________________
[ My sites ]: UnixTutorials : AniFIND : AnimeYume |
|
| Back to top |
|
|
mudrii
l33t
Joined: 26 Jun 2003
Posts: 789
Location: Singapore
|
| Posted: Mon Nov 08, 2004 5:49 am Post subject: |
|
|
If I have 2G RAM I think it may work 
_________________
www.gentoo.ro |
|
| Back to top |
|
|
cheester
n00b
Joined: 08 Nov 2004
Posts: 8
Location: Poland, Poznañ
|
| Posted: Mon Nov 08, 2004 6:10 am Post subject: |
|
|
HI, that's my first post here
I also tried that tip and i must say it wasn't faster. I emerged few apps and emerge time was almost the same so there is no or maybe very little performance gain.
| Forse wrote: | | tmpfs is shared memory and there for will be also placed in swap... |
You can always turn off swap - that's what I did... but I have only 512MB RAM @ 400Mhz 
_________________
http://cheester.skysquad.net | Soldiers don't die, they just go to hell to regroup. |
|
| Back to top |
|
|
inbreed
n00b
Joined: 29 Nov 2003
Posts: 22
Location: Germany
|
| Posted: Mon Nov 08, 2004 7:45 am Post subject: |
|
|
hm... okies,
thx for testing. on my machine it was incredible faster - i dont use swap anyway, so i was able to do it all in ram..
maybe it also depends in your make-config? hmhm.. |
|
| Back to top |
|
|
nyda
Apprentice
Joined: 12 Aug 2004
Posts: 183
|
| Posted: Mon Nov 08, 2004 1:58 pm Post subject: |
|
|
Or your compiler cache kicked in.
| Code: |
root, root/ # genlop -t xorg-x11
* x11-base/xorg-x11
Tue Sep 7 16:19:39 2004 --> x11-base/xorg-x11-6.7.99.904
merge time: 49 minutes and 39 seconds.
Tue Sep 14 21:29:06 2004 --> x11-base/xorg-x11-6.8.0
merge time: 38 minutes and 51 seconds.
Fri Sep 17 15:27:04 2004 --> x11-base/xorg-x11-6.8.0-r1
merge time: 24 minutes and 20 seconds.
|
|
|
| Back to top |
|
|
s4kk3
Apprentice
Joined: 15 Oct 2004
Posts: 232
Location: Finland
|
| Posted: Mon Nov 08, 2004 4:32 pm Post subject: |
|
|
I tested this and not sure if it gives any performance. Havn't done any benchmarks yet.
What is the difference between ramfs and tmpfs? Does ccache affect this? I mean portage uses ccache that is located at hdd, so it would lower performance. This idea sounds great and I will do some bechmarks when I have time.
_________________
My own filemanager project |
|
| Back to top |
|
|
Vanquirius
Retired Dev
Joined: 14 Jun 2002
Posts: 1297
Location: Ethereal plains
|
| Posted: Mon Nov 08, 2004 4:47 pm Post subject: |
|
|
Had forgotten about genlop, I was thinking on using `time` to benchmark it... This makes it easier, having a look at two packages I emerged yesterday using this technique.
| Code: |
* media-video/transcode
Sun Aug 15 19:19:55 2004 --> media-video/transcode-0.6.11
merge time: 8 minutes.
Fri Sep 17 11:22:56 2004 --> media-video/transcode-0.6.11
merge time: 10 minutes and 9 seconds.
Sun Nov 7 21:01:21 2004 --> media-video/transcode-0.6.11
merge time: 10 minutes and 31 seconds.
* media-video/xine-ui
Sun Aug 15 20:48:28 2004 --> media-video/xine-ui-0.9.23-r2
merge time: 2 minutes and 53 seconds.
Sat Sep 25 20:09:08 2004 --> media-video/xine-ui-0.9.23-r2
merge time: 3 minutes and 2 seconds.
Sun Nov 7 21:04:15 2004 --> media-video/xine-ui-0.9.23-r2
merge time: 2 minutes and 54 seconds. |
_________________
Hello. |
|
| Back to top |
|
|
s4kk3
Apprentice
Joined: 15 Oct 2004
Posts: 232
Location: Finland
|
| Posted: Mon Nov 08, 2004 4:53 pm Post subject: |
|
|
unmerged ccache and it seems faster. Maybe some bechmark results today
_________________
My own filemanager project |
|
| Back to top |
|
|
inbreed
n00b
Joined: 29 Nov 2003
Posts: 22
Location: Germany
|
| Posted: Mon Nov 08, 2004 5:26 pm Post subject: |
|
|
good idea s4kk3 - ccache must not be used.
well, sorry for all thous killing thier time on reemerging.
| Code: | root # genlop -t xorg-x11
* x11-base/xorg-x11
Sat Nov 6 16:31:53 2004 --> x11-base/xorg-x11-6.8.0-r1
merge time: 19 minutes and 47 seconds.
merged totally 1 ebuild in 19 minutes and 47 seconds.
|
on my home machine..
| Code: |
cat /proc/cpuinfo | grep MHz
cpu MHz : 1527.110
|
|
|
| Back to top |
|
|
s4kk3
Apprentice
Joined: 15 Oct 2004
Posts: 232
Location: Finland
|
| Posted: Mon Nov 08, 2004 5:37 pm Post subject: |
|
|
Made some bechmarks with mplayer
First without tmpfs or ramfs
| Code: | Mon Nov 8 19:12:10 2004 --> media-video/mplayer-1.0_pre4-r7
merge time: 8 minutes and 59 seconds.
|
Then with ramfs
| Code: | Mon Nov 8 19:26:51 2004 --> media-video/mplayer-1.0_pre4-r7
merge time: 8 minutes and 49 seconds.
|
and finally tmpfs
| Code: | Mon Nov 8 19:41:39 2004 --> media-video/mplayer-1.0_pre4-r7
merge time: 8 minutes and 49 seconds.
|
It didn't give so much performance, but i think ramfs or tmpfs is faster with bigger packages. Have to try with xorg this night.
EDIT: I just got an crazy idea. What would happen if ccache is included in ram too? But only thinking if it's possible
_________________
My own filemanager project |
|
| Back to top |
|
|
Vanquirius
Retired Dev
Joined: 14 Jun 2002
Posts: 1297
Location: Ethereal plains
|
| Posted: Mon Nov 08, 2004 6:40 pm Post subject: |
|
|
| s4kk3 wrote: | | EDIT: I just got an crazy idea. What would happen if ccache is included in ram too? But only thinking if it's possible |
That would be nice, but your ccache size would be heavily limited by the amount of RAM you can give it. In short: your ccache may not be big enough to be useful depending on the range of packages you usually compile. Also, that would mean copying your ccache to and from your hard-disk everytime you reboot for any reason.
_________________
Hello. |
|
| Back to top |
|
|
Viha
Tux's lil' helper
Joined: 04 Oct 2003
Posts: 121
Location: Jon's World
|
| Posted: Mon Nov 08, 2004 6:44 pm Post subject: |
|
|
| Yeah, only if the system is rarely rebooted or shutdown. But maybe it would be possible to use something like rsync to keep the copy on hard drive updated, I don't know. |
|
| Back to top |
|
|
s4kk3
Apprentice
Joined: 15 Oct 2004
Posts: 232
Location: Finland
|
| Posted: Mon Nov 08, 2004 6:53 pm Post subject: |
|
|
| Quote: | | That would be nice, but your ccache size would be heavily limited by the amount of RAM you can give it. In short: your ccache may not be big enough to be useful depending on the range of packages you usually compile. Also, that would mean copying your ccache to and from your hard-disk everytime you reboot for any reason. |
I know that but you can limit ccache to use lesser memory as I have seen it wont use so much space. I had limited ccache to use 100mb space so why that couldn't be included in ram if you have enought ram? And it rarely used that 100mb
_________________
My own filemanager project |
|
| Back to top |
|
|
Vanquirius
Retired Dev
Joined: 14 Jun 2002
Posts: 1297
Location: Ethereal plains
|
| Posted: Mon Nov 08, 2004 7:16 pm Post subject: |
|
|
My experience has been different:
| Code: | # ccache -s
...
cache size 1.8 Gbytes
max cache size 2.0 Gbytes |
_________________
Hello. |
|
| Back to top |
|
|
s4kk3
Apprentice
Joined: 15 Oct 2004
Posts: 232
Location: Finland
|
| Posted: Mon Nov 08, 2004 8:40 pm Post subject: |
|
|
With kdemultimedia I saved about eight minutes with tmpfs, haven't tested with ramfs (with ramfs it looks like it doesn't use ram).
_________________
My own filemanager project |
|
| Back to top |
|
|
s4kk3
Apprentice
Joined: 15 Oct 2004
Posts: 232
Location: Finland
|
| Posted: Tue Nov 09, 2004 4:12 pm Post subject: |
|
|
Damn. How you can compile xorg so fast? With tmpfs it took 52mins to compile it and without 1hour 11mins. Prosessor athlon xp 2100+, 1gb 400 MHz ddr ram. My hdd speed shouldn't affect this cause compilling will be done in ram? Does LDFLAGS increase compilling time?
Here is my make.conf:
| Code: | # These settings were set by the catalyst build script that automatically built this stage
# Please consult /etc/make.conf.example for a more detailed example
CFLAGS="-march=athlon-xp -mtune=athlon-xp -O2 -m3dnow -msse -mmmx -mfpmath=sse387 -ffast-math -fprefetch-loop-arrays -funroll-loops -fforce-addr -pipe ftracer -fomit-frame-pointer"
CHOST="i686-pc-linux-gnu"
CXXFLAGS="-march=athlon-xp -mtune=athlon-xp -O2 -m3dnow -msse -mmmx -mfpmath=sse,387 -ffast-math -fprefetch-loop-arrays -funroll-loops -fforce-addr -pipe -ftracer -fomit-frame-pointer -fvisibility-inlines-hidden -fvisibility=hidden"
MAKEOPTS="-j2"
USE="3dnow 3dnowex 3dnowext amd fPIC fpic mmx mmxex mmxext nptl nvidia ooo-kde openal opengl posix sse video_cards_nvidia videos -arts -man -doc -video_cards_ati -video_cards_radeon -ati -radeon -intel -amd64 -pentium -pentium2 -pentium3 -pentium4 -ia64 -ppc64 -ppc"
PORTDIR_OVERLAY=/usr/local/portage
FETCHCOMMAND="/usr/bin/axel -a -S6 \${URI} -o \${DISTDIR}"
RESUMECOMMAND="/usr/bin/axel -a -S6 \${URI} -o \${DISTDIR}"
LDFLAGS="-Wl,-O1"
LINGUAS="fi"
|
_________________
My own filemanager project |
|
| Back to top |
|
|
nyda
Apprentice
Joined: 12 Aug 2004
Posts: 183
|
| Posted: Tue Nov 09, 2004 9:11 pm Post subject: |
|
|
| s4kk3 wrote: |
| Code: | | CFLAGS="-march=athlon-xp -mtune=athlon-xp -O2 -m3dnow -msse -mmmx -mfpmath=sse387 -ffast-math -fprefetch-loop-arrays -funroll-loops -fforce-addr -pipe ftracer -fomit-frame-pointer" |
|
Ever considered that it might take more time to produce optimized code than normal code? Your depot of CFLAGS might well be a reason why it's slower. You're also missing a - in front of ftracer =)
| s4kk3 wrote: | | I know that but you can limit ccache to use lesser memory as I have seen it wont use so much space. I had limited ccache to use 100mb space so why that couldn't be included in ram if you have enought ram? And it rarely used that 100mb |
Building openoffice will go through those 100MB at least 8 times. So at the point where your OOo is at 12%, it has already wiped the cache for everything else - rendering your CC somewhat... useless 
OOo is extreme, but so is a 100MB CC.
edit: actually forgot to mention the obligatory "-ffast-math will break your system" comment  |
|
| Back to top |
|
|
inbreed
n00b
Joined: 29 Nov 2003
Posts: 22
Location: Germany
|
| Posted: Wed Nov 10, 2004 9:42 am Post subject: |
|
|
uff,
my cflags are something like -O3 -pipe -fomit-frame-pointers -mtune=athlon-xp -march=athlon-xp
thats all.
LDFLAGS="-Wl,-O1 -Wl,--enable-new-dtags -Wl,--sort-common -s" <- maybe they are also dealing with it.. |
|
| Back to top |
|
|
disturbed
Apprentice
Joined: 18 May 2003
Posts: 216
Location: Belgium , Antwerp
|
| Posted: Wed Nov 10, 2004 11:25 am Post subject: |
|
|
whow , my speed increases a lot... Thx 4 the tip...
grtz
_________________
A computer is like airconditioning. When you open windows, it doesn't work. |
|
| Back to top |
|
|
s4kk3
Apprentice
Joined: 15 Oct 2004
Posts: 232
Location: Finland
|
| Posted: Wed Nov 10, 2004 11:28 am Post subject: |
|
|
I have tried with lesser flags. But it didn't make any difference. Maybe it's hardware related. When I installed my first linux (redhat) it decreased my MHz from 1600 to 1300. Maybe it decreased something else at the same time?
_________________
My own filemanager project |
|
| Back to top |
|
|
akoning
n00b
Joined: 30 Jun 2004
Posts: 3
|
| Posted: Fri Nov 12, 2004 8:38 am Post subject: |
|
|
Hi,
I have been doing the same thing for a while now.
I found some post in the forum about running /var/log on a tmpfs for power saving on my laptop and thought why not do compilation on a tmpfs dir.
I figured even if the build is bigger than your ram size the overhead due to swapping would not be significant enough to cause the emerge to be slower than when you work on a 'normal' partition.
On my desktop machine (athlon-xp 2800 + 1G) it run's smooth as a baby.
(I did not do any benchmarks but it is definably faster).
But because of the size of the ram no swap-space is ever used.
On my laptop with 256M it causes freeses. I think that when there is too much swapping on this machine the kernel locks.
This machine is tuned to reduce swapping and I have the laptop-mode switch in the proc switched on. |
|
| Back to top |
|
|
Vanquirius
Retired Dev
Joined: 14 Jun 2002
Posts: 1297
Location: Ethereal plains
|
| Posted: Fri Nov 12, 2004 1:27 pm Post subject: |
|
|
| s4kk3 wrote: | | When I installed my first linux (redhat) it decreased my MHz from 1600 to 1300. |
Any chance you are getting them back? 
_________________
Hello. |
|
| Back to top |
|
|
s4kk3
Apprentice
Joined: 15 Oct 2004
Posts: 232
Location: Finland
|
| Posted: Fri Nov 12, 2004 2:25 pm Post subject: |
|
|
Switched that back to 1600MHz. But everything else was normal in bios. Maybe my kernel sucks. Is there any guide how to build fast kernel? I mean what should I enable and disable. Haven't found any yet.
Ccache gives little more boost when it is enabled in tmpfs but you can't keep that so big. Well it is better than nothing. Limited ccache to 70mb in ram. Maybe I should make a script that makes tar from ccache dir when shutting down and untars it when booting. Then ccache doens't need to collect all the stuff when it is already there.
_________________
My own filemanager project |
|
| Back to top |
|
|
|
Documentation, Tips & Tricks
