| View previous topic :: View next topic |
| Author |
Message |
josephg
l33t
Joined: 10 Jan 2016
Posts: 783
Location: usually offline
|
 Posted: Sun Aug 06, 2017 8:31 am Post subject: How to extend Portage TMPDIR in ramdisk through to disk(s) |
 |
|
my particular pain point this how-to targets are the extremely (long and) resource intensive emerges for larger packages which can't fully compile in tmpfs. i have to create notmpfs workarounds, which means no benefit at all from tmpfs for these bloatwares. so my choices were either tmpfs or notmpfs. and these large bloatwares are the particular pain points which could most benefit from tmpfs. hence, this piece.
instead of tmpfs, i use zram to create a btrfs filesystem in memory, which i extend to disk(s) only when required with as much space as needed by emerge. now emerge can use ramdisk and spill over to disk, rather than an either or only. not touching physical disks gives an immense performance boost. so i try not to use physical swap devices. for step by step instructions, read on.. http://nixventure.blogspot.com/2017/08/gentoo-portage-emerge-tmpdir-extending.html
this might be useful for those who want to emerge in tmpfs but don't have enough space there. this could also be useful for low-memory systems which can additionally benefit from swap on compcache. you might find this useful to extend other use-cases too.
comments and brickbats welcome..
_________________
"Growth for the sake of growth is the ideology of the cancer cell." Edward Abbey
Last edited by josephg on Wed Aug 09, 2017 8:38 am; edited 1 time in total |
|
| Back to top |
|
 |
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3747
Location: Rasi, Finland
|
| Posted: Sun Aug 06, 2017 8:49 am Post subject: |
|
|
I just have swap. Mainly the swap is for hibernation, but if I happen to run out of ram during emerging, then the kernel moves the least used stuff in tmpfs to swap. It has been working before on my earlier setups where I had 8GiB or less RAM. Although it might need some fine tuning of some vm.* settings to get the best out of it.
_________________
..: Zucca :..
| My gentoo installs: | init=/sbin/openrc-init
-systemd -logind -elogind seatd |
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
josephg
l33t
Joined: 10 Jan 2016
Posts: 783
Location: usually offline
|
| Posted: Sun Aug 06, 2017 11:51 am Post subject: |
|
|
| Zucca wrote: | | I just have swap. Mainly the swap is for hibernation, but if I happen to run out of ram during emerging, then the kernel moves the least used stuff in tmpfs to swap. It has been working before on my earlier setups where I had 8GiB or less RAM. Although it might need some fine tuning of some vm.* settings to get the best out of it. |
i have 4G memsize on this one, and emerge continued to crash even with much bigger swap sizes. i particularly remember boost etc was one of my nightmares.. it would take so long, and then eventually crash. meanwhile my keyboard/mouse/etc would just die. sometimes i've had to pull the power. |
|
| Back to top |
|
|
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3463
|
| Posted: Sun Aug 06, 2017 11:54 am Post subject: |
|
|
| Quote: | | Although it might need some fine tuning of some vm.* settings to get the best out of it. |
Some people claim that kernel will never use swap at 0, which means 1 would be the preferred value, down from 60 out of 100. Haven't tested it myself, so you might want to verify it before using either option.
I found 8 GB enough for building stuff in RAM during typical desktop workload. I just had to bump the size of tmpfs from default 50% to 85% (fstab option: size=85%), which is also a neat trick if you're close to the borderline.
| Quote: | | meanwhile my keyboard/mouse/etc would just die. sometimes i've had to pull the power. |
This might be related to general load rather than size of SWAP.
Bumping portage niceness from 0 to 19 helps often a lot with such issues |
|
| Back to top |
|
|
mir3x
Guru
Joined: 02 Jun 2012
Posts: 455
|
| Posted: Mon Aug 07, 2017 9:56 pm Post subject: |
|
|
| Quote: | | meanwhile my keyboard/mouse/etc would just die. sometimes i've had to pull the power. |
try oom killer - magic sys rq + f
Probably u will need to use it few times, it oftens kills X, but its better that hard reset.
_________________
Sent from Windows |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3747
Location: Rasi, Finland
|
| Posted: Mon Aug 07, 2017 10:53 pm Post subject: Few ideas... |
|
|
One thing came into my mind... You could work out something by using bcache feature of Linux. That way you cannot use tmpfs... at least directly. Afaik bcache needs a actual device. zram can create virtual (compressed) storage devices. You could use a zram device as the fast cache drive (also saves memory by compressing) then you'd need the another device (or maybe just a partition) where to put "overflowing" stuff.
The problem with zram is that it consumes more memory as it fills, but does not free it unless you unmount the device and reset it's size (and then create a filesystem back on it).
So you'd propably need to automate the process.
Second idea I have is to use swap as the "overflow" storage as I mentioned before. But now add zswap in to the game. zswap compresses memory pages in ram if memory usage is too high (user selectable limit). If later the memory fills up (even when the memory is compressed), then the least used memory pages are moved to actual swap.
This should be much easier to set up than the zram+bcache hack.
I've not tested either method, but the first one really needs quite a lot of tinkering to get it working propely. For example resetting the bcache-zram device after emerge exits...
_________________
..: Zucca :..
| My gentoo installs: | init=/sbin/openrc-init
-systemd -logind -elogind seatd |
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
jonathan183
Guru
Joined: 13 Dec 2011
Posts: 318
|
| Posted: Tue Aug 08, 2017 12:08 am Post subject: |
|
|
If you had 16G of RAM would you still use zram, would you split between tmpfs and swap? would that split change for emerging packages vs use of the system for other things?
I ended up setting up portage on disc for my previous desktop to build things like Libreoffice ... time for me to revisit this since my system now has more RAM ... |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3747
Location: Rasi, Finland
|
| Posted: Tue Aug 08, 2017 10:15 am Post subject: |
|
|
| jonathan183 wrote: | | If you had 16G of RAM would you still use zram, would you split between tmpfs and swap? |
With zram you can create zram device nodes under /dev and they appear like a block (mass storage) device.
With tmpfs you don't get any device nodes, you just specify a mount point where the "ram portion" starts. tmpfs works more like a network share; you don't format it with a filesystem - tmpfs IS a filesystem of its own. So you cannot use it as a cache -part of bcache (afaik bcache needs a device node). Creating a filesystem image inside tmpfs gives you something close to what zram gives. But it automatically reserves the amount of memory the image takes disk space. So, don't do that. Use zram instead if you really want to go complex.
Now back to your question: I wouldn't use zram. I'd use zswap and tmpfs. Lastly I'd tune vm.swappiness and zswap settings. With zswap I'd propably set 50% of memory for zswap to compress (with that 16GB of RAM). Then I'd create a swap of around 4-8GB. YMMV... I have no need for any memory compression, so I haven't tested this.
| jonathan183 wrote: | | would that split change for emerging packages vs use of the system for other things? |
I'm not sure if I understood your question here... But you can use other applications at the same time, but the performance will be slower if zswap needs to compress (and uncompress) memory pages in addition to compiling overhead. I think kernel (zswap) keeps track which memory areas are in more use and tries to keep those in the uncompressed area while the temporary files that are generated while compiling software are most propably in compressed form in memory (by zswap, if needed). But if the system still runs out of memory after zswap has tried its best the swapping starts and will slow the machine down. vm.swappiness might play HUGE role if/when this starts. I'd guess if you set vm.swappiness to around 50 or 60 kernel has already moved some memory to swap area before you really start running out of memory, thus making swapping at this point less demending for the system.
In conclusion: Enable zswap and tune its settings. Create a (enough large) tmpfs for your portage temp files.
If the performance is poor, tune vm.* and zswap settings.
If you want to go wild, try out zram device as a cache part of bcache (I doubt it'll be any better than zswap+tmpfs).
_________________
..: Zucca :..
| My gentoo installs: | init=/sbin/openrc-init
-systemd -logind -elogind seatd |
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
|
Documentation, Tips & Tricks
