| View previous topic :: View next topic |
| Author |
Message |
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1810
Location: PB, Germany
|
 Posted: Thu May 07, 2015 9:54 am Post subject: |
 |
|
I have squashed /opt/lib/vmware-tools. But almost every reboot it must be resquashed. Checking what has changed like | Code: | | find /opt/lib/vmware-tools -mtime 7 |
shows nothing. What is the cause for the many re-squashing?
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
 |
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Thu May 07, 2015 11:35 am Post subject: |
|
|
| Check the CHANGES directory |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1810
Location: PB, Germany
|
| Posted: Fri May 08, 2015 1:32 pm Post subject: |
|
|
I applied squashmount with lzo to some 13GB git repository, reducing space to ~25% and speeding up rebase from >3min to ~1,3min...
Now the question, resquashing takes /tmp on root fs which has not enough space. How can I change the tmp to the location where the squashfs itself is stored? I can't find if that is a mksquashfs option or squashmount related.
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Fri May 08, 2015 4:22 pm Post subject: |
|
|
| Massimo B. wrote: | | How can I change the tmp to the location where the squashfs itself is stored? |
squashmount --man: TEMPDIR => '/path-to-a-tmpdir-on-partition-with-squashfile'
Note that the change takes effect only after you stop'ed squashmount. |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1810
Location: PB, Germany
|
| Posted: Mon May 11, 2015 7:39 am Post subject: |
|
|
| mv wrote: | | squashmount --man: TEMPDIR => '/path-to-a-tmpdir-on-partition-with-squashfile' |
Thanks, and sorry I did not know --man, only --help and the /usr/share/doc/squashmount-*/README. Why not installing a usual man-page via ebuild?
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Mon May 11, 2015 6:51 pm Post subject: |
|
|
| Massimo B. wrote: | | Why not installing a usual man-page via ebuild? |
The perl-ish way is to have the manpage built into the program, because perl has its own nroff variant "pod" (man perlpod).
| squashmount --help wrote: | Usage:
squashmount [options] *command* [*mask1* *mask2* ...]
Where *command* is one of mount, umount, remount, set, reset,
status, forget, print-*...*
To get an extended help, type squashmount man
Options: [...] |
Of course works, as well... |
|
| Back to top |
|
|
schorsch_76
Guru
Joined: 19 Jun 2012
Posts: 452
|
| Posted: Thu May 28, 2015 6:35 pm Post subject: |
|
|
Today i tried to setup my portage to an squashfsmounted filesystem.
| Code: |
#!/usr/bin/perl (this is only for editors)
# The tools which we have installed; if possible only the first in this list
# is used, but the others are a fallback if that fails.
@order = ('overlayfs');
# Even if we define following is empty it is convenient to use
# this local variable throughout, so that we can simply change it:
DIFF => 1;
RESQUASH_ON_START => 1;
push(@mounts, {
TAG => 'portage',
DIR => '/usr/portage',
FILE => '/usr/portage.sqfs',
CHANGES => '/usr/portage.changes',
READONLY => '/usr/portage.readonly',
THRESHOLD => '40m', # resquash on umount if 40 megabytes changed
COMPRESSION => 'lz4'
});
@umount = ('-i');
|
I started squashmount, it packaged /usr/portage and mounted it but i get the error
| Code: | * [portage]: mounting...
* [portage]: error: fallback to mount --bind
|
I got in my kernel
| Code: |
zcat /proc/config.gz | grep OVERLAY
CONFIG_OVERLAY_FS=m
zcat /proc/config.gz | grep SQUASH
CONFIG_SQUASHFS=y
CONFIG_SQUASHFS_FILE_CACHE=y
# CONFIG_SQUASHFS_FILE_DIRECT is not set
CONFIG_SQUASHFS_DECOMP_SINGLE=y
# CONFIG_SQUASHFS_DECOMP_MULTI is not set
# CONFIG_SQUASHFS_DECOMP_MULTI_PERCPU is not set
CONFIG_SQUASHFS_XATTR=y
CONFIG_SQUASHFS_ZLIB=y
CONFIG_SQUASHFS_LZ4=y
CONFIG_SQUASHFS_LZO=y
CONFIG_SQUASHFS_XZ=y
CONFIG_SQUASHFS_4K_DEVBLK_SIZE=y
CONFIG_SQUASHFS_EMBEDDED=y
CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=3
|
mount --bind works in other use cases too, so .. where micht be the problem?
mount tells me ...
| Code: | mount | column -t
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=10240k,nr_inodes=2048587,mode=755)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
/dev/mapper/desktop-gentoo on / type ext4 (rw,noatime,data=ordered)
tmpfs on /run type tmpfs (rw,nodev,relatime,size=1639252k,mode=755)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
cgroup_root on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
openrc on /sys/fs/cgroup/openrc type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/lib64/rc/sh/cgroup-release-agent.sh,name=openrc)
cpuset on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cpuacct on /sys/fs/cgroup/cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct)
/dev/sda1 on /boot type ext2 (rw,noatime)
/dev/mapper/desktop-home on /home type ext4 (rw)
none on /var/tmp/portage type tmpfs (rw,size=12G)
/usr/portage.sqfs on /usr/portage.readonly type squashfs (ro,noatime)
/usr/portage.readonly on /usr/portage type none (ro,bind)
|
I am on openrc 
EDIT: Problem existed between keyboard and chair ..... overlay ... overlayfs *DOH* |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1810
Location: PB, Germany
|
| Posted: Mon Jun 01, 2015 9:12 am Post subject: |
|
|
What is the purpose of the new /etc/portage/repo.postsync.d/50-squashmount-gentoo?
It seems to remount (re-compress) the files after sync which seems quite useful (I did a global squashmount remount by nightly cron-job so far).
But in that file there is said:
| Quote: | # This file remounts the squashmount mount-point "gentoo" after each syncing
# of the "gentoo" repository.
# Do not use this file if you want to use a mount-point named "gentoo" with
# squashmount without using sync-type = squashdelta |
So without squashdelta this should not be used?
Then the mount-point is called gentoo, while my "gentoo" repo is called portage and located as default in /usr/portage.
Then squashdelta was told to be broken and not available anymore, no?
BTW looking for a low bandwidth consuming sync-type I ended with | Code: | sync-type = git
sync-uri = git://github.com/gentoo/gentoo-portage-rsync-mirror |
..inofficial mirror but quite up-to-date with less than 1 hour delay.
While having some more local data with .git/ metadata I pay that for having the compact delta sync.
A bare repo with metadata only would not even need squashfs as git already has some zlib compression, but a checkout is still required as portage cannot work on a bare git repo only.
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Mon Jun 01, 2015 11:28 am Post subject: |
|
|
| Massimo B. wrote: | | What is the purpose of the new /etc/portage/repo.postsync.d/50-squashmount-gentoo? |
It is for playing nice with the squashdelta module of portage.
| Quote: | | It seems to remount (re-compress) the files after sync which seems quite useful |
I would not consider it that useful: You loose the advantage that you can no longer see the differences (in the CHANGES directory) and you do no longer have the previous tree (as READONLY) accessible for the few cases that you might want to to roll back.
| Quote: | | (I did a global squashmount remount by nightly cron-job so far). |
It is dangerous to do this by a cron-job: You will run into troubles if a long emerge is still running which might access the tree at the same time. I usually do it after I finished emerge -NaDu @world or do not call it at all - since it will automatically be called when I switch off the machine. Of course, the latter is a bad idea if you usually never switch off the machine...
| Quote: | | So without squashdelta this should not be used? |
With squashdelta, it is necessary, because the tree must be remounted when the .sqfs file was modified. Without squashdelta - if you consider it reasonable, you can resquash, of course, but the advantage is questionable.
| Quote: | | Then the mount-point is called gentoo, while my "gentoo" repo is called portage and located as default in /usr/portage. |
When you switch to squashdelta, you will likely want to have to mount points: One for layman overlays (or perhaps also for local overlay) and one for the gentoo repository. You cannot mix them, since the distributed .sqfs-file only contains the gentoo repository. Since it is not simple to check in the script whether squashdelta was used, the name is used as a heuristic guess whether the user switched to squashdelta...
Of course, you can simply modify the script (after all, it is in a CONFIG_PROTECT directory) to match the name of your repository and/or use other squashmount options. The script is more meant as an example (which usually does not hurt if it is installed by default, unless you use the mount-point "gentoo" for something completely different...)
| Quote: | | Then squashdelta was told to be broken and not available anymore, no? |
squashdelta was the unfortunate victim of an argument between the developers: If I understand correctly, it was removed as a "punishment" for a bad tone. Unfortunately, this "punishment" actually hurts the portage users more than the developers. I hope that the developers will make their mind up after a while and put the module back. |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1810
Location: PB, Germany
|
| Posted: Mon Jun 01, 2015 11:49 am Post subject: |
|
|
| mv wrote: | | It is dangerous to do this by a cron-job: You will run into troubles if a long emerge is still running which might access the tree at the same time. I usually do it after I finished emerge -NaDu @world or do not call it at all - since it will automatically be called when I switch off the machine. Of course, the latter is a bad idea if you usually never switch off the machine... |
It's bad for the machine I never halt, and it is bad for the machines I often halt because usually I stop the machine soon before leaving and a very long shutdown compressing around is annoying then.
Of course I check for running emerge processes in that cron-job: | Code: | this_basename="${0##*/}"
this_dirname="${0%/*}"
if pgrep -f "emerge|eix" >/dev/null; then
logger -t $this_basename "squashmount flush skipped due to running emerge."
else
squashmount remount
logger -t $this_basename "squashmount flush finished."
fi
|
| Quote: | | Quote: | | Then squashdelta was told to be broken and not available anymore, no? |
squashdelta was the unfortunate victim of an argument between the developers: If I understand correctly, it was removed as a "punishment" for a bad tone. Unfortunately, this "punishment" actually hurts the portage users more than the developers. I hope that the developers will make their mind up after a while and put the module back. |
Interesting. I will look into squashdelta and how it works if it is back again. I'll see if it has benefits over git as sync-type. If it won't come back as official sync-type... then I would rather forget about is. My git sync-type and squashmount between is already so very custom, that any issues I have with portage won't be supported by anyone on #gentoo.de 
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Mon Jun 01, 2015 12:10 pm Post subject: |
|
|
| Massimo B. wrote: | | and it is bad for the machines I often halt because usually I stop the machine soon before leaving and a very long shutdown compressing around is annoying then. |
To shut down quickly, you can use
(provided that CHANGES is not cleaned during shutdown, of course).
Or setting a reasonable THRESHOLD value can help, too...
Or if the time during booting is not important, there is also RESQUASH_ON_START available. |
|
| Back to top |
|
|
Petros
n00b
Joined: 28 Mar 2015
Posts: 50
|
| Posted: Fri Jun 05, 2015 7:07 pm Post subject: |
|
|
I was wondering if putting my /lib(64) on to the RAM would speed up my system, and if is doable via squashmount.
What could go wrong and what I have to be careful of? |
|
| Back to top |
|
|
schorsch_76
Guru
Joined: 19 Jun 2012
Posts: 452
|
| Posted: Fri Jun 05, 2015 7:16 pm Post subject: |
|
|
I changed for one "squashpoint" the co,pression level to see how it behaves different than lz4. Now i try to remount it, but tit ells me, there is no need. I tried
| Code: |
squashmount remount -s src
|
| Code: |
squashmount umount src
squashmount forget src
squashmount mount src
|
just creating a file bigger than the threshold let it resquash it. Are there other options available? I thought from the manpage a "-s" on remount would do this .... 
EDIT: Maybe it would be nice to check on unmount and/or resquash if some process is sitting in this mountpoint. When a process is using a file in it, resquash/umount will fail. (Hint: lsof)
EDIT: A very very nice and neat tool you wrote there!  |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Fri Jun 05, 2015 8:04 pm Post subject: |
|
|
| schorsch_76 wrote: | | I thought from the manpage a "-s" on remount would do this ... |
squashmount -s remount ... will resquash if there is any change - independent of the THRESHOLD.
However, if there is absolutely no change there is also no need for resquashing. The only exceptional use case I am aware of is the one you mention: When you want to compare various compression methods. However, this use-case is so exceptional that I don't think that it deserves a special option. It is better to call mksquashfs manually when you want to compare these methods.
| Quote: | | EDIT: Maybe it would be nice to check on unmount and/or resquash if some process is sitting in this mountpoint. When a process is using a file in it, resquash/umount will fail. (Hint: lsof) ;) |
lsof is not really reliable. AFAIK there is no reliable method to check this.
Moreover, since the default is to fall back to lazy umounting after an umount failure, resquash will usually succeed anyway (and just print a warning). |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Fri Jun 05, 2015 8:12 pm Post subject: |
|
|
| Petros wrote: | | I was wondering if putting my /lib(64) on to the RAM would speed up my system, and if is doable via squashmount. |
I never tried, but I guess using zram or zcache will give you more improvement. For libs like libc, this is pointless anyway, since they are surely always in use by some running programm and thus mmapped anyway - making another mmap copy costs practically zero for the kernel.
| Quote: | | What could go wrong and what I have to be careful of? |
When you would compress glibc without keeping a copy, you probably cannot start squashmount, because commands like "mount" will need glibc to operate.
Thus, very likely you will have the problem that you cannot pull yourself up by your own bootstraps...
If you have a rescuecd, this is not unrepairable 'though: You just have to "unsquash" the directory again from the rescue cd. |
|
| Back to top |
|
|
Petros
n00b
Joined: 28 Mar 2015
Posts: 50
|
| Posted: Fri Jun 05, 2015 8:25 pm Post subject: |
|
|
Thank you very much. Indeed, squashing the directory that glibc resides in, is going to render this lib inaccessible.
Maybe a ramdisk or something like that could be a more elegant solution. |
|
| Back to top |
|
|
schorsch_76
Guru
Joined: 19 Jun 2012
Posts: 452
|
| Posted: Sun Jun 07, 2015 10:15 am Post subject: |
|
|
Did you ever place the rootfs on such a "squashpoint" like some embedded systems like routers do it? [1] The point is often the update from that rootfs. You need an other pc for it and mount/create the rootfs.sqfs file there...
I think about it, for my amdgeodelx system....
[1] http://www.tldp.org/HOWTO/html_single/SquashFS-HOWTO/ |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Sun Jun 07, 2015 11:33 am Post subject: |
|
|
| schorsch_76 wrote: | | Did you ever place the rootfs on such a "squashpoint" like some embedded systems like routers do it? |
I had tried years ago with aufs (and squash_dir at that time), and there was no problem. You will have to mount --bind some other directories inside (like /sys /proc etc), and you have to umount them before you can umount the squash filesystem: Since squashmount-10.1.0 this should be easy with the $after_mont and $before_umount hooks. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Sun Jun 07, 2015 11:42 am Post subject: |
|
|
| schorsch_76 wrote: | | Maybe it would be nice to check on unmount and/or resquash if some process is sitting in this mountpoint. When a process is using a file in it, resquash/umount will fail. (Hint: lsof) |
After sleeping it over: An additional check (even if it is not reliable) cannot hurt.
squashmount-12.0.0 provides now a corresponding --lsof option (and $lsof variable to set the default), and the check defaults to "on", except in the shutdown init scripts where it is explicitly overridden.
| Quote: | | I thought from the manpage a "-s" on remount would do this .... |
Since this is almost a FAQ, meanwhile, squashmount-12.0.0 has a new state THRESHOLD=-2 (and corresponding option --squash-force) which forces a resquash, independent of whether any changes had occured. |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1810
Location: PB, Germany
|
| Posted: Fri Jul 24, 2015 9:16 am Post subject: |
|
|
Interesting comment about why libreoffice can fail with squashmount: 536558#c7
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Fri Jul 24, 2015 11:51 am Post subject: |
|
|
| Massimo B. wrote: | | Interesting comment about why libreoffice can fail with squashmount: 536558#c7 |
Interesting. However, I cannot reproduce it. For me libreoffice just always worked with squashmount. |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1810
Location: PB, Germany
|
| Posted: Fri Aug 14, 2015 6:39 am Post subject: |
|
|
Occasionally I can't sync the mv overlay with eix-sync anymore: | Code: | >>> Syncing repository 'mv' into '/var/lib/layman/mv'...
>>> Starting layman sync for mv...
* Running Git... # ( cd /var/lib/layman/mv && /usr/bin/git pull --depth=1 )
U games-rpg/m5figur-mv/ChangeLog
U games-rpg/m5figur-mv/Manifest
A games-rpg/m5figur-mv/m5figur-mv-2.4.ebuild
Pull is not possible because you have unmerged files.
Please, fix them up in the work tree, and then use 'git add/rm <file>'
as appropriate to mark resolution and make a commit.
* Failure result returned from Git
*
* Errors:
* ------
* Failed to sync overlay "mv".
* Error was: Syncing overlay "mv" returned status 1!
* db.sync() |
I'm not the git expert yet, but after some trials I got that fixed with | Code: | | git reset --hard origin/master |
, but I need to do that occasionally, it is not an issue with other overlays.
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Sat Aug 15, 2015 9:49 am Post subject: |
|
|
| I consider this a layman bug: IMHO, layman should not only pull but actually "hard" update. (At the very least, this should be the default, perhaps with a possibility to opt out, although I do not see any reason for such an option.) |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Sun Aug 16, 2015 4:10 pm Post subject: |
|
|
| Just to make it clear: The layman update problem appears whenever the git repository is rebased (e.g. if something is permanently removed from the history). This does not happen very often, but occassionally it is reasonable, e.g. if an unrelated file was pushed by mistake. |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1810
Location: PB, Germany
|
| Posted: Mon Aug 31, 2015 8:23 am Post subject: |
|
|
There it is again:
| Code: | From git://anongit.gentoo.org/user/mv
+ 3327936...a7f519f master -> origin/master (forced update)
Auto-merging www-plugins/noscript/Manifest
CONFLICT (add/add): Merge conflict in www-plugins/noscript/Manifest
Auto-merging www-plugins/noscript/ChangeLog
CONFLICT (add/add): Merge conflict in www-plugins/noscript/ChangeLog
Auto-merging sys-apps/schedule/Manifest
CONFLICT (add/add): Merge conflict in sys-apps/schedule/Manifest
Auto-merging sys-apps/schedule/ChangeLog
CONFLICT (add/add): Merge conflict in sys-apps/schedule/ChangeLog
Auto-merging sys-apps/cpi/Manifest
CONFLICT (add/add): Merge conflict in sys-apps/cpi/Manifest
Auto-merging sys-apps/cpi/ChangeLog
CONFLICT (add/add): Merge conflict in sys-apps/cpi/ChangeLog
Auto-merging media-video/video-mv/Manifest
CONFLICT (add/add): Merge conflict in media-video/video-mv/Manifest
Auto-merging media-video/video-mv/ChangeLog
CONFLICT (add/add): Merge conflict in media-video/video-mv/ChangeLog
Automatic merge failed; fix conflicts and then commit the result.
* Failure result returned from Git |
So this was a rebase again? On the mv overlay it happens quite often, I've never seen on other overlays. So I should file this as layman bug? What is the correct command for that hard update?
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
|
 |
Gentoo Forums Forum Index
 Documentation, Tips & Tricks |
All times are GMT
Goto page Previous 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 Next
|
| Page 7 of 10 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
