| View previous topic :: View next topic |
| Author |
Message |
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
 Posted: Sat Sep 15, 2012 1:20 pm Post subject: |
 |
|
| Gordex wrote: | | tried it today but didnt get it to work properly :/ |
Hard to say what you tried, but none of the effects you described were ever reported or did ever occur on one of my systems.
| Quote: | | Compression-var seems to be ignored..I got gzip compression all the way |
Perhaps you have not compiled squashfs-tools with support for any other compression algorithm. If you do not set COMPRESSION, the default is even xz.
| Quote: | | an error message indicated that it tried to umount the readonly before the aufs when shutdown |
The code does it certainly in the correct way. Perhaps you had mounted aufs manually in addion or the umounting failed for some reason. Without more information, it is not possible to say what went wrong.
| Quote: | | openrc doesnt seem to launch it altho added to default runlevel.. |
This is the strangest thing, of course, because whether openrc starts it or not is not a question of the script. It indicates that there is something completely broken in your setup - perhaps the names of init.d/... and conf.d/ do not match, or you tried to add squash_dir instead of the symlink or whatever. Again, without more information, it is not possible to say what went wrong. |
|
| Back to top |
|
 |
Gordex
n00b
Joined: 10 Jul 2008
Posts: 26
|
| Posted: Sat Sep 15, 2012 3:07 pm Post subject: |
|
|
I tried to doublecheck everything according to the README and got it working somehow now I believe  thanksalot.. the problem remaining is that COMPRESSION="xz" or lzo .. are still ignored .. I can manually squash it that way (mksquashfs -comp xz ..) but ur script takes gzip here anyway
not sure why |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Sat Sep 15, 2012 3:40 pm Post subject: |
|
|
Try to insert into line 495 of /etc/init.d/squash_dir:
| Code: | | eval "einfo mksquashfs options: ${sqfsdaddopt}" |
When compressing, this should display you the -comp-Option actually used. If -comp does not appear here, somewhere COMPRESSION must have been explicitly defined to the empty string in your config, because the default "xz" is set in line 311. |
|
| Back to top |
|
|
Gordex
n00b
Joined: 10 Jul 2008
Posts: 26
|
| Posted: Sat Sep 15, 2012 4:29 pm Post subject: |
|
|
I did but nothin seems to have changed -_-
| Code: | # /etc/init.d/squash_portage start
* It seems you start squash_portage for the first time with this configuration:
* The squashed file /usr/portage.sqfs does not exist yet.
* So I will try to initialize...
* Squashing /usr/portage to /usr/portage.sqfs
Parallel mksquashfs: Using 4 processors
Creating 4.0 filesystem on /usr/portage.sqfs, block size 131072.
[===========================================================================================================
===========================================================================================================
================================================================================|] 133549/133549 100%
Exportable Squashfs 4.0 filesystem, gzip compressed, data block size 131072
compressed data, compressed metadata, compressed fragments, no xattrs
duplicates are removed
Filesystem size 68404.20 Kbytes (66.80 Mbytes)
26.03% of uncompressed filesystem size (262780.17 Kbytes)
Inode table size 1658360 bytes (1619.49 Kbytes)
32.97% of uncompressed inode table size (5030387 bytes)
Directory table size 1511461 bytes (1476.04 Kbytes)
36.17% of uncompressed directory table size (4178856 bytes)
Number of duplicate files found 8488
Number of inodes 157094
Number of files 133523
Number of fragments 2009
Number of symbolic links 0
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 23571
Number of ids (unique uids + gids) 1
Number of uids 1
portage (250)
Number of gids 1
portage (250)
* Removing all in /usr/portage
* Now its content is only available as squashfs.
* Name of the squashfs file: /usr/portage.sqfs
* Mounting /usr/portage.sqfs as /usr/portage ... |
Admin edit: Added line breaks to unbroken line of "=" characters. --pjp |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Sat Sep 15, 2012 5:01 pm Post subject: |
|
|
| Gordex wrote: | | I did but nothin seems to have changed |
Now I understand: You were speaking about the "initial" mksquashfs, not about the one which is "normally" used when the directory is re-squashed.
You are right, here the option is ignored by mistake. Thanks for the report. I will fix it. |
|
| Back to top |
|
|
Gordex
n00b
Joined: 10 Jul 2008
Posts: 26
|
| Posted: Sat Sep 15, 2012 5:20 pm Post subject: |
|
|
ah awesome I see.. thx
edit: you might add that aufs3 needs to be compiled with useflag fuse..I think thats what causes my earlier problems but not 100% sure |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1810
Location: PB, Germany
|
| Posted: Mon Nov 12, 2012 2:00 pm Post subject: |
|
|
Hi, again. Is there any rescue if aufs is not available?
Today I started my new kernel, but I forgot to build aufs3 kernel module. Now the emerge aufs3 works but fails to install, probably because /var/db is readonly.
I tried to install the module and the binaries from /var/tmp/portage manually but modprobe aufs3 still says "module not found" eventhough the .ko file is in the right directory (and found by bash_completion).
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Mon Nov 12, 2012 5:38 pm Post subject: |
|
|
| Massimo B. wrote: | | Hi, again. Is there any rescue if aufs is not available? |
Yes, but it is inconvenient: You can manually copy the readonly /var/db to some new temporary directory, then stop the squash_...-scripts, rename, /var/db to something else, and then rename the new temporary directory to /var/db. (If /var/db does not even a sane [readonly] content, you can also use unsquashfs to unpack /var/db.sqfs manually).
Of course, this works only if you have sufficient disk space for the unpackad copy of /var/db, but perhaps you can temporarily get the free space by moving something to a backup medium which you do not need for the restauration.
Then emerge aufs (hopefully it works now).
Finally rename both directories back and then reboot (or start squash_... if it works).
When everything is running you should remove the old data of /var/db/pkg/sys-fs/aufs-... and move the new data (from your last emerge) which is now in the renamed-back temporary directory to /var/db/pkg/sys-fs/aufs-...
After this the temporary directory can be deleted. |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1810
Location: PB, Germany
|
| Posted: Tue Nov 13, 2012 11:19 am Post subject: |
|
|
Thanks for the help. That worked. Though I needed to force ORDER="aufs" in all the configurations, maybe because of the recent change. I think ORDER does mean the order of testing. But eventhough aufs already was in lsmod, squash_dir restart failed because of missing overlayfs.
If aufs3 will be the next default aufs I hope it goes to the kernel soon without requiring the kernel patches. Next kernel switch I need to remember to emerge aufs3 first against the new sources and build the aufs3-patched sources afterwards.
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Tue Nov 13, 2012 5:08 pm Post subject: |
|
|
| Massimo B. wrote: | | I think ORDER does mean the order of testing. But eventhough aufs already was in lsmod, squash_dir restart failed because of missing overlayfs. |
Yes, ORDER means the order of trying. So it should work even with the default ORDER but perhaps only print some error messages - but it should not fail if aufs succeeds.
| Quote: | | If aufs3 will be the next default aufs I hope it goes to the kernel soon without requiring the kernel patches. |
Unfortunately, the chances for this are practically zero, since despite a huge interest from the users, the kernel maintainers are not interested in including any type of union filesystem. Some kernel maintainers had announced that perhaps they might give overlayfs a chance. Although I am not so optimistic (since many months it is ignored, practically without any comments - the same has already happened to union mount for many years), maybe this is the first union type filesystem which has a chance to go into the kernel. For this reason, I have changed the default. I will of course change the default immediately back, if it seems more likely that aufs is included or overlayfs is going to be rejected. |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1810
Location: PB, Germany
|
| Posted: Thu Nov 22, 2012 8:53 am Post subject: |
|
|
Thanks, the discussion about integrating aufs3 was the same with squashfs which finally was integrated into 2.6.29 after half of the Linux world was already using it. [german article]
Some other idea, when I was thinking about compressing binaries such as large firefox using UPX for faster startup: I could also just add things like /usr/lib64/firefox to squash_dir. That should just work with upgrades and re-squashing for now, since I even restart squash_dir once per day by cron for potential re-squashing.
EDIT: I successfully squashed firefox, from 43M to 20M, and startup feels a bit faster...
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1810
Location: PB, Germany
|
| Posted: Tue Dec 04, 2012 9:02 am Post subject: |
|
|
Hi again. I have a daily cronjob for re-squashing. How save actually is re-squashing by squash_dir restart when currently writing to this filesystem?
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Tue Dec 04, 2012 3:01 pm Post subject: |
|
|
| Massimo B. wrote: | | Hi again. I have a daily cronjob for re-squashing. How save actually is re-squashing by squash_dir restart when currently writing to this filesystem? |
It is not save; you can loose any new information and perhaps even get a broken squash file. Even the remounting itself might fail if the filesystem is busy at that time. |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1810
Location: PB, Germany
|
| Posted: Wed Dec 19, 2012 10:07 am Post subject: squash_dir from rescue live-cd chroot |
|
|
Before the last openrc update | Code: | Tue Dec 18 21:18:32 2012 <<< sys-apps/openrc-0.9.8.4
Tue Dec 18 21:18:40 2012 >>> sys-apps/openrc-0.11.8 |
..it was just possible to do squash_dir start or restart from inside a live-cd chroot. There (using Finnix on PPC) aufs and squashfs modules are provided. It was possible to mount the squash_dir parts rw and even recompress. So portage was completely functional from inside chroot.
Since the latest openrc update I get the following: | Code: | $ squash_dir zap
* You are attempting to run an openrc service on a
* system which openrc did not boot.
* You may be inside a chroot or you may have used
* another initialization system to boot this system.
* In this situation, you will get unpredictable results!
* If you really want to do this, issue the following command:
* touch /run/openrc/softlevel
/etc/init.d/squash_db is not running... |
Doing the touch /run... openrc tries to start bad dependencies: | Code: | $ squash_dir start
Starting db...
* Caching service dependencies ... [ ok ]
* Mounting security filesystem ... [ ok ]
* Mounting debug filesystem ... [ ok ]
* Mounting cgroup filesystem ... [ ok ]
* /dev is already mounted
* Starting udev ... [ ok ]
* Populating /dev with existing devices through uevents ... [ ok ]
* Waiting for uevents to be processed ... [ ok ]
* Checking local filesystems ...
/dev/sda5 is in use.
e2fsck: Cannot continue, aborting.
* Operational error [ !! ]
* Mounting local filesystems ... [ ok ]
* Mounting /var/db.sqfs as /var/db ...
mount: /var/db.sqfs is already mounted
* Failed mounting /var/db.sqfs as /var/db.readonly [exit with 1]
rmdir: failed to remove �/var/db.readonly�: Device or resource busy [ !! ]
* ERROR: squash_db failed to start
Skipping /etc/init.d/squash_layman due to previous error
Skipping /etc/init.d/squash_local_portage due to previous error
Skipping /etc/init.d/squash_portage due to previous error |
Is there any way to avoid the dependencies to get squash_dir in a working state within chroot as it was working before?
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Wed Dec 19, 2012 5:24 pm Post subject: Re: squash_dir from rescue live-cd chroot |
|
|
| Massimo B. wrote: | | squash_dir start or restart from inside a live-cd chroot. |
In the new squash_dir-12.6 there is now a START command which executes the action of the "start" command but does not actually start the script. You can use the STOP command (which exists already quite a while) to execute the action of the "stop" command of such an officially non-started script. Of course, you must use these hacks very carefully: Double execution or stopping can have undesired results, especially if the temporary filename feature is used for the readonly directory. |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1810
Location: PB, Germany
|
| Posted: Tue Jan 08, 2013 9:14 am Post subject: |
|
|
Thanks. I was able to start squash_dir even within chroot from live-cd now.
Next bug report, I tried to squash new directories: | Code: | $ cat /etc/conf.d/squash_icedtea6
DIRECTORY="/usr/lib/icedtea6"
DIR_CHANGE="${DIRECTORY}.changes"
DIR_SQUASH="${DIRECTORY}.readonly"
ORDER="aufs"
COMPRESSION="" |
| Code: | $ /etc/init.d/squash_icedtea6 start
squash_icedtea6 | * Caching service dependencies ... [ ok ]
squash_icedtea6 | * It seems you start squash_icedtea6 for the first time with this configuration:
squash_icedtea6 | * The squashed file /usr/lib64/icedtea6.sqfs does not exist yet.
squash_icedtea6 | * So I will try to initialize...
squash_icedtea6 | * Squashing /usr/lib64/icedtea6 to /usr/lib64/icedtea6.sqfs
squash_icedtea6 |SYNTAX:mksquashfs source1 source2 ... dest [options] [-e list of exclude... |
EDIT: Parameter order has changed somehow, works now with this patch: | Code: | $ diff -u /etc/init.d/squash_dir /etc/init.d/squash_dir.new
--- /etc/init.d/squash_dir 2013-01-08 10:54:05.946782911 +0100
+++ /etc/init.d/squash_dir.new 2013-01-08 10:53:21.788301570 +0100
@@ -247,8 +247,8 @@
return 2
}
einfo "Squashing ${DIRECTORY} to ${FILE_SQFS}"
- mksquashfs ${COMPRESSION:+-comp "${COMPRESSION}"} ${MKSQUASHFS} \
- "${DIRECTORY}" "${FILE_SQFS}" || {
+ mksquashfs "${DIRECTORY}" "${FILE_SQFS}" \
+ ${COMPRESSION:+-comp "${COMPRESSION}"} ${MKSQUASHFS} || {
Error ${?} "mksquashfs failed. Exiting."
return
} |
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Tue Jan 08, 2013 8:45 pm Post subject: |
|
|
| This is fixed in squash_dir-12.8 |
|
| Back to top |
|
|
tabascoz
n00b
Joined: 14 May 2003
Posts: 45
Location: Rio de Janeiro
|
| Posted: Thu Aug 01, 2013 6:10 pm Post subject: |
|
|
Hello,
Did someone has written a systemd init script for squash_dir? Since the last update to gnome 3.8 and, thus, the need to switch to systemd , i had been unable to use this great toot.
_________________
Tabascoz |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Fri Aug 02, 2013 8:48 am Post subject: |
|
|
| tabascoz wrote: | | Did someone has written a systemd init script for squash_dir? |
AFAIK systemd is too poor to do this. I will not support this bad systemd concept, anyway. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Wed Aug 14, 2013 5:51 pm Post subject: |
|
|
| mv wrote: | | AFAIK systemd is too poor to do this. |
It turned out that it can be done with systemd as well - only it is not possible to send the output to the terminal; that's a restriction due to the bad systemd concept. Thus, here is the announcement
squash_dir-13.0 does support systemd, experimentally.
It also contains a script openrc-wrapper which allows to use many native openrc services in other init-systems like systemd. This might be interesting also for other openrc init-scripts.
So this is the implementation: The script is native openrc, but can be fully used in systemd by the openrc-wrapper. The advantage of this solution for gentoo users is that the same configuration can be used for systemd as for openrc, i.e. squash_dir does not have to be configured twice on different places. The disadvantage is of course that native systemd users might not want to configure in /etc/conf.d and to create squash_* symlinks in /etc/init.d which is currently necessary even if openrc is not installed. In the long run, squash_dir might be replaced e.g. by a perl-script which manages all functionality in a single script which can be called externally and which manages its own database of "mounted" directories in e.g. /run, but I will not have time in the near future to implement such a universal solution. Anyway, this would be different project and should have a different name... |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20496
|
|
| Back to top |
|
|
Khumarahn
Apprentice
Joined: 17 Apr 2009
Posts: 199
|
| Posted: Mon Sep 24, 2018 1:23 pm Post subject: |
|
|
What is the state of squashfs with squashdelta for portage tree?
Using the lines from teh wiki
| Code: |
# for daily squashfs snapshots
sync-type = squashdelta
sync-uri = mirror://gentoo/../snapshots/squashfs
|
results in
| Code: |
# emerge --sync
!!! Repository 'gentoo' has sync-type attribute set to unsupported value: 'squashdelta'
!!! Installed sync-types are: '['cvs', 'git', 'rsync', 'svn', 'webrsync']'
!!! Repository 'gentoo' has sync-type attribute set to unsupported value: 'squashdelta'
!!! Installed sync-types are: '['cvs', 'git', 'rsync', 'svn', 'webrsync']'
|
|
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Mon Sep 24, 2018 7:20 pm Post subject: |
|
|
| Khumarahn wrote: | | What is the state of squashfs with squashdelta for portage tree? |
It is abandoned, and AFAIK there are no plans to revive it. |
|
| Back to top |
|
|
Khumarahn
Apprentice
Joined: 17 Apr 2009
Posts: 199
|
| Posted: Mon Sep 24, 2018 7:40 pm Post subject: |
|
|
Thanks!!
What is the easiest way to keep portage in squashfs, and recompress in on emerge --sync? |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20496
|
| Posted: Mon Sep 24, 2018 8:49 pm Post subject: |
|
|
I've been using tmpfs and squashfs.
Create a tmpfs.
Create a squasfs of the portage tree.
When I sync, the squashfs is extracted to a tmpfs, the sync occurs to the in memory tmpfs and then a new squashfs is created. The tmpfs is then discarded (freeing the memory) and the new portage tree squashfs is mounted.
That process requires enough free memory to temporarily hold the tree during that sync / squashfs process.
Also, /usr/portage/distfiles and /usr/portage/packages are kept elsewhere. These directories were configurable using
DISTDIR & PKGDIR in make.conf. man make.conf now references repos.conf. I've not found new documentation for their use in repos.conf (but since make.conf seems to work, I haven't made it a priority).
If I ever get around to making the script more generic, I had planned to post it.
_________________
Quis separabit? Quo animo? |
|
| Back to top |
|
|
|
Documentation, Tips & Tricks
