| View previous topic :: View next topic |
| Author |
Message |
gringo
Advocate
Joined: 27 Apr 2003
Posts: 3793
|
 Posted: Tue Oct 18, 2011 9:40 am Post subject: |
 |
|
| mv wrote: | | The patch should be fixed now to build in all cases. |
nice, thanks a lot !
cheers |
|
| Back to top |
|
 |
costel78
Guru
Joined: 20 Apr 2007
Posts: 407
|
| Posted: Sat Jan 21, 2012 6:20 pm Post subject: |
|
|
Is it possible to port the init script to systemd ?
I looked over actual openrc script, but it is too far complex to simple migrating the commands 
_________________
Sorry for my English. I'm still learning this language. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Sun Jan 22, 2012 8:20 am Post subject: |
|
|
| costel78 wrote: | | Is it possible to port the init script to systemd? |
I don't know. AFAIK systemd does not have proper dependencies and probably also not a standard route during shutdown which is crucial for the script.
Actually, I consider the whole systemd a very bad idea (running permanently daemons which take resources only to save 1-2 seconds during booting is not sane), so I hope that I will never be forced to use it, and I am not at all interested in porting something to a system with a bad concept.
| Quote: | | I looked over actual openrc script, but it is too far complex to simple migrating the commands |
Essentially, it is the "start" and "stop" which would have to be called at the proper place. The other provided command are only for wrapper scripts to check the current state (e.g. whether it would be saved). Probably one could put a part in a "library" which is sourced. |
|
| Back to top |
|
|
costel78
Guru
Joined: 20 Apr 2007
Posts: 407
|
| Posted: Sun Jan 22, 2012 9:53 am Post subject: |
|
|
Thank you for your answer!
Yes, I already tried to simulate the logics behind and, so far, I found no way to start tar and bzip compress of the portage tree BEFORE /usr/portage was umounted.
What was the simulation:
- a separate partition for /usr/portage (without distfiles, of course) mounted automatically at boot via initscript (noauto in fstab)
- during shutdown compress it (tar -cjpf ....) and umount it
As you said, I found no way to prevent systemd to try to umount the partition and kill tar process.
It seems that all thing it's to risky.
_________________
Sorry for my English. I'm still learning this language. |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Sun Jan 22, 2012 10:19 am Post subject: |
|
|
| costel78 wrote: | | As you said, I found no way to prevent systemd to try to umount the partition and kill tar process. |
At least the mount and umount commands from sys-apps/util-linux provide a way to call helper scripts for mount and umount of certain filesystem types. I have never dealt with them and do not know how compatible this setting is. But it might be a possible to invent a "new" filesystem type and to do the actual action in these helper scripts. However, this needs to be well-tested: e.g. aufs also uses such helper scripts, so "recursive" callings of such scripts must be possible.
The advantage of this approach would be that it works with any init-system, the disadvantage is that it probably works only with util-linux.
Currently, I have no time to implement such a thing. |
|
| Back to top |
|
|
slycordinator
Advocate
Joined: 31 Jan 2004
Posts: 3065
Location: Korea
|
| Posted: Tue Feb 28, 2012 8:58 pm Post subject: |
|
|
I'm having a problem with mv's squash_dir initscript.
My portage squashfs file is at /squashed_dirs/portage.sqfs and the portage.readonly and portage.changes are placed inside that same directory. And I named the portage initscript squash_portage
My portage.changes directory has a whole bunch of subdirs in it currently, since I did an "emerge --sync" earlier, but if I do "/etc/init.d/squash_portage restart" I simply get:
| Code: | /etc/init.d/squash_portage restart
* Unmounting /usr/portage ... [ ok ]
* Mounting /squashed_dirs/portage.sqfs as /usr/portage ... [ ok ] |
And the portage.sqfs is not recompressed with the portage.changes directory containing the same stuff it did previously.
_________________
My political stance/bias
slycordinator != slycoordinator |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Wed Feb 29, 2012 9:55 am Post subject: |
|
|
| slycordinator wrote: | | And the portage.sqfs is not recompressed |
This is strange. I cannot reproduce it here. Are you sure that you do not have a .no-save file and not set any threshold?
If yes, please help me debugging: - Which version of squash_dir are you using? To simplify debugging, please try with the latest from the mv overlay (from today).
- Is the output of
| Code: | | /etc/init.d/squash_portage print_dir_change |
indeed your portage.changes which contains data? In particular, does | Code: | | ls $(/etc/init.d/squash_portage print_dir_change) |
show that content?Does some of the following two commands output "1" to stderr? | Code: | /etc/init.d/squash_portage need_squash
/etc/init.d/squash_portage will_squash |
|
|
| Back to top |
|
|
slycordinator
Advocate
Joined: 31 Jan 2004
Posts: 3065
Location: Korea
|
| Posted: Wed Feb 29, 2012 7:59 pm Post subject: |
|
|
Note: This happens sporadically. Like when I shutdown the computer last night, the portage squashfs file got recompressed and the ones I've got for /var/db and /usr/share/texmf-dist didn't, but at least now (using version 11.6) doing a "/etc/init.d/squash_db restart" made it recompress on the first try. And the need_squash and will_squash variables showed that the tex directory will be recompressed. Guess it's fixed but not sure, given how sporadic it would happen.
edit: PEBKAC. Think it was the threshold setting.
_________________
My political stance/bias
slycordinator != slycoordinator |
|
| Back to top |
|
|
js08
n00b
Joined: 04 Mar 2008
Posts: 47
|
| Posted: Sat Mar 24, 2012 12:36 am Post subject: |
|
|
Hi,
I'm no longer able to update my portage tree (squash_dir, unionfs_fuse [everything from mv-overlay])
sys-fs/fuse-2.8.7, kernel 3.2.11-gentoo or 3.3-gentoo, net-misc/rsync-3.0.9-r1
emerge --sync produces the following error:
| Code: |
app-backup/backup-manager/files/
app-backup/backupninja/
app-backup/backuppc/
app-backup/backuppc/files/
app-backup/backuppc/files/3.2.0/
app-backup/bacula/
app-backup/bacula/ChangeLog
40.52K 100% 59.51kB/s 0:00:00 (xfer#24, to-check=1056/4230)
app-backup/bacula/Manifest
8.54K 100% 12.22kB/s 0:00:00 (xfer#25, to-check=1055/4230)
app-backup/bacula/bacula-5.2.5.ebuild
10.09K 100% 14.42kB/s 0:00:00 (xfer#26, to-check=1048/4230)
app-backup/bacula/bacula-5.2.6.ebuild
10.09K 100% 14.09kB/s 0:00:00 (xfer#27, to-check=1047/4230)
rsync: rename "/usr/portage/app-backup/bacula/.bacula-5.2.6.ebuild.rJDBEZ" -> "app-backup/bacula/bacula-5.2.6.ebuild": Software caused connection abort (103)
app-backup/bacula/files/
app-backup/bacula/files/5.0.2/
app-backup/bacula/files/5.0.3/
app-backup/bacula/files/5.2.3/
app-backup/boxbackup/
app-backup/boxbackup/files/
app-backup/ccollect/
app-backup/cdbackup/
app-backup/cdbkup/
app-backup/cpdup/
app-backup/cpdup/files/
app-backup/dar/
app-backup/dar/files/
rsync: failed to set times on "/usr/portage/app-crypt": Software caused connection abort (103)
rsync: opendir "/usr/portage/app-crypt" failed: Transport endpoint is not connected (107)
rsync: recv_generator: failed to stat "/usr/portage/app-crypt/metadata.xml": Transport endpoint is not connected (107)
rsync: recv_generator: mkdir "/usr/portage/app-crypt/WiRouterKeyRec" failed: Transport endpoint is not connected (107)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/usr/portage/app-crypt/acr38u" failed: Transport endpoint is not connected (107)
...
|
_________________
Train Hard Or Don't Train At All |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Sat Mar 24, 2012 10:25 am Post subject: |
|
|
| js08 wrote: | Hi,
I'm no longer able to update my portage tree (squash_dir, unionfs_fuse [everything from mv-overlay])
sys-fs/fuse-2.8.7, kernel 3.2.11-gentoo or 3.3-gentoo, net-misc/rsync-3.0.9-r1
emerge --sync produces the following error: |
And it worked before (i.e. with other kernels)? Then it looks to me like a race condition problem in unionfs-fuse or an incompatibility of fuse with the newest kernels.
If the start of the error message is not reproducible, I would guess on the former and suggest that you report the problem upstream at unionfs-fuse. |
|
| Back to top |
|
|
js08
n00b
Joined: 04 Mar 2008
Posts: 47
|
| Posted: Sat Mar 24, 2012 7:46 pm Post subject: |
|
|
| mv wrote: | | js08 wrote: | Hi,
I'm no longer able to update my portage tree (squash_dir, unionfs_fuse [everything from mv-overlay])
sys-fs/fuse-2.8.7, kernel 3.2.11-gentoo or 3.3-gentoo, net-misc/rsync-3.0.9-r1
emerge --sync produces the following error: |
And it worked before (i.e. with other kernels)? Then it looks to me like a race condition problem in unionfs-fuse or an incompatibility of fuse with the newest kernels.
If the start of the error message is not reproducible, I would guess on the former and suggest that you report the problem upstream at unionfs-fuse. |
yes, everything worked fine until 3-21-2012.
and yes the start of the error message is not reproducible. but everything else on two different hosts and 1 virtual machine.
_________________
Train Hard Or Don't Train At All |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Mon Apr 02, 2012 10:41 pm Post subject: |
|
|
| js08 wrote: | | and yes the start of the error message is not reproducible. but everything else on two different hosts and 1 virtual machine. |
As I mentioned, please report upstream at unionfs-fuse - this needs help by the corresponding specialist (it probably is some race issue). |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1820
Location: PB, Germany
|
| Posted: Wed Aug 29, 2012 11:09 am Post subject: |
|
|
Hi, since Portage gets too slow on my old ppc Notebook I found this Tip...
Where is the recent version of these init scripts maintained and who does it currently? I just found http://en.gentoo-wiki.com/wiki/Squashed_Portage_Tree, there are already newer versions than the original author wrote in the first post of this thread.
Usually I avoid using non-official approaches which tend to be un-maintained after a while, but since removing the SquashFS approach for the default Portage is just easy and the benefits of this tweak are just too nice.
Wouldn't it be nice to make the Howto a bit more official and centrally maintained by creating an ebuild with the init script in some of the overlays, so I don't need to copy/paste the init-script code from the browser into the editor?
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Wed Aug 29, 2012 11:22 am Post subject: |
|
|
The project (squash_dir, consisting not only of the init-scripts but also of a shell-wrapper with zsh-completion and an extensive README) is currently maintained on github.
The corresponding ebuild is in the mv overlay (available by layman). |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1820
Location: PB, Germany
|
| Posted: Fri Aug 31, 2012 11:14 am Post subject: |
|
|
Sorry that I overlooked the middle of this thread. I'm also using sys-fs/squash_dir now from your mv overlay. Thank you very much for maintaining these scripts, nice piece of work, especially the very good README.
The only issue for now is that aufs3 sources don't build on PPC. On amd64 its working fine.
Is there any COMPRESSION ("gzip", "xz", "lzo") advice for maximal performance (say slow 1core cpu and slow drive)? This is more important for me than space for some slow old machines and its the main reason that made me come to this thread, after reading Portage_tips.
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Fri Aug 31, 2012 8:52 pm Post subject: |
|
|
| Massimo B. wrote: | | The only issue for now is that aufs3 sources don't build on PPC. |
You might try the git sources (aufs-99999999.3, also available from the mv overlay). Let us hope that the kernel developers come to their mind and finally include aufs or at least overlayfs, although the latter has several drawbacks compared to aufs.
| Quote: | | Is there any COMPRESSION ("gzip", "xz", "lzo") advice for maximal performance |
For compression, lzo is undoubtfully the fastest (and least effective) and xz the slowest (and most effective). However, except for the time on shutdown only decompression time plays a role. Concerning decompression, lzo is also the fastest, but surprisingly xz is not much slower. Since xz needs to access less data (because it compresses much better), it can be that on some systems (especially those with not much memory or huge cache) xz is even the winner for decompression, speedwise. Your really have to try.
On the other hand, if you have a really slow system and a huge directory, it can be that compression time of xz is really inacceptable - on some systems, I end up with gtip for this reason... |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1820
Location: PB, Germany
|
| Posted: Mon Sep 03, 2012 8:50 am Post subject: |
|
|
Because I can't afford long startup or stop time, I restart the squashing on daily base: | Code: | $ cat /etc/cron.daily/1squashfs_flush
#!/usr/bin/env bash
for rc in /etc/runlevels/boot/squash_*; do
if [[ 0 != "$($rc will_squash 2>&1 >/dev/null)" ]]; then
$rc restart
fi
done |
Synchronizing a 1 week old Portage made this difference: | Code: | du -sh /usr/portage*
311M /usr/portage
52M /usr/portage.changes
280M /usr/portage.readonly
67M /usr/portage.sqfs |
...which already exceeded my threshold of 40M.
Btw, how do I understand the disk usage? Is it like this, /usr/portage and /usr/portage.readonly are only virtual sizes while the only real consumption on the host fs are these of the *.changes and *.sqfs files only?
I have 3 squash_dir setups now: /usr/portage, /var/db/ and /var/lib/layman.
Because I need to use webrsync on some hosts, I've seen that after a new webrsync of the same snapshot it seems that all has been re-written: | Code: | $ du -sh /usr/portage*
711M /usr/portage
711M /usr/portage.changes
280M /usr/portage.readonly
67M /usr/portage.sqfs |
Let's see if app-portage/emerge-delta-webrsync is different here.
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Mon Sep 03, 2012 1:16 pm Post subject: |
|
|
| Massimo B. wrote: | | Because I can't afford long startup or stop time, I restart the squashing on daily base: |
Are you aware that there is a squash_dir wrapper for this? | Quote: | | squash_dir restart |
should do the same as your script (only that by default it will stop on errors). Moreover the latter has the advantage that you can easily pass options to e.g. ignore the threshold.
| Quote: | | Btw, how do I understand the disk usage? Is it like this, /usr/portage and /usr/portage.readonly are only virtual sizes while the only real consumption on the host fs are these of the *.changes and *.sqfs files only? |
Exactly.
| Quote: | | I've seen that after a new webrsync of the same snapshot it seems that all has been re-written. |
This is because of the COW mechanism of aufs: If something changes in a file, even if it is only the filestamp or some attributes, the file must be copied into the .changes directory. I guess webrsync changes all filestamps, at least temporarily. With other union-type filesystem the necessary COW might be even more intensive, e.g. (IIRC this was the case in some versions of unionfs-fuse) if only the directory filestamp changes then the whole directory might need to be written to *.changes - it really depends which mechanisms are used to merge the directories. |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1820
Location: PB, Germany
|
| Posted: Mon Sep 03, 2012 1:33 pm Post subject: |
|
|
| mv wrote: | Are you aware that there is a squash_dir wrapper for this? | Quote: | | squash_dir restart should do the same as your s |
cript (only that by default it will stop on errors). Moreover the latter has the advantage that you can easily pass options to e.g. ignore the threshold.
|
No I wasn't, thanks. squash_dir restart really simplifies that task or cronjob.
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1820
Location: PB, Germany
|
| Posted: Thu Sep 06, 2012 1:22 pm Post subject: |
|
|
With some manual work to get aufs3 building on ppc as described in the bug [1], your squash_dir is working fine. You could add ~ppc keywords to squash_dir and runtitle.
[1] Bug 433497 - sys-fs/aufs3-3_p20120813: unrecognized command line option.
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1820
Location: PB, Germany
|
| Posted: Wed Sep 12, 2012 1:31 pm Post subject: |
|
|
When a new kernel has no aufs module, then squash_dir scripts fail but has half mounted the read.only parts. This is ok.
After I got the aufs module built and loaded, squash_dir restart or start do not work. | Code: | $ squash_dir restart
/etc/init.d/squash_db is not running
/etc/init.d/squash_layman is not running
/etc/init.d/squash_local_portage is not running
/etc/init.d/squash_portage is not running
$ squash_dir start
Starting db...
squash_db | * Mounting /var/db.sqfs as /var/db ...
squash_db |mount: /var/db.sqfs is already mounted
squash_db | * Failed mounting /var/db.sqfs as /var/db.readonly [exit with 1]
squash_db |rmdir: failed to remove `/var/db.readonly': Device or resource busy [ !! ]_db |
squash_db | * ERROR: squash_db failed to start
Skipping /etc/init.d/squash_layman due to previous error
Skipping /etc/init.d/squash_local_portage due to previous error
Skipping /etc/init.d/squash_portage due to previous error
|
I first needed to umount all read.only stuff to get it starting. Maybe you can add some error safety here.
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Wed Sep 12, 2012 1:57 pm Post subject: |
|
|
| Massimo B. wrote: | | When a new kernel has no aufs module, then squash_dir scripts fail but has half mounted the read.only parts. |
The problem is that it is not clear what to do:
One possibility is to add an "umount" option which tries to umount even if the start failed. However, this can have horrible undesired effects if the start failed for a different reason or if directories with the temporary name feature are involved.
The other possibility is to start anyway successfully in such a situation, because the directory is mounted but just not writable.
The disadvantage of the latter is that you will not realize that something is broken unless you study the logs (or actually attempt to write).
The real problem is that it is not possible to report to openrc a "partial success". |
|
| Back to top |
|
|
Massimo B.
Veteran
Joined: 09 Feb 2005
Posts: 1820
Location: PB, Germany
|
| Posted: Thu Sep 13, 2012 2:57 pm Post subject: |
|
|
I would like to improve performance, throughput and disk usage on a WebDAV server, which I mount via net-fs/davfs2. Now I think about if it is a good idea to store the content with squash_dir there.
Disadvantage would be of course that I can't view the files from the web-interface anymore. But I use the WebDAV only as central synchronisation for different sites using net-misc/unison.
Now the problem would be that maybe any change on the re-compressed squashfs would lead to a complete upload of the whole file again instead of sending delta only. Maybe this reason makes squash_dir the wrong solution for that, what do you think?
_________________
HP ZBook Power 15.6" G8 i7-11800H|HP EliteDesk 800G1 i7-4790|HP Compaq Pro 6300 i7-3770 |
|
| Back to top |
|
|
mv
Watchman
Joined: 20 Apr 2005
Posts: 6780
|
| Posted: Thu Sep 13, 2012 4:02 pm Post subject: |
|
|
| Massimo B. wrote: | | Maybe this reason makes squash_dir the wrong solution for that, what do you think? |
Yes, I guess this is the wrong solution. Maybe you can use a VCS like git instead? |
|
| Back to top |
|
|
Gordex
n00b
Joined: 10 Jul 2008
Posts: 26
|
| Posted: Sat Sep 15, 2012 4:46 am Post subject: |
|
|
tried it today but didnt get it to work properly :/
Compression-var seems to be ignored..I got gzip compression all the way
an error message indicated that it tried to umount the readonly before the aufs when shutdown
openrc doesnt seem to launch it altho added to default runlevel..as there was no output in the rc.log and no proper mounts after reboot
I guess the script is a bit outdated
anyway nice idea man! |
|
| Back to top |
|
|
|
Documentation, Tips & Tricks
