e2compr - transparent compression filesystem

[matthiaswinkler]

Hi,

I'm working currently on the ext2 compression filesystem.
Longtime this project seemed to be abandoned so I started to do it myself.

I finished a beta version and I'm seeking for some feedback.
http://www.xeer.de/e2compr2.6.21.1_beta1.patch

The patch can be applied to a current 2.6.21.1 kernel. I also
tested the patch today on a 2.6.22.5. There is one, very easy to solve,
reject (.rej). So to give it a try:

- Download one of those kernels
- extract
- cd linux-2.6.xxx
- patch -p1 < e2compr2.6.21.1_beta1.patch
- make menuconfig (and configure the feature: filesystems / ext2 / compression / use bzip res. gzip.)
- make ....

I would be very happy about some expiriences and error
descriptons.


greets Matthias


Major changes:
* completly ported inode->i_mutex

* clever CONFIG_GZ_HACK to reject "uncompressable" files
(according to their extension) early. The IOCTL in ioctl.c
to set the compression on the file already rejects such
extensions now.

* new create_empty_buffers_e2c() was necessary, because the
"extra" pages shouldn't have a valid i_mapping!

* proper block initalization (bdev-bug) in:
- create_empty_buffers_e2c()
- ext2_get_cluster_blocks

* in file.c copied:
ext2_mapping_read from generic_mapping_read() from 2.6.21.1 mm/filemap.c
ext2_file_aio_read from generic_file_aio_read from 2.6.21.1 mm/filemap.c

...with one single change at ext2_mapping_read in label page_ok:
A new Page-Flag (page-flags.h) the so called "PG_compr"-Flag is
checked to assure the corresponding page is not under
compression/decompression. This was necessary because
generic_mapping_read() doesn't lock() the page in ALL cases!!!
Otherwise the generic_mapping_read() would have to lock EVERY page
in the whole system before returning it....

* Fixed HiMem-Support: Balanced ALL kamp/kunmap calls. Unbalanced
functions cause the system to hang at "kmap_himem()" after some
time. Can be seen with magic-sysctrl and "altgr + prtscr + W".

* ext2_decompres_cluster() didn't mark uptodate pages for writeback.
Don't know how this method could have ever worked...

* ext2_compress_cluster() caused an always increasing amount of dirty-pages
(cat /proc/vmstat) which couldn't be wrote back by sync/umount.
I think this was due the ClearPageDirty ath the end of ext2_compress_cluster().

* introduced ext2_get_dcount() to savely determine if a file is really "open"
and to abort compression/decompression in that case.

* Removed gzip completely and not working assembler code. Replaced by the
kernels zlib, which is pretty the same code...

* New kernel configuration interface

* Rollback of some unecessary "fixes"...

[matthiaswinkler]

maybe I should mention this: it's BETA ... please use it at your own risk!

[phil_r]

I've applied the patch but I don't see the compression option.
_________________
Just when you think you know the answers, I change the questions.

[phil_r]

I recompiled the kernel anyway (made another unrelated change), now this happens:-

LD fs/exportfs/built-in.o
CC [M] fs/exportfs/expfs.o
LD [M] fs/exportfs/exportfs.o
LD fs/ext2/built-in.o
CC [M] fs/ext2/balloc.o
fs/ext2/balloc.c:23:29: error: linux/ext2_fs_c.h: No such file or directory
In file included from fs/ext2/balloc.c:24:
fs/ext2/debug.h:124:1: warning: "CHECK_PAGE" redefined
fs/ext2/debug.h:44:1: warning: this is the location of the previous definition
fs/ext2/debug.h:138:1: warning: "DUMP" redefined
fs/ext2/debug.h:127:1: warning: this is the location of the previous definition
fs/ext2/balloc.c: In function ‘ext2_get_group_desc’:
fs/ext2/balloc.c:54: warning: implicit declaration of function ‘EXT2_SB’
fs/ext2/balloc.c:54: warning: initialization makes pointer from integer without a cast
fs/ext2/balloc.c:56: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:57: warning: implicit declaration of function ‘ext2_error’
fs/ext2/balloc.c:60: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:65: warning: implicit declaration of function ‘EXT2_DESC_PER_BLOCK_BITS’
fs/ext2/balloc.c:66: warning: implicit declaration of function ‘EXT2_DESC_PER_BLOCK’
fs/ext2/balloc.c:67: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:75: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:77: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:78: error: invalid use of undefined type ‘struct ext2_group_desc’
fs/ext2/balloc.c: In function ‘read_block_bitmap’:
fs/ext2/balloc.c:98: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:105: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c: In function ‘reserve_blocks’:
fs/ext2/balloc.c:116: warning: initialization makes pointer from integer without a cast
fs/ext2/balloc.c:117: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:121: warning: implicit declaration of function ‘percpu_counter_read_positive’
fs/ext2/balloc.c:121: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:122: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:128: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:129: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:129: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:140: warning: implicit declaration of function ‘percpu_counter_mod’
fs/ext2/balloc.c:140: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c: In function ‘release_blocks’:
fs/ext2/balloc.c:148: warning: initialization makes pointer from integer without a cast
fs/ext2/balloc.c:150: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c: At top level:
fs/ext2/balloc.c:156: warning: ‘struct ext2_sb_info’ declared inside parameter list
fs/ext2/balloc.c:156: warning: its scope is only this definition or declaration, which is probably not what you want
fs/ext2/balloc.c: In function ‘group_reserve_blocks’:
fs/ext2/balloc.c:160: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:163: warning: implicit declaration of function ‘sb_bgl_lock’
fs/ext2/balloc.c:164: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:167: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c: In function ‘group_release_blocks’:
fs/ext2/balloc.c:177: warning: initialization makes pointer from integer without a cast
fs/ext2/balloc.c:181: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:182: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c: In function ‘ext2_free_blocks’:
fs/ext2/balloc.c:200: warning: initialization makes pointer from integer without a cast
fs/ext2/balloc.c:202: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:212: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:214: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:221: warning: implicit declaration of function ‘ext2_debug’
fs/ext2/balloc.c:225: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:226: warning: implicit declaration of function ‘EXT2_BLOCKS_PER_GROUP’
fs/ext2/balloc.c:227: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:246: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:246: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:247: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:247: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:248: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:248: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:248: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:250: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:250: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:250: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c: In function ‘ext2_new_block’:
fs/ext2/balloc.c:361: warning: initialization makes pointer from integer without a cast
fs/ext2/balloc.c:362: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:364: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:369: error: ‘EXT2_DEFAULT_PREALLOC_BLOCKS’ undeclared (first use in this function)
fs/ext2/balloc.c:369: error: (Each undeclared identifier is reported only once
fs/ext2/balloc.c:369: error: for each function it appears in.)
fs/ext2/balloc.c:390: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:391: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:392: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:393: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:404: warning: passing argument 1 of ‘group_reserve_blocks’ from incompatible pointer type
fs/ext2/balloc.c:406: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:416: warning: passing argument 1 of ‘grab_block’ makes pointer from integer without a cast
fs/ext2/balloc.c:432: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:434: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:440: warning: passing argument 1 of ‘group_reserve_blocks’ from incompatible pointer type
fs/ext2/balloc.c:452: warning: passing argument 1 of ‘grab_block’ makes pointer from integer without a cast
fs/ext2/balloc.c:459: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:475: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:478: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:480: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:481: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:482: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:482: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:482: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:488: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:492: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c:507: warning: implicit declaration of function ‘EXT2_I’
fs/ext2/balloc.c:507: error: invalid type argument of ‘->’
fs/ext2/balloc.c:523: error: invalid type argument of ‘->’
fs/ext2/balloc.c: In function ‘ext2_count_free_blocks’:
fs/ext2/balloc.c:599: error: invalid type argument of ‘->’
fs/ext2/balloc.c:603: error: dereferencing pointer to incomplete type
fs/ext2/balloc.c: In function ‘block_in_use’:
fs/ext2/balloc.c:612: error: invalid type argument of ‘->’
fs/ext2/balloc.c:612: error: invalid type argument of ‘->’
fs/ext2/balloc.c:612: error: invalid type argument of ‘->’
fs/ext2/balloc.c: In function ‘ext2_bg_has_super’:
fs/ext2/balloc.c:644: warning: implicit declaration of function ‘EXT2_HAS_RO_COMPAT_FEATURE’
fs/ext2/balloc.c:644: error: ‘EXT2_FEATURE_RO_COMPAT_SPARSE_SUPER’ undeclared (first use in this function)
fs/ext2/balloc.c: In function ‘ext2_bg_num_gdb’:
fs/ext2/balloc.c:661: error: ‘EXT2_FEATURE_RO_COMPAT_SPARSE_SUPER’ undeclared (first use in this function)
fs/ext2/balloc.c:664: error: invalid type argument of ‘->’
make[2]: *** [fs/ext2/balloc.o] Error 1
make[1]: *** [fs/ext2] Error 2
make: *** [fs] Error 2
[root@gateway linux]#
_________________
Just when you think you know the answers, I change the questions.

[matthiaswinkler]

Hi Phil,

I guess you are using the gentoo-sources. The patchs works fine
with the described vanilla source from kernel.org.

To fix it for the gentoo-sources you must fix the rejects,
produced after patch command:

find ./linux-2.6.21.1 | grep "\.rej"

and now manually process every file and
insert the code fagemnts which could't be patched automtically.

but this all is for experienced users.
it's a beta version of a filesystem driver.
unless you want to help developping you shouldn't
go any further. you might loose your data or
crash your system....

[matthiaswinkler]

Hi Everybody!

I released a new patch which I consider as stable, so far.
It is possible to patch it to the current 2.6.22.6-kernel
and it does all the necessary kernel configuration now by
itself. Get it here:

http://www.xeer.de/e2compr2.6.22.5.patch.gz


I also did fix up the remaining errors:

25 Sep 2007
Matthias Winkler <m.winkler@unicon-ka.de>
* System stalled with a lot of I/O during compression of
USB-Sticks. Seems generic_osync_inode() should not be
called in ext2_compress_cluster. Therefore I replaced
it with ll_rw_block() to write the modified blocks
directly back to disk. This gave also a ~100% better
performance for compression.

9 Sep 2007
Matthias Winkler <m.winkler@unicon-ka.de>
* fixed bdev-bug. this bug appeared primarily when
files contained holes. A page with holes, which
was dirty caused ext2_get_cluster_blocks [ext2_get_block()]
to create ALL blocks of the page, even if there were holes!
These allocated hole-blocks weren't set to 0 anywhere and
therefore contained invalid data. I changed the
code to never allocate these holes.

* ext2_truncate() added again to ext2_compress_cluster for
uncompressed clusters. Fixes filesize errors reported by
"e2fsck -f /dev/..."