Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
split a file like a raid0
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
SarahS93
l33t
l33t


Joined: 21 Nov 2013
Posts: 728

PostPosted: Sun Jun 09, 2024 1:21 am    Post subject: split a file like a raid0 Reply with quote

i put a videofile in a luks container with detached header. that is not enough for me.
what ways are there to split the luks file by put every second byte into file2 and all other not second bytes into file1?
like a (software) raid0 with 2 files/disks.
software raid0 i dont like becouse there are raid superblocks with them are visible that file1 and file2 are from a raid.
byte1 -> file1
byte2 -> file2
byte3 -> file1
byte4 -> file2
byte4 -> file1
....
or not bytes, chunks a little bit bigger?

i think security through obscurity is what i am looking for.
is there a way to add file1 and file2 on the fly together to mount them and use it, like a raid0 with mdadm?
Back to top
View user's profile Send private message
Banana
Moderator
Moderator


Joined: 21 May 2004
Posts: 1721
Location: Germany

PostPosted: Sun Jun 09, 2024 8:05 am    Post subject: Reply with quote

I do not have a verified solution, but those to topics here, sound familiar:
https://www.linuxquestions.org/questions/linux-general-1/linux-command-to-copy-every-other-byte-from-file-708566/
https://www.unix.com/shell-programming-and-scripting/130940-split-file-into-chunks-low-high-byte.html
_________________
Forum Guidelines

PFL - Portage file list - find which package a file or command belongs to.
My delta-labs.org snippets do expire
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54577
Location: 56N 3W

PostPosted: Sun Jun 09, 2024 9:19 am    Post subject: Reply with quote

SarahS93,

There is no need to have raid superblocks. You need to have that data in a notebook though.

The very old raid format supported external superblocks or no superblqcks.
The drawback is that only manual assembly is possible and the raid elements must be listed in the right order at assemble time.

Raid0 gets you 'chunksize' pieces, not bytes. You can also have a lot more that two elements in a raid0

If you want to play, make a few empty files.
Use losetup to attach them to /dev/loopX
Tell mdadm to assemble the /dev/loop devices into a raid set. Do not use --create, as that writes the metadata that you don't want.
Make a filesystem on the raid device.

Raid5 will give you redundant data unless you use the special device missing for one raid element.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
SarahS93
l33t
l33t


Joined: 21 Nov 2013
Posts: 728

PostPosted: Fri Jun 14, 2024 3:26 pm    Post subject: Reply with quote

fallocate -l 10G datei1
fallocate -l 10G datei2
fallocate -l 10G datei3
losetup /dev/loop1 datei1
losetup /dev/loop2 datei2
losetup /dev/loop3 datei3
mdadm --assemble /dev/md0 --level=0 --raid-devices=3 /dev/loop1 /dev/loop2 /dev/loop3
mdadm: :option --level not valid in assemble mode

mdadm --assemble /dev/md0 /dev/loop1 /dev/loop2 /dev/loop3
mdadm: no recogniseable superblock on /dev/loop1
mdadm: /dev/loop1 has no superblock - assembly aborted

do i need "--build" first? and after that "--assemble"?
( https://raid.wiki.kernel.org/index.php/A_guide_to_mdadm#Build
mdadm --build /dev/md0 --level=0 --raid-devices=3 /dev/loop1 /dev/loop2 /dev/loop3
mdadm: RUN_ARRAY failed: Invalid argument

what do i make wrong?


:edit
grep "CONFIG_MD_RAID" /usr/src/linux/.config
Code:
# CONFIG_MD_RAID0 is not set
# CONFIG_MD_RAID1 is not set
# CONFIG_MD_RAID10 is not set
# CONFIG_MD_RAID456 is not set
is that the problem?!
Back to top
View user's profile Send private message
SarahS93
l33t
l33t


Joined: 21 Nov 2013
Posts: 728

PostPosted: Sat Jun 15, 2024 4:53 am    Post subject: Reply with quote

with set "CONFIG_MD_RAID0=y" it works
Code:
mdadm --build /dev/md0 --level=0 --raid-devices=3 /dev/loop1 /dev/loop2 /dev/loop3
mdadm: array /dev/md0 built and started.


Code:
mdadm --detail /dev/md0
/dev/md0:
           Version :
     Creation Time : Sat Jun 15 06:54:48 2024
        Raid Level : raid0
        Array Size : 31457280 (30.00 GiB 32.21 GB)
      Raid Devices : 3
     Total Devices : 3

             State : clean
    Active Devices : 3
   Working Devices : 3
    Failed Devices : 0
     Spare Devices : 0

            Layout : -unknown-
        Chunk Size : 64K

Consistency Policy : none

    Number   Major   Minor   RaidDevice State
       0       7        1        0      active sync   /dev/loop1
       1       7        2        1      active sync   /dev/loop2
       2       7        3        2      active sync   /dev/loop3






with
Code:
mdadm --verbose --assemble /dev/md0   /dev/loop1 /dev/loop2 /dev/loop3
mdadm: looking for devices for /dev/md0
mdadm: no recogniseable superblock on /dev/loop1
mdadm: /dev/loop1 has no superblock - assembly aborted

it doesnt work

i do not understand "assemble"?



is "Chunk Size : 64K" OK?

is the build-way OK?
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54577
Location: 56N 3W

PostPosted: Sat Jun 15, 2024 11:12 am    Post subject: Reply with quote

SarahS93,

Quote:
is "Chunk Size : 64K" OK?

The Chunk Size is the amount of data written to one device before the next device is used.

In your case, that's 64k each to dev/loop1 /dev/loop2 /dev/loop3 before it repeats ... dev/loop1 /dev/loop2 /dev/loop3 ...

64k is a good start. It wants to be an integer multiple of both the filesystem block size and the underlying block device black size.
Writing partial blocks or partial chunks is slow.

I've not used raid without super blocks. Just read about it.
Maybe you need to pass the metadata version too as that determines where the filesystem starts on the raid set.

-- edit --
man mdadm:
       Build  Build  an array that doesn’t have per-device metadata (superblocks).  For these sorts of arrays, mdadm cannot differ‐
              entiate between initial creation and subsequent assembly of an array.  It also cannot perform any checks that  appro‐
              priate  components have been requested.  Because of this, the Build mode should only be used together with a complete
              understanding of what you are doing.


You always build the array. Assemle checks that the components do form a bona fide array which requires the superblock data.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum