how to split a file in not equally large parts?

[SarahS93]

how to split a 75GB file in to 7 not equally parts?
....:
75gb_file.1 = 11GB
75gb_file.2 = 3GB
75gb_file.3 = 26GB
75gb_file.4 = 17GB
75gb_file.5 = 6GB
75gb_file.6 = 8GB
75gb_file.7 = 4GB

with "split" all files have the same size, that does not work.

[Hu]

Would this work?

[SarahS93]

it works, thats great, thanks!

[toralf]

[SarahS93]

with
shuf -i 1-100 -n 4
i become 4 random numbers

but whats the way to split a number randomly into multiple numbers give the number and n groups?
for instance, if the number is 100 and the number of groups is 4 it should give any random list of 4 numbers that add upto 100:4
input number = 100
number of groups = 4
...
25 33 12 30
11 19 47 23
8 17 40 35

(i am looking for an other way for the "11 3 26 17 6 8 4" where i must not calculate myself)

i found
awk 'BEGIN{srand()} {s=int(rand()*7); print > (FILENAME"."s)}' file75G
but the spitet files togehter are 1 byte bigger the the not splittet file, dont know why
but the splittet files are all round about 10,7G, thats not what i am looking for.

[eccerr0r]

That's a NP problem... Hard to create a solution (though there are a lot), though this is checkable (almost like the knapsack problem).

Lots of heuristics however... easiest one is to take your max, pick a random number from 1..max, subtract it from the original, then rinse and repeat on the remaining. The latter pieces will tend to be unfairly small however -- hence this is a heuristic.

Another heuristic is to take max/n pieces at first and then randomly shuffle a bit to each bin until one is happy with the results... ugly and could take a lot of iterations.

I hope this is just a theoretical problem, else could you disclose why one would need this??!
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

[szatox]

Sure, let's play

[SarahS93]

eccerr0r
i am looking for a way to split a file into equally parts.
The way from Hu in Post #2 is great, but i must calculate the partsize myself.

szatox
thanks for your lines, it works!
sometimes i become a number 0
is there a way to exclude the number 0 ?

bigger blocks is not what i am looking for.
how do i modify $k ?
can you give me please an example

i am looking for a way that 100:4 =
27 25 33 15 is good
25 23 35 17 is good too
5 7 6 82 is not good
10 10 10 60 is not good
...

[szatox]

[eccerr0r]

What I still don't get is why is

25 25 25 25

not the best solution if you want "equally" parts, this is fast, simple, and deterministic?
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

[pjp]

The title says "not equally large parts".
_________________
Quis separabit? Quo animo?

[Hu]

Yes, but a later post in the thread specifies equally large parts. If we want unequally large parts, my preference would be to divide the file into chunks of sizes 1, 1, 1, ... and N-(num chunks). Using one byte for most chunks makes the algorithm easy, and guarantees the parts are not all of equal size. If instead we need to have every chunk use a unique size, then make the first chunk 1 byte, the second 2 bytes, and so on, until the last chunk is the remainder of the file. Since there is no stated purpose for this exercise, it is difficult to determine whether this solution is acceptable.

OP: what are you trying to achieve with this split?

[eccerr0r]

Yes, I was wondering because this seems to have no practical purpose other than as a thought experiment... if there is one I'd like to know.
Does it even have to be random... 24 26 23 27 works (and this is a precooked heuristic - start with equal numbers and every other chunk you add one subtract one, add two, subtract two...keep going until you run out, should work fine n chunks less than total divided by n... no randomness at all and every file is different size and all are about the size of the average... up until there are a huge number of chunks for a small file....

... why....
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

[pjp]

[eccerr0r]

random small pieces are just as difficult/annoying to properly reassemble as larger similar sized pieces, so hence a good reason for these pieces still remains...

The file is still plaintext anyway so important data can still be gleaned from it. If you encrypt it, well, the encryption adds entropy anyway and the piece scrambling is just another step to reconstruct where a larger key size would have been just as good with fewer steps to reconstruct.

Hence this still just looks like an academic exercise.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

[pjp]

Well, if you somehow guess the first separation is at 10GB and are able to rejoin each with the same offset, that's easier than figuring different offsets at each step. I'm not saying it's a good idea or at all secure, but past posts lead me in that direction. Maybe it's academic, maybe it isn't.
_________________
Quis separabit? Quo animo?

[SarahS93]

the soloutions here are very acceptable, thanks!

it must not be random, 24, 26, 23 and 27 is good too.
100:4 is only an example

there is a practical purpose for me!

i have encrypted container files from 22gb upto 77gb
they i will split in 5 or upto 9 parts with unequel size.
after that, i do split each of the parts again into equel size parts and put them into a wrong sequence into a new file.

[szatox]

[pjp]

[szatox]

[Hu]

Until the most recent post from OP, it looked like OP was trying to use this split+scramble in lieu of, rather than in addition to, proper cryptography. For the very first post, I was willing to consider that this was just an attempt to deal with weird size caps on an upload service, and splitting the file into pieces allowed them to be uploaded to a service that would not take the full file in a single step.

History shows many examples of people inventing self-perceived "great" designs to use to obfuscate data, calling it encryption, and then having it fail horribly when serious attackers (whether cryptography researchers who publish their attack, or malicious attackers who use the attack for direct gain) examine the algorithm and find that an output "ciphertext" can be converted back to the plaintext form in very reasonable time. Therefore, I (and, I think several other posters) have a strong bias toward assuming that anyone asking for this sort of thing is doing it as part of a homemade encryption scheme that will likely fail the first time a serious attacker tries to analyze it. The responsible thing to do in that case is to warn the requester that this is not a secure way to conceal data, and that using a well verified cryptographic protocol will be both easier and safer.

Yes, physical assault can break even the best cryptographic schemes, but one of the goals of good cryptography is that an attacker's only choices are (1) a brute force search of a large key space, which ideally should take so long on average that the attacker chooses not to bother or (2) attack the endpoint via espionage / assault.

If this is being used in addition to a known good cryptography scheme, then I think it is still pointless because it adds complexity with little value. However, if OP wants to overcomplicate a secure system, and does so in a way that does not reduce its security, that is OP's choice.

[SarahS93]

if i read it right, short form, scramble and split makes it not bad, but not better for more securtiy?

create an encrypted file, and into it an encrypted file too, and so on and so on ... how deep, 5 times? diferent chippers, no headers at the encrypted files (storage for all 5 separated) - is this more sercure than 1 encrypted file with scamble and split?

encrypted file (header detached) - is it possible to identify this file as an encrypted file and as an encrypted luks file?

[szatox]

SarahS93, explanation in a nutshell here: https://xkcd.com/538/

The wrench attack does have some disadvantages, like it's not exactly stealthy, but brute force is best applied to the weakest link, and it ain't AES right now.
Encrypting stuff with different algorithms and different (unrelated) passwords could help with not-yet-discovered vulnerabilities which could potentially make decryption trivial in the future, but it won't protect you from the wrench in hands of a determined attacker, while still piling up inconveniences for the users. And sufficiently inconvenienced users write passwords down on sticky notes next to their monitors.

[SarahS93]

with

[szatox]

Maybe put the whole input through a single pv instead of putting each chunk through a different one?

Instead of
for ... pv ... done < input
use this
pv input | for ... done
_________________
Make Computing Fun Again