| View previous topic :: View next topic |
| Author |
Message |
nokilli
Apprentice
Joined: 25 Feb 2004
Posts: 218
|
 Posted: Tue Jan 14, 2025 4:45 pm Post subject: |
 |
|
| pingtoo wrote: | | However I don't see the benefit of doing this. |
Why do we use rsync? To download only a few bytes instead of a great many. Mounting distfiles as a remote volume and allowing a local rsync process to access it--as opposed to an rsync server which for a Gentoo repository has to be a load--could conceivably save bandwidth.
Enough bandwidth to justify a big change? I don't have these answers.
I came in the door with a desire to more securely apply system updates though. If this becomes a thing, I'll get to scratch that itch. And I'm glad about that.
_________________
Today is the first day of the rest of your Gentoo installation. |
|
| Back to top |
|
 |
pingtoo
Veteran
Joined: 10 Sep 2021
Posts: 1394
Location: Richmond Hill, Canada
|
| Posted: Tue Jan 14, 2025 5:06 pm Post subject: |
|
|
| nokilli wrote: | | pingtoo wrote: | | However I don't see the benefit of doing this. |
Why do we use rsync? To download only a few bytes instead of a great many. Mounting distfiles as a remote volume and allowing a local rsync process to access it--as opposed to an rsync server which for a Gentoo repository has to be a load--could conceivably save bandwidth.
Enough bandwidth to justify a big change? I don't have these answers.
I came in the door with a desire to more securely apply system updates though. If this becomes a thing, I'll get to scratch that itch. And I'm glad about that. |
I don't know Gentoo Foundation/Gentoo Maintenance Management running cost. It is not my concern in this conversation. I am just interesting in term of new idea and trying to understand it.
so I assume by "desire to more securely apply system updates" you are thinking two risk factors, in transit and at rest.
Are you thinking wget/curl transport may be riskier than NBD? Or you are concern the source "distfiles" at Gentoo (or mirror) may be tempered?
Another question, Rre we settle yet for doing NBD have more benefit than current rsync/wget/curl method?
I am not trying to debate with you, I am trying to understand you point. Because I got a sense that from you initial port, you believe there is/are benefit from using NBD as transport. however through out the conversation I have not yet learn the benefit.
I do use NBD in my daily usage but for different reason. My main nodes are ARM based machine and are SBC boards so limited local storage. And I like to work in docker. So I use NBD to supply storage for docker. my NBD source still SBC boards just that I installed NVME on it so I run them as NAS. And I recently start using my MBP-M2 with attached USB disks, so I plan to also run VM on the MBP with NBD to share its storage. |
|
| Back to top |
|
|
nokilli
Apprentice
Joined: 25 Feb 2004
Posts: 218
|
| Posted: Tue Jan 14, 2025 5:10 pm Post subject: |
|
|
Isn't there an opportunity here?
Gentoo Hardened built a name for itself by dedicating itself to security at the process level.
Why not a new face, dedicating itself to protecting user privacy?
The first step? Can't have system update processes that have simultaneous access to user data and encrypted connections over the Internet.
Solution? Use Gentoo >INSERT NAME HERE<, dedicated to protecting your privacy by ensuring that the update process can never expose your data to the Internet!
How to do that? All data downloaded over the Internet to update your system comes from a read-only volume and features as your client: the Linux Kernel!
Another thing Gentoo can do that maybe the other distros can't? Not sure how this could ever work with apt or dnf.
rpm-ostree doesn't even support using a proxy. Gentoo was doing it in the '90s.
Why focus on privacy? Because I can read the news.
_________________
Today is the first day of the rest of your Gentoo installation. |
|
| Back to top |
|
|
pingtoo
Veteran
Joined: 10 Sep 2021
Posts: 1394
Location: Richmond Hill, Canada
|
| Posted: Tue Jan 14, 2025 5:17 pm Post subject: |
|
|
| nokilli wrote: | Isn't there an opportunity here?
Gentoo Hardened built a name for itself by dedicating itself to security at the process level.
Why not a new face, dedicating itself to protecting user privacy?
The first step? Can't have system update processes that have simultaneous access to user data and encrypted connections over the Internet.
Solution? Use Gentoo >INSERT NAME HERE<, dedicated to protecting your privacy by ensuring that the update process can never expose your data to the Internet!
How to do that? All data downloaded over the Internet to update your system comes from a read-only volume and features as your client: the Linux Kernel!
Another thing Gentoo can do that maybe the other distros can't? Not sure how this could ever work with apt or dnf.
rpm-ostree doesn't even support using a proxy. Gentoo was doing it in the '90s.
Why focus on privacy? Because I can read the news. |
This conversation become very strange now. I don't see how a transport mechanism somehow leak privacy information, Do you mind to elaborate more? |
|
| Back to top |
|
|
nokilli
Apprentice
Joined: 25 Feb 2004
Posts: 218
|
| Posted: Tue Jan 14, 2025 6:00 pm Post subject: |
|
|
| pingtoo wrote: | | Are you thinking wget/curl transport may be riskier than NBD? Or you are concern the source "distfiles" at Gentoo (or mirror) may be tempered? |
Yes, any kind of transport is riskier than NBD I feel, for this concern. NBD uses the Linux kernel as it's client. It's communicating with the server that is exposing a read-only volume. The opportunity for mischief here appears to be next to nil.
As opposed to letting an installer have simulaneous access to my home directory and an encrypted Internet connection, which is why I fled my last distro.
Recognize that it is not a question of trusting Gentoo developers as much as it is trusting that the process is foolproof and beyond exploitation by bad actors.
| pingtoo wrote: | | Another question, Rre we settle yet for doing NBD have more benefit than current rsync/wget/curl method? |
Do you mean is it more efficient? I'd expect that it would be less efficient in the single file case, but because it enables rsync in an unusual way, much more efficent in the many file case.
| pingtoo wrote: | | I do use NBD in my daily usage but for different reason. My main nodes are ARM based machine and are SBC boards so limited local storage. And I like to work in docker. So I use NBD to supply storage for docker. my NBD source still SBC boards just that I installed NVME on it so I run them as NAS. And I recently start using my MBP-M2 with attached USB disks, so I plan to also run VM on the MBP with NBD to share its storage. |
In the home network environment, NBDs are marvelous, especially because you can so easily combine them with the other dm devices. I am stubbornly resisting moving over to file systems like btrfs, xfs, zfs, which otherwise have some wonderful capabilities, but which I feel don't work as well with the block device model.
It's a good place to point out again that what I'm describing is a read-only network block device, and which appears to present a much simpler use case than writable devices.
_________________
Today is the first day of the rest of your Gentoo installation. |
|
| Back to top |
|
|
pingtoo
Veteran
Joined: 10 Sep 2021
Posts: 1394
Location: Richmond Hill, Canada
|
| Posted: Tue Jan 14, 2025 6:23 pm Post subject: |
|
|
| nokilli wrote: | | pingtoo wrote: | | Are you thinking wget/curl transport may be riskier than NBD? Or you are concern the source "distfiles" at Gentoo (or mirror) may be tempered? |
Yes, any kind of transport is riskier than NBD I feel, for this concern. NBD uses the Linux kernel as it's client. It's communicating with the server that is exposing a read-only volume. The opportunity for mischief here appears to be next to nil.
As opposed to letting an installer have simulaneous access to my home directory and an encrypted Internet connection, which is why I fled my last distro.
Recognize that it is not a question of trusting Gentoo developers as much as it is trusting that the process is foolproof and beyond exploitation by bad actors.
| pingtoo wrote: | | Another question, Rre we settle yet for doing NBD have more benefit than current rsync/wget/curl method? |
Do you mean is it more efficient? I'd expect that it would be less efficient in the single file case, but because it enables rsync in an unusual way, much more efficent in the many file case.
| pingtoo wrote: | | I do use NBD in my daily usage but for different reason. My main nodes are ARM based machine and are SBC boards so limited local storage. And I like to work in docker. So I use NBD to supply storage for docker. my NBD source still SBC boards just that I installed NVME on it so I run them as NAS. And I recently start using my MBP-M2 with attached USB disks, so I plan to also run VM on the MBP with NBD to share its storage. |
In the home network environment, NBDs are marvelous, especially because you can so easily combine them with the other dm devices. I am stubbornly resisting moving over to file systems like btrfs, xfs, zfs, which otherwise have some wonderful capabilities, but which I feel don't work as well with the block device model.
It's a good place to point out again that what I'm describing is a read-only network block device, and which appears to present a much simpler use case than writable devices. |
Thank you for the explaination.
Understand your privacy concern now.
Please allow me to point out as current Gentoo Portage implementation this is unavoidable because running emerge require root. so the concern of preventing unwanted access to private data installed on Gentoo node while installation going is not possible if emerge is the tool of choose. Allowing unknown account to execute emerge with root privilege is a system administration issue. Using NBD transport or letting emerge call wget/curl will not make any different.
I am try to get a sense from you about the using NBD transport for benefit beside of privacy. My point been using NBD have no performance benefit nor cost benefit so I don't recommend enable NBD for distfiles.
BTW, using wget/curl is one way transport therefor it is also read-only. |
|
| Back to top |
|
|
nokilli
Apprentice
Joined: 25 Feb 2004
Posts: 218
|
| Posted: Wed Jan 15, 2025 1:48 am Post subject: |
|
|
What trade-offs are you willing to make to secure your system?
Everybody's different. Different skill sets, different tolerances.
Honestly, what I want is to only actively administer one machine. And I've decided that's going to be the Gentoo on my home server.
I want the system on my workstation to be exactly the opposite. I'm happy to pull in packages as they're needed but if I can also just freeze the installation and never think about it again and just stay in sway/foot/emacs and sideload flatpaks to take the load for whatever other apps I want to install, that would be amazing.
What protects the workstation if I'm not religiously updating it? It can only connect to the home server over a wifi network dedicated to that purpose on a machine where net.ipv4.ip_forward=0. And then the firewall can further slam the door shut.
So I can access nfs and the new thing is running Podman containers for your network-facing stuff and accessing the admin interfaces via web and this lets me do that too.
If there's a risk that I can easily mitigate using my skill set, then I'm going to want to do that. My skill set isn't amazing. And a problem with Gentoo is that, this quickly becomes obvious.
With this one simple trick, I can update my system and be sure no private data is being exposed in the process. I can already do this on Gentoo right now. Having a nbd-based distfile sitting underneath my local distfiles directory would simply streamline the process.
And then I have another machine I can use to do normal Internet stuff and that I really don't care about, or think about.
I am at peak Gentoo right now. The server was a painless install. I went OpenRC because the other pain point was configuring hostapd to let me have the private network on Ubuntu and it appears to be a use case that the NetworkManager guys aren't supporting. The workstation was a painless install, and I went systemd with that because of the integration opportunities that seem to be developing between sway and flatpak.
I changed a USE flag in my make.conf and did the emerge -uDNa @world and got a list of ten packages and it was a five-minute build. I remember when doing would result in hundreds of packages and the build took the day. I don't know whether it's because machines are faster today or the switch to a pure wayland install but it just feels like a different distro. I got what I wanted on the first try.
What really set me off on this is Fedora. I was using their Sway Atomic Spin on the workstation immediately prior to this new Gentoo install. They use rpm-ostree. This is around eight years old I believe. Not only does rpm-ostree not support proxied access to their servers, but it appears that when the community cobbled together a workaround, Fedora then broke it with a new update. I saw that and it was like, check please.
Ubuntu just casually gives us snapd, which takes simultaneous root access/encrypted network access to all new heights with it's attitude of "we'll read-and-write to your home directory anytime we like and no, you don't even need to know about it." Well, wow.
That said, apt like emerge has been supporting proxied access for awhile too.
Open source is a community of good people who can easily be infiltrated by bad people. A Linux user can't be too careful these days.
_________________
Today is the first day of the rest of your Gentoo installation. |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20575
|
| Posted: Wed Jan 15, 2025 3:17 am Post subject: |
|
|
| nokilli wrote: | | What trade-offs are you willing to make to secure your system? |
If I were a mirror administrator I'd sacrifice untested solutions in favor of those with known risks that have been exposed to scrutiny over a long period of time.
If I were concerned about Gentoo stealing my private data, I'd stop using it.
_________________
Quis separabit? Quo animo? |
|
| Back to top |
|
|
nokilli
Apprentice
Joined: 25 Feb 2004
Posts: 218
|
| Posted: Wed Jan 15, 2025 3:58 am Post subject: |
|
|
| pjp wrote: | | nokilli wrote: | | What trade-offs are you willing to make to secure your system? |
If I were a mirror administrator I'd sacrifice untested solutions in favor of those with known risks that have been exposed to scrutiny over a long period of time.
If I were concerned about Gentoo stealing my private data, I'd stop using it. |
Are you the one paying for the mirror?
My concern is with a process, not Gentoo specifically. If I were concerned about Gentoo stealing my private data, I wouldn't have installed it on both workstation and server.
I have a process which lets me mitigate that threat that is both simple and effective, and Gentoo lets me achieve that, and to my knowledge, in a way that it alone is capable of. I don't think the rsync over nbd approach makes as much sense as for a binary distribution, but I could be wrong.
_________________
Today is the first day of the rest of your Gentoo installation. |
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20575
|
| Posted: Wed Jan 15, 2025 4:48 am Post subject: |
|
|
| nokilli wrote: | | Are you the one paying for the mirror? |
That's an unclear question. If I'm signing the checks for the hardware, utilities, physical space, and labor, then I can assure you I'm not going to care about technical solution A vs B. What I'm going to care about are costs and risks. If I care enough about the mirror to donate any of that, "paying for the mirror" is unlikely to be a cost with which I am concerned. And I will be interested in who made a poor decision to use a solution that unnecessarily caused a security problem. Those are not inexpensive. Other US organizations don't "pay" for bandwidth (.gov, .edu, ?). If you meant something else, please clarify.
| nokilli wrote: | | My concern is with a process, not Gentoo specifically. If I were concerned about Gentoo stealing my private data, I wouldn't have installed it on both workstation and server. |
I did see that you mentioned "package manager" generically, but then it seemed like you were focusing on Gentoo.
The process is interesting. Potentially even the general idea as a solution. I'd be curious to see test results from real world usage. I've considered trying nbd locally. But other comments in this thread make it sound less promising than it initially seemed. I also prefer less complexity, especially with PC-grade hardware.
| nokilli wrote: | | I have a process which lets me mitigate that threat that is both simple and effective, and Gentoo lets me achieve that, and to my knowledge, in a way that it alone is capable of. I don't think the rsync over nbd approach makes as much sense as for a binary distribution, but I could be wrong. |
Maybe I'm misunderstanding. I thought you wanted Gentoo (and/or it's mirror partners) to provide these nbd devices. Otherwise I'm not sure why you couldn't use an nbd device with any other OS / distro if they offered it.
I guess I don't understand how having "private" access to distfiles differs from private access binaries provided by a distro.
_________________
Quis separabit? Quo animo? |
|
| Back to top |
|
|
nokilli
Apprentice
Joined: 25 Feb 2004
Posts: 218
|
| Posted: Wed Jan 15, 2025 5:32 am Post subject: |
|
|
There are two use cases and I am absolutely guilty of confusing the two, so let me try again.
Securely updating my system by using a read-only nbd server is something that can work on any distro, assuming they set it up that way.
Saving bandwidth by using a read-only nbd server is entirely different. We can do it because we have source tarballs and they are very efficiently updated from one version to the next by rsync over nbd.
So the way I see it, Gentoo is eligible for a win-win using nbd as an installation medium whereas the other distros only get a win.
I think you guys are misreading the general fatigue the computer-using community is feeling over the constant stream of exploits, esp. the ones they paid for with their own tax dollars. Because you have the skills, you don't perceive the threat in the same way? You can flip this bit or tweek a conf file and know you're secure. The rest of us maybe aren't so lucky.
Securing something by hitting it with a rock will appeal to many people.
| pjp wrote: | | I'd be curious to see test results from real world usage. |
As one of the few users who has openly admitted to syncing the distfile directory to my local drive in the past (this was many years ago) I am willing to step up and attempt redemption. Can you help me get in touch with one of the providers you are using? AWS would let you do large transfers for free if it was within house; does your provider offer the same policy?
I'm working on a quality nbd server now in asyncio. The plan would be to copy as many (and older too) stage3 files as possible, then maintain a current copy of the distfiles directory with the caveat mentioned before that all tarballs be accompanied by their expanded equivalents, so people could explore the different emerge -u intensities. This could be run as a service that is exposed only to the developers, along with the patch to ebuild fetch replacing wget(?) with rsync. For your consideration.
_________________
Today is the first day of the rest of your Gentoo installation. |
|
| Back to top |
|
|
nokilli
Apprentice
Joined: 25 Feb 2004
Posts: 218
|
| Posted: Wed Jan 15, 2025 5:55 am Post subject: |
|
|
I'm calling it glut-master and it accepts a glut parameter which I can preview to elicit feedback.
| Code: | | gentoo $ glut-master --host monolith.gentoo.org --port 2001 --path /omg/its/so/big --glut "((1000,100),(4000,50),(7000,20))" |
So you pass a tuple of tuples with two values each, a block count and a task count. We then create queues for each tuple. Whenever we're going to service a block request from the kernel, we get the number of blocks served so far to that client, then scan through the tuples and select the first value that is greater to that number, and put the request in the associated queue. The second number provides the number of clients we process for that queue in that iteration. There is an implicit final tuple of (Math.MAXINT, 1), which then processes the dd'rs one at a time per iteration.
So, in the example given, we are processing 171 tasks per iteration, giving high priority to people who are using the service responsibly (requested <1000 blocks thus far).
I feel this is better than using the heap as I intimated before. Maybe get to target a micro instance.
_________________
Today is the first day of the rest of your Gentoo installation. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Gentoo Chat
