View previous topic :: View next topic |
Author |
Message |
Genone Retired Dev
Joined: 14 Mar 2003 Posts: 9538 Location: beyond the rim
|
Posted: Mon May 12, 2003 5:34 pm Post subject: |
|
|
It's easier than you might think. Setting up the server is creating /etc/exports ("man 5 exports") and "/etc/init.d/nfs start" in most cases. When I first created this setup I thought it would be difficult but it's a lot easier than setting up a samba or apache server. |
|
Back to top |
|
|
dufeu l33t
Joined: 30 Aug 2002 Posts: 924 Location: US-FL-EST
|
Posted: Mon May 12, 2003 6:52 pm Post subject: |
|
|
Genone wrote: | It's easier than you might think. Setting up the server is creating /etc/exports ("man 5 exports") and "/etc/init.d/nfs start" in most cases. When I first created this setup I thought it would be difficult but it's a lot easier than setting up a samba or apache server. |
Well, there are a few details. :-/
"man 5 exports" doesn't return anything if you don't have nfs-utils emerged.
After emerging nfs-utils, /etc/init.d/ has 'nfs' and 'nfsmount' added to it. I'm not yet clear as to why this is so since 'netmount' seems to mount NFS shares fine. I do see that 'nfs' is required for starting the NFS server though.
I think I've got a handle on what I need to do when I get home.
On the client's side, in addition to what I already posted, I need to update 'hosts.deny' and 'hosts.allow' for 'portmap'.
hosts.deny -- add "portmap:ALL"
hosts.allow -- add "portmap: 192.168.0.0/255.255.255.0
For the server:
1) I need to update hosts.allow & hosts.deny as above and also for 'lockd', 'mountd', 'statd' and 'rquotad'.
2) rc-update add nfs default
3) create /etc/exports with the lines:
Code: |
/usr/portage 192.168.0.0/255.255.255.0(ro)
/usr/portage/distfiles 192.168.0.0/255.255.255.0(rw)
/usr/portage/packages/$HOME 192.168.0.0/255.255.255.0(rw)
|
I think that is it. I'll find out for sure when I get home tonight.
Anyone have an opinion if I set this up on my firewall? I have iptables set up _very_ restrictively and basically only permit internet access to my ssh login (root's dissallowed). I realize that this is not pure secure practice, but I'm fairly comfortable that my firewall is pretty secure. It's really the only computer I'm will to leave running 24 hours. It has a 45watt power supply. _________________ People whom think M$ is mediocre, don't know the half of it. |
|
Back to top |
|
|
Genone Retired Dev
Joined: 14 Mar 2003 Posts: 9538 Location: beyond the rim
|
Posted: Mon May 12, 2003 7:39 pm Post subject: |
|
|
dufeu wrote: | On the client's side, in addition to what I already posted, I need to update 'hosts.deny' and 'hosts.allow' for 'portmap'.
hosts.deny -- add "portmap:ALL"
hosts.allow -- add "portmap: 192.168.0.0/255.255.255.0 |
wasn't necessary for me, but your setup may differ.
dufeu wrote: | 3) create /etc/exports with the lines:
Code: | /usr/portage 192.168.0.0/255.255.255.0(ro)
/usr/portage/distfiles 192.168.0.0/255.255.255.0(rw)
/usr/portage/packages/$HOME 192.168.0.0/255.255.255.0(rw) |
|
As was said on the mailinglist this won't work, you can't export subdirectories if their parents are already exported. You either have to export /usr/portage RW or move distfiles and packages out of /usr/portage.
dufeu wrote: | It's really the only computer I'm will to leave running 24 hours. It has a 45watt power supply. |
Nice |
|
Back to top |
|
|
dufeu l33t
Joined: 30 Aug 2002 Posts: 924 Location: US-FL-EST
|
Posted: Mon May 12, 2003 9:38 pm Post subject: |
|
|
Genone wrote: | dufeu wrote: | 3) create /etc/exports with the lines:
Code: | c
/usr/portage/distfiles 192.168.0.0/255.255.255.0(rw)
/usr/portage/packages/$HOME 192.168.0.0/255.255.255.0(rw) |
|
As was said on the mailinglist this won't work, you can't export subdirectories if their parents are already exported. You either have to export /usr/portage RW or move distfiles and packages out of /usr/portage.
|
You are absolutely correct. I read that. I noted it. And I still went ahead and wrote it down wrong. /etc/exports should be:
Code: |
/usr/portage 192.168.0.0/255.255.255.0(rw)
|
Thanx for catching this. _________________ People whom think M$ is mediocre, don't know the half of it. |
|
Back to top |
|
|
TRauMa n00b
Joined: 26 Nov 2002 Posts: 43 Location: Germany
|
Posted: Mon May 12, 2003 11:59 pm Post subject: Finding out if a package-update is there |
|
|
Uhm, I hope I just missed it, but there were 4 "I-want-bleding-edge-and-rsync-ten-times-a-day-if-I-wait-for-a-new-version-of-a-package"-guys and no one mentions Gentoo Linux Stable? Check for new versions there, every 60 mins if you like. |
|
Back to top |
|
|
Davin n00b
Joined: 13 May 2003 Posts: 45 Location: United States
|
Posted: Tue May 13, 2003 2:21 am Post subject: |
|
|
I read this thread in its entirety and the discussion of effective NFS usage in a Gentoo environment jogged my memory a little. This is vaguely off-topic, but it does concern minimalizing Gentoo resources on a network of Gentoo machines.
https://forums.gentoo.org/viewtopic.php?t=53595 |
|
Back to top |
|
|
hmiller68 Tux's lil' helper
Joined: 21 Dec 2002 Posts: 77
|
Posted: Thu May 15, 2003 5:56 am Post subject: Question regarding the use of mirrorselect. |
|
|
Would I be in the same boat as hardcoding a particular server in config by using the mirrorselect tool in portage ? I mean in actuallity it is infact doing just that isn't ,hardcoding the server in /etc/make.conf to start with the first server selected every time? |
|
Back to top |
|
|
really Guru
Joined: 27 Aug 2002 Posts: 430 Location: nowhere
|
Posted: Thu May 15, 2003 11:29 am Post subject: |
|
|
why is it so important for some people to rsync every hour or so?
i do it when i want to install a new program or update one i see has an update, or if the GLSA shows i really need to update a package..
i rsync like, this week i was i think 3 times (needed to update wine and installed openoffice (gooood i hate it) and checked if something needs updating with emerge -up system..)
last week i didnt rsync at al.. and i think i rsynced once the week before that to see if emerge -up system and world shows something new
really unnecesery to have a cronjob rsyncing..
its not like yuu cant live for some days without having the latest of x program.
hell i right now have not yet got gcc3.3. ... gcc3.2 is just fine.. it does it job
but i will update in a week or so..
proposal (dont have the time to read the whole threa, maybe someone already proposed it, but):
all the rsyncers could make a site which someone updates every hour with the information on what if anything has been updated/changed. the normal rsyncers could then make a script to look for information on that site. and the site itself gets its information from developers or just one sync..
so all the ppl who feel the need to rsync every x minutes could just look for specific information on a site, not have to sync with the whole tree.
and if they then want to update a program they have, they could just download the ebuild for it from say bugs.gentoo.org if it is there and install it, no need to download the whole tree. _________________ NoManNoProblem
Get lost before you get shot. |
|
Back to top |
|
|
oxo42 n00b
Joined: 18 May 2003 Posts: 1
|
Posted: Mon May 19, 2003 12:08 am Post subject: |
|
|
henke wrote: | Right now I'm wating for xfree-4.3.0-r3 (Castle Rock driver baby ) How do I find out if this package has been released without rsyncing? |
Okay this is just a temporary measure, but i wrote a php script to browse an rsync server. http://rucus.ru.ac.za/~oxo/rsync/rsync-browser.tar.bz2
Please keep in mind that this was a quick hack for my own benefit and is far from perfect. I also use modlayout so the html will look imcomplete |
|
Back to top |
|
|
CRC Tux's lil' helper
Joined: 30 Mar 2003 Posts: 90 Location: Dallas, TX, USA
|
Posted: Mon May 24, 2004 11:57 pm Post subject: Re: Another way to handle sync |
|
|
Manuzhai wrote: |
Is that about right? I would propose an extra layer, then:
1. You get a list (this could be implemented in XML very well) of all the ebuilds currently in Portage. This would include versions as well as the stability for different architectures and dependencies. If the file gets to big, it could be split up into several files for every category. This list will be your primary tree.
2. Ebuilds are only saved on your computer for packages that you actually emerged. This means rsync does not have to be used at all for ebuilds. Some info can be cut from the ebuild because it is already in the XML file.
3. The actual files are retrieved from the internet as stated in the ebuild, and the package is built and compiled.
The advantages:
- Only use rsync (possibly) for the XML files
- Portage tree is much smaller (XML also compresses very well)
- Easy to get info from the tree for emerge as well as external tools
- Less bandwidth used as ebuilds are only transferred as necessary
|
I think there is some good information here. I think what we need to look at is determining if anything has changed without going through a a full rsync in the entire tree, and limiting the "pull" of ebuilds to just what we are interested in.
Basically, I think you are saying that instead of constant rsync, we do it "on-demand", with a central smaller database file to coordinate it. So, if you emerge say "gaim", we grab the latest unmasked ebuild (which will depend on which fork you use, ie: if you are using ACCEPT_KEYWORDS, plain/regular Gentoo, a break-my-gentoo system, or some "stable-server" off-shoot). Then, you grab that ebuild, then from that information figure out what dependant ebuilds need to be grabbed.
In addition, since many ebuilds are just version bumps, if the system could see that you already have an ebuild for version 0.76 on your system, it could transmit just the diffs between 0.77 and 0.76. Just rename the old ebuild to the new one and rsync - rsync can send diffs, even compressed diffs, automatically.
Also, this means you don't transfer ebuilds for packages you don't care about. An emerge sync would only update the master database of which ebuilds are "current" that simply transmits the diff's from the current version - and a simple timestamp check on that one master would pretty much end needless rsync's quickly! Also, this master could perhaps be indexed like a database to speed up searches. We'd still have people hitting this file often, but see below on why this won't be bad (re: Bit-Torrent to download).
Finally, at this point, your /usr/portage is basically a cache containing only the ebuilds that you've installed - so they can be diff'd with the latest version as an update, and that master database of versions, plus whatever distfiles are still around. If you could share that Bit-Torrent style, that would be really ideal. Say, maybe your system has a minimum time it has to stay on the network for every sync, and stays on the duration of every emerge - contributing to the network during the compilation and installation process, and maybe a few minutes longer. The whole /usr/portage could be shared, distfiles and all. This would use torrent files (or embed the torrent information in the master database) in place of md5 sums. Wanna rsync constantly? Fine! Your donating your resources every time you do, and needless rsyncs just help everyone else out more than they hurt.
If you could do source diffs between versions that would help too, but I think this would be an addition to the ebuild, some option in the file that says that if a particular old distfile exists, then grab the patch instead of a new distfile. Either way, you'd be fetching new distfiles and ebuilds from all the people who currrently have them anyway, and not just the mirrors, so rushes when new versions come out becomes a good thing! More people to share with - exactly the point of bit-torrent.
This would involve some pretty invasive changes to portage, but I think it would be well worth it. Anyone want to try making an "rsync-over-bittorrent" ? How about a central datase of packages that lists current versions of packages and .torrent information for each version (so that you don't need .torrent files maybe, just query the info)? I bet if people did it, or did some proof-of-concept work on it, Gentoo would use it.
All Feedback appreciated! I'm not construction the solution so much as trying to get people thinking about what the requirements are for doing this "the right way". _________________ Unix/Linux Consulting & Hosting
We Support Gentoo!
http://CoolRunningConcepts.com
Freenode: Taro! |
|
Back to top |
|
|
CRC Tux's lil' helper
Joined: 30 Mar 2003 Posts: 90 Location: Dallas, TX, USA
|
Posted: Tue May 25, 2004 1:18 am Post subject: |
|
|
Just to add - this isn't totally bizarre. Imagine if all the official mirrors supported bittorrent, plus the users ... you'd always have plenty of uploaders.
Look at this post for more info:
http://lists.debian.org/debian-devel/2003/07/msg00473.html
Another possibility would be to look into using PDTP instead of Bit-Torrrent, so you don't have .torrent files and such. This looks like a closer match for the requirement, but isn't nearly ready for general use.
http://pdtp.org/static/index.html _________________ Unix/Linux Consulting & Hosting
We Support Gentoo!
http://CoolRunningConcepts.com
Freenode: Taro! |
|
Back to top |
|
|
salam Apprentice
Joined: 29 Sep 2005 Posts: 226
|
Posted: Wed Dec 21, 2005 1:44 pm Post subject: a question about rsync etiquette |
|
|
this rule is quite easy to understand - the purpose is not to allow excessive load on the servers. but i have a question:
it is clear, that the IP of the abusive user will be dropped in iptables. but what if there are many users? (NAT-ed networks) let's say there are 30 gentoo users and each one syncs 1x/day. none of them broke the etiquette, but the sync mirror sees 30 syncs/day because of NAT. i think it cannot be solved from the server side very well, but what do you think about it? is it a common problem(like on IRC, one abuser gets many other banned)? |
|
Back to top |
|
|
certocivitas Apprentice
Joined: 29 Dec 2004 Posts: 194
|
Posted: Wed Dec 21, 2005 1:50 pm Post subject: |
|
|
Most of the time wouldn't the people sharing an IP address be in close proximity to each other. The correct etiquette for Gentoo users in that situation should be to setup a rsync mirror (and http-replicator for that matter). |
|
Back to top |
|
|
aidanjt Veteran
Joined: 20 Feb 2005 Posts: 1118 Location: Rep. of Ireland
|
Posted: Wed Dec 21, 2005 2:04 pm Post subject: |
|
|
30 Gentoo users sitting next to each other and none of them had the bright idea of configuring an rsync mirror?.. thats disapointing. |
|
Back to top |
|
|
tomk Bodhisattva
Joined: 23 Sep 2003 Posts: 7221 Location: Sat in front of my computer
|
|
Back to top |
|
|
zialien n00b
Joined: 27 Jun 2006 Posts: 31 Location: Melbourne, Australia
|
Posted: Sat Jun 02, 2007 8:23 am Post subject: just a quick question |
|
|
I will be converting my XP box desktop to gentoo soon (my laptop is gentoo and has been for well over a year). I am considering making the desktop a local rsync mirror. My question is if i use the instruction on http://gentoo-wiki.com/HOWTO_Local_Rsync_Mirror i get the feeling that the files i have in the servers distfiles will not be used, is this assumption correct.
Basically what i am asking is if i have my laptop updating packages will is actually attempt to get the source files from my local server or not. I know that i will get the portage from my mirror but i am not sure about the distfiles. The main reason i ask is because of this line:
exclude = distfiles/ packages/
it makes me think it will not use my mirror for the distfiles which defeats the main purpose of using the local mirror.
and the last question, assuming the above happens (that is my laptop will attempt to get the files from my servers distfiles) if the file does not exist will it download it off a propper mirror and place a copy on my server?
Thanks for any help |
|
Back to top |
|
|
mark_alec Bodhisattva
Joined: 11 Sep 2004 Posts: 6066 Location: Melbourne, Australia
|
|
Back to top |
|
|
zialien n00b
Joined: 27 Jun 2006 Posts: 31 Location: Melbourne, Australia
|
Posted: Sat Jun 02, 2007 9:18 am Post subject: |
|
|
thanks for that i saw http-replicator eairler and thought it may be the answer but didnt bother looking anyfurther, my bad. thanks again. |
|
Back to top |
|
|
|