| View previous topic :: View next topic |
| Author |
Message |
vadius
n00b
Joined: 08 Oct 2002
Posts: 12
Location: Tulsa, OK
|
 Posted: Fri Feb 14, 2003 7:29 am Post subject: Help yourself and gentoo out - minimizing bandwidth usage |
 |
|
Here is a nifty trick that I use. I run several Gentoo servers on my college campus, and many of my friends use Gentoo as well. We found that it almost seems a waste to have to rsync and waste someone else's bandwidth for redundant information, or to flood ibiblio with requests for the same files, so we set up our own local rsync + (pseudo) ibiblio site. However, due to university restrictions, we cannot provide our services outside of our firewall  .
But, we have limited our bandwidth burden on the rest of the world, as well as saved ourselves a lot of time (as we all have 100Mbps access to our local net).
You too can help yourself and gentoo!
First, set up an rsync server (detailed here). The only changes I make are that I only rsync every two hours (for minimizing bandwidth), and I am not listed globally (university policy). However, you should always sync at :00 and :30 if you are going to be listed!
So then just add the following line to the end of the /etc/make.conf of anyone you want to let rsync to you (assuming your sever name is server.domain.com):
| Code: | | SYNC="rsync://server.domain.com/gentoo-x86-portage" |
Don't you just hate it when you have two boxes, and they are both downloading mozilla or something huge? Solve this by using a common download directory! It can be a bit hairy though. You could, theoretically, just physically mirror the gentoo files and set up your ibiblio mirror, but this is rather big (many many gigs). The way we do it is through NFS, although Samba could be used if you want to keep all of your files on a Windows box for some reason.
Here are the steps to configure NFS to shorten download times. One of the computers will have to be a designated file server.
- Make sure all of the kernels supports network file systems, including NFS (and go ahead and throw in samba as well). NOTE: The kernel from the Live-CD does not support network file systems. This means, sadly, you cannot take advantage of this when installing a new system.
- Perform an emerge nfs-utils on the server and every client.
- On your server, add the following line to the /etc/exports file:
| Code: | | /usr/portage/distfiles machine1(rw,no_root_squash) machine2(rw,no_root_squash) ..... |
And replace "machine1" and "machine2" and so on with the names of the machines you want to export the files too. NOTE: this allows any and all of these machines to have full access to your /usr/dportage/distfiles folder.
On the server, fire up the NFS server:
| Code: | | /etc/init.d/nfs start |
And probably go ahead and stick in the startup list:
| Code: | | rc-update add nfs default |
On each of the machines you specified above, add the following line to /etc/fstab:
| Code: | | servername:/usr/portage/distfiles /usr/portage/distfiles nfs rsize=8192,wsize=8192,timeo=14,rw,intr |
replacing "servername" with the name of your server.
On each of the client machines, I'd go ahead and move /usr/portage/distfiles somewhere else, in case something goes wrong (just delete it later if everything works):
| Code: | | mv /usr/portage/distfiles /usr/portage/olddistfiles |
Then go ahead and re-create the directory:
| Code: | | mkdir /usr/portage/distfiles |
And finally, mount it:
| Code: | | mount /usr/portage/distfiles |
After doing this you should have a nice, smoother, faster network, and you'll give Gentoo a break  .
Questions, comments, etc should be sent to ch@utulsa.edu, or just reply to this message. |
|
| Back to top |
|
 |
Matje
l33t
Joined: 29 Oct 2002
Posts: 619
Location: Hasselt, Belgium
|
| Posted: Fri Feb 14, 2003 2:13 pm Post subject: |
|
|
I was wondering, what happens when two pc's start the same emerge at the same time. Won't the download get b0rked? I share my distfiles too, and I was just wondering this
_________________
Life is like a box of chocolates... Before you know it, it's empty... |
|
| Back to top |
|
|
vadius
n00b
Joined: 08 Oct 2002
Posts: 12
Location: Tulsa, OK
|
| Posted: Fri Feb 14, 2003 4:01 pm Post subject: |
|
|
| Matje wrote: | | I was wondering, what happens when two pc's start the same emerge at the same time. Won't the download get b0rked? I share my distfiles too, and I was just wondering this |
I've never tested this...
I'd imagine that the second emerge would compute a bad MD5 checksum(since it is not finished downloading), and die, leaving the other intact. I've never actually encountered a bad checksum before, so this may not be the case.
Any one else actually know the answer to this? |
|
| Back to top |
|
|
bumpus
n00b
Joined: 14 Feb 2003
Posts: 64
Location: Cedar Rapids, IA, USA
|
| Posted: Fri Feb 14, 2003 5:26 pm Post subject: Exactly |
|
|
| Quote: | | I'd imagine that the second emerge would compute a bad MD5 checksum(since it is not finished downloading), and die, leaving the other intact. I've never actually encountered a bad checksum before, so this may not be the case. |
This is exactly what happens. I've never actually tried it with more than one machine, but I'll have it happen on occasion when I'm trying to "pipeline" the emerge process. Sometimes if I know I have a lot of downloading to do I'll run emerge --fetchonly -u world and then follow up a few minutes later with emerge -u world. That way the compiler never has to wait for a download to finish.
_________________
-------------
Just because I can. |
|
| Back to top |
|
|
puddpunk
l33t
Joined: 20 Jul 2002
Posts: 681
Location: New Zealand
|
| Posted: Fri Feb 14, 2003 9:54 pm Post subject: |
|
|
| There is work being done for making the Portage build/fetch process paralell. It has been a common request on bugs.gentoo.org! |
|
| Back to top |
|
|
ebrostig
Bodhisattva
Joined: 20 Jul 2002
Posts: 3152
Location: Orlando, Fl
|
| Posted: Fri Feb 14, 2003 11:29 pm Post subject: |
|
|
| Matje wrote: | | I was wondering, what happens when two pc's start the same emerge at the same time. Won't the download get b0rked? I share my distfiles too, and I was just wondering this |
No, the download process is just "Read-Only".
Since there are 2 different PC's doing the emerge, you are OK.
The scenario described above is just the same as (in principle) what you do now when the emerge process donwloads files from Ibiblio or any other server.
Erik
_________________
'Yes, Firefox is indeed greater than women. Can women block pops up for you? No. Can Firefox show you naked women? Yes.' |
|
| Back to top |
|
|
delta407
Bodhisattva
Joined: 23 Apr 2002
Posts: 2876
Location: Chicago, IL
|
| Posted: Sat Feb 15, 2003 2:20 am Post subject: |
|
|
If you use prozilla, the risk of partial files being shared is minimized. Prozilla, being a segmented downloader, doesn't create the original filename until all parts have completed downloading -- instead, it downloads filename.prz[1-4] (or whatever) and only when all data is transferred combines them into filename. I would suggest using prozilla on your mirror boxen.
_________________
I don't believe in witty sigs. |
|
| Back to top |
|
|
kram
n00b
Joined: 29 Nov 2002
Posts: 35
|
| Posted: Sat Feb 15, 2003 2:27 am Post subject: |
|
|
| Quote: | | NOTE: The kernel from the Live-CD does not support network file systems. This means, sadly, you cannot take advantage of this when installing a new system. |
You don't have to use the LiveCD to install Gentoo, I use Knoppix. Much better, IMO, because you can do a lot more with your system while installing Gentoo 
You can use the installation guide on the web using Mozilla, and Knoppix supports a lot more stuff than the LiveCD (including NFS, Samba and my TV card ) |
|
| Back to top |
|
|
scout
Veteran
Joined: 08 Mar 2003
Posts: 1991
Location: France, Paris en Semaine / Metz le W-E
|
| Posted: Sat Mar 08, 2003 10:46 pm Post subject: |
|
|
| In my campus we just share our distfile by ftp and add ourselves in GENTOO_MIRRORS in the make.conf |
|
| Back to top |
|
|
revil
n00b
Joined: 06 Oct 2002
Posts: 68
Location: Japan
|
| Posted: Tue Mar 11, 2003 8:13 am Post subject: |
|
|
| network engineering at my campus has a gentoo rsync mirror.... |
|
| Back to top |
|
|
flickerfly
l33t
Joined: 08 Nov 2002
Posts: 677
Location: Lanham, MD
|
| Posted: Mon Mar 17, 2003 5:51 pm Post subject: For the forums |
|
|
This doesn't apply to the sync discussions, but does hit the topic:
Does the new light theme for the forums help gentoo save bandwidth/process time at all, or is it just a user side change? Could they offer us a less intensive theme? I have phpBB running on a p133, 64m and I setup the default theme to something with almost no pictures. I'd be glad to use this in order to ease server abuse until something better gets worked out.
_________________
An Evil Genious' Guide to Sheeple and How To Avoid Becoming One | 0x4C9EF4A |
|
| Back to top |
|
|
smarm
n00b
Joined: 10 Jan 2003
Posts: 36
|
| Posted: Thu Mar 20, 2003 3:42 am Post subject: |
|
|
I had a very similar idea, while playing tribes of all things. I'd like to expand on the idea if I may.
Why not make something similar to a rsync proxy server, give it a list of your world files, have it download at updates :00 and :30, ie --fetchonly, then have all of your gentoo boxen download the tarballs from the server. Seems it would save you from changing your gentoo boxen and many extraneous downloads. That way you could have a large common world list encompassing all of your boxen behind your proxy and save bandwidth. |
|
| Back to top |
|
|
kamikaz3
Apprentice
Joined: 06 Feb 2003
Posts: 187
|
| Posted: Thu Mar 20, 2003 8:03 pm Post subject: |
|
|
This rules
I have 3 gentoo systems
this saves both bandwidth and diskspace
I skipped the rsync part because that isn't really useful in my case
I don't like the part where rsync is running every 2 hour or ..., this doesn't save bandwidth in my case, it creates extra
So now I have one 1 machine keeping all the install files on a seperate partition (I changed portage to /home/portage and /home is on a seperate partition) |
|
| Back to top |
|
|
pYrania
Retired Dev
Joined: 27 Oct 2002
Posts: 650
Location: Cologne - Germany
|
| Posted: Sat Apr 19, 2003 12:13 am Post subject: |
|
|
| kamikaz3 wrote: | I skipped the rsync part because that isn't really useful in my case
I don't like the part where rsync is running every 2 hour or ..., this doesn't save bandwidth in my case, it creates extra
|
exactly that's what i think.
i mean, how many gentoo boxes do you run that it reduces bandwith when syncing every 2 hrs instead of syncing from every single box on demand?
_________________
Markus Nigbur |
|
| Back to top |
|
|
otaci
n00b
Joined: 23 Apr 2003
Posts: 3
|
| Posted: Wed Apr 23, 2003 8:38 am Post subject: |
|
|
Thanks for that link to an rsync mirror, been meaning to do that.
Regarding sharing distribution files, I just set up a squid proxy server, set the http_proxy and ftp_proxy environment variables on each host, and then got automated sharing, with ageing etc. Also have to make sure all the internal hosts are using the same Gentoo mirror. Works like a charm. |
|
| Back to top |
|
|
Ari Rahikkala
Guru
Joined: 02 Oct 2002
Posts: 370
Location: Finland
|
| Posted: Mon Oct 13, 2003 12:18 pm Post subject: |
|
|
Perhaps you should alias "emerge sync" to "ssh root@server emerge sync" 
_________________
<laurentius> gentoo linux?
<ari> Yesh.
<laurentius> they look horny |
|
| Back to top |
|
|
dan2003
Apprentice
Joined: 11 May 2003
Posts: 258
Location: somewhere near london, UK
|
| Posted: Sun Nov 16, 2003 12:59 am Post subject: |
|
|
quick Q, in the howto it says that syncs should be done every 00 and every 30 past the hour, it doesnt say where to set this. I notice the --timeout=1800 in the example conf file and that that is probably 1800 seconds which happens to be half an hour. Is this related to it? or should a cron job run the thing every half hour? I ask this because i have set up a local on site server to reduce the load on the mirrors on the net as i have several boxes which i sync. I want to have one box sync once per day and have the other boxes sync from it. I have set the sync option in the make.conf of the client boxes to sync from the server, and the server sync= option to the result of the netselect and rsync commands but don't know where to set the update interval.
Cheers |
|
| Back to top |
|
|
Squinky86
Retired Dev
Joined: 25 Mar 2003
Posts: 309
Location: Alabama, USA
|
| Posted: Sun Nov 16, 2003 2:59 am Post subject: |
|
|
Use crontab to run the rsync update script at set times (in this case, 0,30 for the minutes).
I have my local rsync server sync only once every day at ~5:00 in the morning and then send me an email about what needs to be updated. That way, I know if I want to update anything or not! It's a big time/bandwidth saver.
_________________
Me |
|
| Back to top |
|
|
dan2003
Apprentice
Joined: 11 May 2003
Posts: 258
Location: somewhere near london, UK
|
| Posted: Sun Nov 16, 2003 5:25 pm Post subject: |
|
|
Another Q or 6,
what should i use to sync? should i be using the script that is detailed on the page? i.e. rsync-gentoo-portage.sh?
if i ran emerge sync from the crontab this would just update the systems own portage tree. I assume if i run the rsync-gentoo-portage.sh script it will update the tree i have in /space/gentoo/portage. Should i make the too the same i.e. use /usr/portage instead of /space/gentoo/portage?
otherwise to sync the box portage tree i have to do an emerge sync that will just sync the /usr/portage to match the /space/gentoo/portage.
I suspect i've got confused somewhere here!
I begin to think i dont need /space/gentoo/portage at all, and that i only need to do emerge sync on the server, have rsyncd running and then my other boxes sync from the server /usr/portage via rsyncd?
Is the /space/gentoo/portage bit and the script only required to host a portage server on a none gentoo box?
Please help me clear this up! |
|
| Back to top |
|
|
amne
Bodhisattva
Joined: 17 Nov 2002
Posts: 6378
Location: Graz / EU
|
| Posted: Wed Nov 19, 2003 9:14 am Post subject: |
|
|
| dan2003 wrote: | | quick Q, in the howto it says that syncs should be done every 00 and every 30 past the hour, it doesnt say where to set this. |
this is only important for the official gentoo-rsync-mirrors (as a user might sync, sync again to another mirror and have his portage tree downgraded if the mirrors aren't synced at the same time). as a private person with a private rsync mirror for your network, there's no need to sync at a specific time. you also shouldn't sync more often than once a day (except when getting lots of important GLSAs every 2 hours, etc). |
|
| Back to top |
|
|
hadfield
Retired Dev
Joined: 18 Mar 2003
Posts: 308
Location: Vancouver, BC, Canada
|
| Posted: Wed Nov 19, 2003 10:07 pm Post subject: |
|
|
| Quote: | | as a private person with a private rsync mirror for your network, there's no need to sync at a specific time. you also shouldn't sync more often than once a day |
Exactly. For all of you who are going to set up your rsync to automatically run twice an hour for your private rsync server, perhaps you should read the rsync etiquette guidelines first .
http://www.gentoo.org/news/en/gwn/20030505-newsletter.xml#doc_chap1_sect3 |
|
| Back to top |
|
|
warthog
Tux's lil' helper
Joined: 11 Aug 2003
Posts: 82
Location: Seattle (Eastside)
|
| Posted: Fri Dec 12, 2003 7:03 pm Post subject: private rsync questions |
|
|
| dan2003 wrote: |
if i ran emerge sync from the crontab this would just update the systems own portage tree. I assume if i run the rsync-gentoo-portage.sh script it will update the tree i have in /space/gentoo/portage. Should i make the too the same i.e. use /usr/portage instead of /space/gentoo/portage?
|
I have the same question, but I haven't found an answer searching the forums.
I recently setup a private rsync mirror on my own home network so that I can sync my three machines without making three connections to the gentoo mirrors. However, on the machine that is actiing as the private mirror, should it sync to itself? Or should /usr/portage and /space/gentoo/portage be combined? What is the proper thing to do?
| dan2003 wrote: |
otherwise to sync the box portage tree i have to do an emerge sync that will just sync the /usr/portage to match the /space/gentoo/portage.
|
That's my confusion exactly! |
|
| Back to top |
|
|
dan2003
Apprentice
Joined: 11 May 2003
Posts: 258
Location: somewhere near london, UK
|
| Posted: Sat Dec 13, 2003 11:33 pm Post subject: |
|
|
| nice to see im not alon on this one! been a while since i first posted the question and im still none te wiser! |
|
| Back to top |
|
|
warthog
Tux's lil' helper
Joined: 11 Aug 2003
Posts: 82
Location: Seattle (Eastside)
|
| Posted: Sun Dec 14, 2003 12:44 am Post subject: |
|
|
| dan2003 wrote: | | nice to see im not alon on this one! been a while since i first posted the question and im still none te wiser! |
Well... I did some more searching and I found this discussion which I think helps to answer our question, if not bring up some additional ones:
https://forums.gentoo.org/viewtopic.php?t=59134
in this discussion they use /usr/portage instead of a seperate partition. I have since changed my configuration to use /usr/portage and it seems to be working well. Somebody correct me if I'm wrong to do this. |
|
| Back to top |
|
|
fusion
Tux's lil' helper
Joined: 02 Nov 2002
Posts: 119
|
|
| Back to top |
|
|
|
Documentation, Tips & Tricks
