GF15: Rsync etiquette OR How often should I sync?

[pjp]

Navigation: [Gentoo Fundamentals] [Table of Contents]

Rsync etiquette guidelines

The continued growth of Gentoo Linux has placed more and more demands on our mirror system. Both source mirrors as well as rsync mirrors continue to show dramatic increases in usage. As we continue to grow, the importance of using our mirroring system responsibly becomes more critical. As such, here are some rsync etiquette guidelines to keep in mind as you emerge sync:

• Sync 1-2 times per day, maximum. There's being on the bleeding edge, and there is being just plain silly. Analysis of rsync logs show that a few discourteous users syncing 10, 15 or even 25 times per day are using a disproportionate amount of rsync mirror resources. Rsync mirror maintainers have been encouraged to use iptables rules to limit people who are abusing the system.
  
• Use the rotations, not individual servers Please do not single out specific rsync mirrors and hard-code them in your configuration files. Doing so places undue stress on particular mirrors. By using country- or continent-specific rotations, you are able to select servers that are geographically close to you, yet still distribute the load over a number of servers.
  
• Report bad mirrors on bugs.gentoo.org If you notice a server in the rotation that is not responding or is showing signs of other problems, please report it on bugs.gentoo.org immediately. Many users simply assume that someone else will file the bug report, which results in nobody ever finding a bug report until a Gentoo developer happens to notice the problem. By filing bug reports sooner, we can catch and remove problematic mirrors from our rotation much faster.

Remember that all of our rsync mirrors rely entirely upon donated resources. Thus, being respectful of these donated resources is not only a common courtesy, but essential if we are to support the continued growth of Gentoo Linux.


Followup and Clarification

To follow up on the GWN article and clarify a few points:

• These are guidelines, not rules. We are asking for your help in keeping our rsync mirror system sustainable for the future. That said, if you occasionally have to sync a third time in a 24 hour period, you're not going to have jack-booted thugs show up on your doorstep with a cluebat...
  
• Only the most aggregious users will be rate limited. Syncing more than twice a day won't automatically get you rate limited. Syncing against one particular server, 50 times per day almost certainly will get you rate limited. Don't be stupid and you don't have to worry about it.
  
• Don't single out particular servers I can't stress this one enough. Not only is it poor etiquette, it's stupid as well -- you draw attention to yourself because your IP address hits the logs of that server over and over. If you use rsync.<continent>.gentoo.org or rsync.<country>.gentoo.org, you'll spread your usage over many servers. This is a Good Thing.

This is a very real problem, folks. One of our German mirror admins ran a script against his logs and found that the top 5 users were consuming over 50% of the CPU resources going to rsync on that machine. (Those users have all been rate limited, btw.) We're not trying to be jerks about this -- but when I have multiple rsync mirror admins complaining to me about the amount of resources we're taking up, we have to do something about it. With your cooperation and assistance, this GWN article will be all we have to do.
_________________
Quis separabit? Quo animo?