| View previous topic :: View next topic |
| Author |
Message |
handsomepete
Guru
Joined: 21 Apr 2002
Posts: 548
Location: Kansas City, MO
|
 Posted: Wed May 29, 2002 5:17 am Post subject: robots.txt? |
 |
|
I've got this file (robots.txt) in my /usr/portage directory. Google tells me that it's supposed to be something that search engines source to see what they can and can't catalog (or something like that), but I don't know how it would've gotten in the portage directory. I'm new to this webserver stuff, so any ideas/opinions would be appreciated. According to Apache's logs various IPs try to access it every now and again.
| Code: |
:/usr/portage:
[jeff@lydia.nerd.obsessive] >cat robots.txt
User-agent: *
Disallow: /
|
|
|
| Back to top |
|
 |
fghellar
Bodhisattva
Joined: 10 Apr 2002
Posts: 856
Location: Porto Alegre, BR
|
| Posted: Wed May 29, 2002 5:23 am Post subject: Re: robots.txt? |
|
|
| handsomepete wrote: | | Google tells me that it's supposed to be something that search engines source to see what they can and can't catalog (or something like that) |
Wget also respects it, so whenever you run wget on a directory (not any specific file), it will try to find this file first.
_________________
| www.gentoo.org | www.tldp.org | www.google.com | |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Portage & Programming
