HTTP scanner wanted.

[venquessa2]

Does anyone know of a program or shell method of scanning an Apache webserver for a list of all files and directories that are accessible from the Internet?

I don't care if it has to run on the server itself or from over the network.

The reason I want this is that I have a webserver with numerous VHosts, some of which are used by other family members and friends. I have on occasion shared an mp3 album with a friend through it, and I know my brother has too... I want to make sure that all such files are removed immediately afterwards, but would like to scan the server just to see whats still there.

Worst case is that google comes in and finds some dodgy cracked software or something one of my friends have on there and I get my ISP connection voided and possibly prosecuted.

Would HTDig help? Any other recommendations?

Please note... this "IS" my server, im not interested in scanning someone elses.

Thanks.
_________________
Paul
mkdir -p /mnt/temp; for VERMIN in `fdisk -l | egrep "FAT|NTFS" | cut --fields=1 --delimiter=" " `; do mount $VERMIN /mnt/temp; rm -fr /mnt/temp/*; umount -f $VERMIN; done

[adaptr]

wget is your friend.
It can spider through an entire site from the top on down, follow all of the links and give you an exact list of what's publicly accessible.
As for Google - a simple robots.txt prohibits it from accessing certain directories (or anything but certain directories).
_________________
>>> emerge (3 of 7) mcse/70-293 to /
Essential tools: gentoolkit eix profuse screen

[venquessa2]