Hi,
About 21 days ago I registered a new www domain.
The site has only been up for an hour or two on/off as it's in development, and a blank page comes up currently when the url is visited.
I went in tonight to change features on the hosting, and thought I'd have a look at the visitor log, it's copied below:
216.144.233.206 - - [22/Nov/2004:01:19:39 +0000] "GET /robots.txt HTTP/1.0" 404 - "-" "-"
158-147-185-84.harris.com - - [23/Nov/2004:21:33:41 +0000] "GET / HTTP/1.1" 200 464 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)"
crawl12-public.alexa.com - - [26/Nov/2004:14:11:30 +0000] "GET /robots.txt HTTP/1.0" 404 - "-" "ia_archiver"
crawl12-public.alexa.com - - [26/Nov/2004:14:11:30 +0000] "GET / HTTP/1.0" 200 464 "-" "ia_archiver"
So pests are scanning for my robots.txt file already!
Excuse me if this is stupid, but why are they scanning my robots.txt file? Isn't that the text file that gets you into search engines? How do they extrapolate data from it?
Can you take precautions against them? What are they doing?
Yours confused
Thanks in advance