PPRuNe Forums - View Single Post - Pest-bots already scanning...
View Single Post
Old 1st December 2004 | 13:30
  #2 (permalink)  
Memetic
Supercalifragilistic
expialidocious
20 Anniversary
 
Joined: Sep 2001
Posts: 589
Likes: 0
From: Essex, UK
Why do you say pest?

Hi WG774,

It looks like your site is being crawled by the bot for www.alexa.com

Now i'd not choose to add the Alexa tool bar to my own machines, but i'm curious, why do you consider it's attempt to index your site as the actions of a pest?

regards

Memetic

P.S. Looking for your robots.txt file means the bot is being well behaved. The following might be of interest:

http://www.robotstxt.org/wc/exclusion-admin.html

Last edited by Memetic; 1st December 2004 at 14:11.
Memetic is offline