Harmful robots

**Harry** · #1 (**permalink**) 14-12-06, 12:29 PM

Crawling of robots helps in indexing our site but still there are certain robots which are not preferred or invited for crawling,
( we specify them in our robots.txt file ). Why such robots are harmful for our site.:eek:

kev woodman · #2 (**permalink**) 14-12-06, 01:27 PM

They could be spidering your site looking for email addresses to spam, or preparing a list of possible attack vectors or doing any other kind of information gathering that you wouldn't necessarily want to happen.

paul · #3 (**permalink**) 14-12-06, 01:41 PM

You can get in more detail about it here through the link at <b> searchnewz.com/topstory/news/sn-2-20060620BlockingHarmfulBotsFromYourSite.html </b>

kev woodman · #4 (**permalink**) 14-12-06, 01:50 PM

Good article Paul which makes the point (that I think we've made before) that robots.txt just isn't up to the job of you want to block the bad guys. To be honest the only time I ever use Robots.txt properly is if I want to block 'good' bots like Google from parts of a site, I certainly wouldn't depend on it.

paul · #5 (**permalink**) 14-12-06, 02:13 PM

In such situation I would prefer to block specific robots and IPs in our code for those robots especially which ignore robots.txt and that we have caught making nuisance.

kev woodman · #6 (**permalink**) 14-12-06, 03:12 PM

server-side is definitely the way to go

**Harry** · #7 (**permalink**) 15-12-06, 07:56 AM

Yes, if you are hosted on linux server then you can probably block the IP by using .htaccess file