Robots.txt or .htaccess file

[harry]

Which file you will prefer to prevent unnecessary robots from crawling your site;
using a robots.txt file or a .htaccess file.

[kev woodman]

Robots if you're just interested in keeping out spiders. You can do all sorts of clever stuff with .htaccess but it's probably over the top for blocking search engines. If you're talking about securing an area of the site then a proper user authentication/management system is a better solution than both of the above.

[Rodney]

IMO using robots.txt file is a good option for stopping unnecessary spiders from crawling your website.

[chris]

I would probably go with robots.txt file to prevent robots from crawling the site. I think Kev has explained it in a nut-shell.

[harry]

Thanks for your suggestions,
I think using robots.txt will not solve the main problem because there are some bad robots which don’t obey or follow robots.txt file it would be better to prevent them by specifying in .htaccess file as this file would not allow them to enter in the site itself.

[kev woodman]

You might find that using htaccess is no more effective against bad bots I'm afraid Harry. If you were going to the trouble of harvesting using a bot you'd probably code it to identify itself as a 'good' bot anyway. Still every little helps.

[harry]

kev, using log files once you found who are the culprits then you can block IP addresses from where these bad robots are coming from using .htaccess file,
then there would be no way for bad rebots for peeping into your site.

[kev woodman]

Well if it was me doing the harvesting I'd just fake the IP and the agent identification and I'd get the script to randomise these as well...and just for good measure I'd route through proxies as well. If you're trying to stop bots harvesting emails and the like you're going to need to use some sort of server side scripting as htaccess or a robots file just won't cut it.

[harry]

kev, I think it’s a programmer view. I know you would like to go behind scripts.
Although your suggestion is worth.

[kev woodman]

Scripting Web bots is a really interesting topic - they are really autonomous or semi-autonomous software agents and the good ones include a fair amount of AI. You could make a fair amount of money scripting them if you don't mind walking on the dark side for a bit.

[harry]

I hope we should be at constructive end while scripting for these web bots and should take care they will not hurt anybody.