Go Back   Web Hosting UK Forums | Linux Windows Dedicated Server and cPanel VPS Hosting Forum > Services > Web Designing and Development

Reply
 
LinkBack Thread Tools Display Modes
  #1 (permalink)  
Old 09-10-2007, 04:13 PM
SBh SBh is offline
Member
 
Join Date: Sep 2007
Posts: 58
Default Want ip or bot string for all popular bots.

All search bots has some specific part in their bot-name or bot ip by which it can be uniquely identified. Identification are done in two ways:
* By the bot string- There are certain fixed string by which it can identified as a specific bot. Like google bot it's bot name always has a part 'googlebot'. By this we can identify it as google bot.
* By it's IP address- Some bots have fixed IP address by which they crawl.

So, what I want is a list for all popular bots along with bot string or bot IP address, by which it can be uniquely identified.
Actually, I want this to make my site behave differently for search engine bots while for normal visitors it will behave normally.

I have seen this type of system in phpBB, so I was thinking to implement this in my site, for that i need the complete lists of bots.

If someone can help me please do so, it will be highly appreciated.
Reply With Quote
  #2 (permalink)  
Old 09-22-2007, 06:01 AM
paul's Avatar
Senior Member
 
Join Date: Apr 2006
Location: Norway
Posts: 1,642
Default

Not sure but probably this will help you if you're looking to track bot/crawlers; tracksite.ru/ts/web-bot-crawler-spider_tracking.html.en.html
Also you can refer some of the relevant info through the white papers mentioned in honeynet.org.
__________________

Reply With Quote
  #3 (permalink)  
Old 09-22-2007, 06:35 PM
SBh SBh is offline
Member
 
Join Date: Sep 2007
Posts: 58
Default

Thanks for the help that is quite helpful.
However it is not what I was looking for. I found this searchenginedictionary.com/spider-names.shtml
This one is quite matching to my need. If you find something similar to this one then let me know.
Reply With Quote
  #4 (permalink)  
Old 09-24-2007, 09:35 AM
paul's Avatar
Senior Member
 
Join Date: Apr 2006
Location: Norway
Posts: 1,642
Default

Sound good you find that, I checked that link that seems to have a comprehensive list of robot with location, I bookmarked it, thanks for the link SBh!
__________________

Reply With Quote
  #5 (permalink)  
Old 10-15-2007, 05:01 PM
SBh SBh is offline
Member
 
Join Date: Sep 2007
Posts: 58
Default

Ok.. I have a new problem
Actually I can not find any difference between the Googlebot for Search engine and Googlebot for GoogleAdSense. It seems that both use same bot with name GOOGLEBOT.
So, Can anybody tell me how to distinguish those two.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is On
Trackbacks are On
Pingbacks are On
Refbacks are On
Forum Jump


All times are GMT. The time now is 10:43 PM.
Copyright 2002-2007 WebHosting.uk.com. All rights reserved.
Web Hosting UK Forum