Problem with Google accessing site.

[postie]

Thought I'd put up a quick post here describing the problems I've been having over the last few days.

I use google webmaster tools a fair bit, and noticed that both my sitemap.xml and robots.txt were coming back as not found. (or other errors like "inaccessible" )

I checked everything and even uploaded a new robots file, but the problem persisted and eventually I had a natter with technical support on Live chat.

The upshot of the natter was that the server firewall is blocking Google's IP. The tech guy, unblocked all IPs on the firewall. I tested if my site could be accessed. Google was able to and it returned a "success" status.

So, a couple of hours go by, I check again.. and Google is blocked. Another, longer chat with Tech support, same answer, the server firewall is blocking Google. Unblock IP's on the firewall and again.. Google is able to access my site.

Just checked and guess what? Indeed. Google isn't able to access my site. So, instead of asking Tech support to unblock Google's IP, I thought I'd mention it here. As it might get the issue resolved, instead of daily asking Tech for Google's IP's to be unblocked from the server firewall.

[sysadmin]

Originally Posted by postie

Thought I'd put up a quick post here describing the problems I've been having over the last few days.

I use google webmaster tools a fair bit, and noticed that both my sitemap.xml and robots.txt were coming back as not found. (or other errors like "inaccessible" )

I checked everything and even uploaded a new robots file, but the problem persisted and eventually I had a natter with technical support on Live chat.

The upshot of the natter was that the server firewall is blocking Google's IP. The tech guy, unblocked all IPs on the firewall. I tested if my site could be accessed. Google was able to and it returned a "success" status.

So, a couple of hours go by, I check again.. and Google is blocked. Another, longer chat with Tech support, same answer, the server firewall is blocking Google. Unblock IP's on the firewall and again.. Google is able to access my site.

Just checked and guess what? Indeed. Google isn't able to access my site. So, instead of asking Tech support to unblock Google's IP, I thought I'd mention it here. As it might get the issue resolved, instead of daily asking Tech for Google's IP's to be unblocked from the server firewall.

Hi,

You can try whitelisting the GoogleBot IP ranges, which can be found on their webmastertools site.

Here are some of them for your reference:

216.239.32.0/19 # Googlebot
64.233.160.0/19 # Googlebot
66.249.80.0/20 # Googlebot
72.14.192.0/18 # Googlebot
209.85.128.0/17 # Googlebot
66.102.0.0/20 # Googlebot
74.125.0.0/16 # Googlebot
64.18.0.0/20 # Googlebot
207.126.144.0/20 # Googlebot

Google-Sitemaps/1.0
66.249.84.68
66.249.85.84
72.14.194.21
72.14.195.223

Googlebot/2.1
66.249.67.65

Google (Various User-agents)

66.249.66.195
66.249.70.147
66.249.84.10
66.249.85.129
72.14.194.29
74.125.16.1

66.249.70.36
74.125.16.70

More info : Does my host block Google - Network and Firewall troubleshooting
Complete IP list : http://iplists.com/nw/google.txt

Good luck