ConfigServer Community Forum

hey, i see a post of someone his cron got a block and crawler bots as well.
viewtopic.php?p=20673&sid=bbe730ccbd82c ... 746#p20673

I notice sometimes google bot got a block with more other ones... what we can do about that?

I look on csf.ringore
but I didn't understand.
is not get IPs

so what i actually need to do if the crawler have like that:
crawl-66-249-64-130.googlebot.com

and if i need to comment out all of this:
# For example, the following are all valid entries:
# www.configserver.com
# .configserver.com
# .configserver.co.uk
# .googlebot.com
# .crawl.yahoo.net
# .search.msn.com

The file is not detailed, you dont know if is block them or give them a pass.
you didn't specified the word "comment out" so is can be something also.... no?

Please help and direction what is right and if i comment it out, is actually good or bad? i didn't get the explanation on the file
is not detailed right and people can do mistakes.

Hi.
if you have:

so what i actually need to do if the crawler have like that:
crawl-66-249-64-130.googlebot.com

Thenm on the csf.rignore file you should add:
.googlebot.com

The following is the reason why you have to write it like that:
google bots have a lot of different IPs and servers. All the servers are defined per example:
crawl-###-###-###-###.googlebot.com
where the "###" represents octects on the rage of 0 to 255 each
So, if you want to ignore all the google bots you have to do it like ".googlebot.com", and that will ignore any google bot servers.

Regards,
Sergio