CSF blocking all bots (google, msn, yahoo etc.)¨

[bluedog]

Hi, can anybody tell me how can I allow bots from Google, MSN etc. to crawl my website without being temporally blocked?
Thanks a lot!

[ForumAdmin]

You need to look at using /etc/csf/csf.rignore

[bluedog]

Thanks for the answer. I know that, but what do I have to add there? I dont want to add all google IPs, that will be really long list and I dont know the IPs for Bing and other bots.

[ForumAdmin]

That file is not for adding IP addresses:

# The following is a list of domains and partial domain that lfd process
# tracking will ignore based on reverse and forward DNS lookups. An example of
# its use is to prevent web crawlers from being blocked by lfd, e.g.
# .googlebot.com and .crawl.yahoo.net
#
# You must use either a Fully Qualified Domain Name (FQDN) or a unique ending
# subset of the domain name which must begin with a dot (wildcards are NOT
# otherwise permitted)
#
# For example, the following are all valid entries:
# www.configserver.com
# .configserver.com
# .configserver.co.uk
# .googlebot.com
# .crawl.yahoo.net
# .search.msn.com
#
# The following are NOT valid entries:
# *.configserver.com
# *google.com
# google.com (unless the lookup is EXACTLY google.com with no subdomain
#
# When a candidate IP address is inspected a reverse DNS lookup is performed on
# the IP address. A forward DNS lookup is then performed on the result from the
# reverse DNS lookup. The IP address will only be ignored if:
#
# 1. The results of the final lookup matches the original IP address
# AND
# 2a. The results of the rDNS lookup matches the FQDN
# OR
# 2b. The results of the rDNS lookup matches the partial subset of the domain
#
# Note: If the DNS lookups are too slow or do not return the expected results
# the IP address will be counted towards the blocking trigger as normal

[bluedog]

Thanks a lot!!!