Utilizing CSF to block all bots while allowing an exception for Google bots
Posted: 30 Aug 2023, 14:44
Hello everyone,
I'm inquiring about the possibility of utilizing CSF to block all bots while allowing an exception for Google bots. Google has kindly provided us with a list of whitelisted IPs at the following links:
Googlebot IPs: https://developers.google.com/static/se ... lebot.json
Special Crawlers IPs: https://developers.google.com/static/se ... wlers.json
User-Triggered Fetchers IPs: https://developers.google.com/static/se ... chers.json
Google IPs: https://www.gstatic.com/ipranges/goog.json
I've attempted to research this on various forums but haven't been able to find a solution. Most resources seem to focus on blocking specific IPs rather than implementing an exclusion for Google bots. Your insights would be greatly appreciated.
I'm inquiring about the possibility of utilizing CSF to block all bots while allowing an exception for Google bots. Google has kindly provided us with a list of whitelisted IPs at the following links:
Googlebot IPs: https://developers.google.com/static/se ... lebot.json
Special Crawlers IPs: https://developers.google.com/static/se ... wlers.json
User-Triggered Fetchers IPs: https://developers.google.com/static/se ... chers.json
Google IPs: https://www.gstatic.com/ipranges/goog.json
I've attempted to research this on various forums but haven't been able to find a solution. Most resources seem to focus on blocking specific IPs rather than implementing an exclusion for Google bots. Your insights would be greatly appreciated.