CSF LFD Ignore ASN Member IP addresses
Posted: 05 Mar 2020, 17:50
I'm seeking ideas for the most native way to create a configuration to make LFD ignore all IP addresses within a specific Autonomous System Number.
The use case is this: I don't want to block legitimate web crawlers on my server which may trigger failure logs due to HTTP 403's and HTTP 404's. This is frequently a common behavior for websites I host with many pages, which are crawled by search engines. While I can create an /etc/csf/csf.rignore entry for search engines such as .google.com and .googlebot.com, there are other legitimate web crawlers which do not utilize rDNS. An example of a big legitimate web crawler which does not use rDNS is Facebook, Inc. Facebook suggests whitelisting an entire ASN per https://developers.facebook.com/docs/sh ... s/crawler/
Is there any native feature in CSF/LFD which would support ignoring a named ASN? The less non-native scripting, the better, but if no other options are available, I suppose a script would have to be the final solution - unless WayToTheWeb group will add this feature into CSF/LFD.
The use case is this: I don't want to block legitimate web crawlers on my server which may trigger failure logs due to HTTP 403's and HTTP 404's. This is frequently a common behavior for websites I host with many pages, which are crawled by search engines. While I can create an /etc/csf/csf.rignore entry for search engines such as .google.com and .googlebot.com, there are other legitimate web crawlers which do not utilize rDNS. An example of a big legitimate web crawler which does not use rDNS is Facebook, Inc. Facebook suggests whitelisting an entire ASN per https://developers.facebook.com/docs/sh ... s/crawler/
Is there any native feature in CSF/LFD which would support ignoring a named ASN? The less non-native scripting, the better, but if no other options are available, I suppose a script would have to be the final solution - unless WayToTheWeb group will add this feature into CSF/LFD.