High number of ICMP ping packets ONLY when CSF is on

[GoWilkes]

I'm running CentOS v7.9.2009 with WHM/cPanel. I use Cloudflare and CSF with the Cloudflare extension.

A few weeks ago, my sites started throwing intermittent Cloudflare 520 errors. The server load was fine and there was nothing in the Cloudflare logs, but I saw tons of these in /var/log/messages:

Feb 5 20:12:08 xxxx kernel: Firewall: ICMP_IN Blocked IN=venet0 OUT= MAC= SRC=13.234.35.125 DST=xxx.xx.xx.xx LEN=36 TOS=0x00 PREC=0x00 TTL=225 ID=25927 DF PROTO=ICMP TYPE=8 CODE=0 ID=24 SEQ=17491

Feb 5 20:12:09 xxxx kernel: Firewall: ICMP_IN Blocked IN=venet0 OUT= MAC= SRC=3.27.243.34 DST=xxx.xx.xx.xx LEN=36 TOS=0x00 PREC=0x00 TTL=238 ID=17887 DF PROTO=ICMP TYPE=8 CODE=0 ID=9 SEQ=20172

Feb 5 20:12:09 xxxx kernel: Firewall: ICMP_IN Blocked IN=venet0 OUT= MAC= SRC=3.25.244.230 DST=xxx.xx.xx.xx LEN=36 TOS=0x00 PREC=0x00 TTL=235 ID=37271 DF PROTO=ICMP TYPE=8 CODE=0 ID=9 SEQ=20172

Feb 5 20:12:09 xxxx kernel: Firewall: ICMP_IN Blocked IN=venet0 OUT= MAC= SRC=3.27.215.45 DST=xxx.xx.xx.xx LEN=36 TOS=0x00 PREC=0x00 TTL=238 ID=34018 DF PROTO=ICMP TYPE=8 CODE=0 ID=9 SEQ=20172

Feb 5 20:12:09 xxxx kernel: Firewall: ICMP_IN Blocked IN=venet0 OUT= MAC= SRC=54.226.52.109 DST=xxx.xx.xx.xx LEN=36 TOS=0x00 PREC=0x00 TTL=233 ID=5851 DF PROTO=ICMP TYPE=8 CODE=0 ID=32 SEQ=18750
All of the SRC= IPs trace back to Amazon.

I flushed all of the temporary and permanent blocks in CSF, but that didn't help. I also made sure that all Cloudflare IPs were whitelisted. But I found that if I disabled CSF then the problem went away!

I sort of forgot about it until a few days ago, and then I turned CSF back on. Within 30 seconds I started seeing a spike in ICMP_IN Blocked; again, all pointing to Amazon IPs. When I looked at the "Last 100 ip tables log" in CSF, 93 of the last 100 were ICMP.

I posted in the Cloudflare forum with no help. I reached out to my server provider, too, but they don't see a problem on their end.

Any suggestions?

[Sergio]

Hi.
Please read https://download.configserver.com/csf/readme.txt
maybe some of your questions will be answered there.

Sergio

[GoWilkes]

Haha, it's been awhile since I've been told to RTFM! LOL

I've been using CSF for 14 years, though, and I'm pretty familiar with it. I didn't see anything in the doc that gave me a clue on this issue, though. If you think I'm overlooking something, can you give me a clue of what I should be looking for?

[Sergio]

Sorry if that didn't help.

But in there you can read a complete info about CloudFlare, that may be can give you a light on what else you could check.

My apologize in trying to help you.

27. CloudFlare
##############

This features provides interaction with the CloudFlare Firewall.

As CloudFlare is a reverse proxy, any attacking IP addresses (so far as
iptables is concerned) come from the CloudFlare IP's. To counter this, an
Apache module (mod_cloudflare) is available that obtains the true attackers
IP from a custom HTTP header record (similar functionality is available
for other HTTP daemons.

However, despite now knowing the true attacking IP address, iptables cannot
be used to block that IP as the traffic is still coming from the CloudFlare
servers.

CloudFlare have provided a Firewall feature within the user account where
rules can be added to block, challenge or whitelist IP addresses.

Using the CloudFlare API, this feature adds and removes attacking IPs from that
firewall and provides CLI (and via the UI) additional commands.

There are several restrictions to using this feature:

1. All lfd blocks will be temporary blocks so that csf/lfd can keep blocks in
sync with CloudFlare

2. Automatic blocks via lfd are limited to LF_MODSEC and LF_CXS triggers as
only through these can the domain name be determined. Any users that own
domains that are involved in the trigger will get a block in their
CloudFlare Firewall. Additionally, any users with the special case "any"
will also get blocks

3. The temporary/permanent config of the lfd settings are ignored and CF_TEMP
is used instead

4. LF_TRIGGER must not be used, the feature will not work with it enabled

5. mod_cloudflare or similar must be used to report real IP in the Apache logs

6. URLGET must be set to 2 (i.e. LWP) must be used

7. If PERMBLOCK is used, the last tempblock will remain and never be cleared.
So any CloudFlare Firewall entries must be manually cleared in CloudFlare
or via CLI

8. There are restrictions imposed by CloudFlare to the number of rules that
can be created depending on the type of account used. See
https://goo.gl/ssGu7v for more information

9. When restarting csf, any old temporary blocks will still be created for lfd
to clear when it restarts

10. All interaction with CloudFlare is at User-level, not Zone-level

11. If using the CloudFlare cPanel user plugin, it must be v7+

CF_TEMP should be configured taking into account the maximum number of rules
that the CloudFlare account allows: https://goo.gl/ssGu7v

All CloudFlare users for the domains that are involved in LF_MODSEC and
LF_CXS triggers will have a CloudFlare rule added. Any CloudFlare account
configured to use the special case "any" field value in csf.cloudflare will
have a CloudFlare rule added regardless of domain.

NOTE: You should always list the CloudFlare IP addresses in /etc/csf/csf.ignore
to prevent them from being blocked by lfd from https://www.cloudflare.com/ips/

[GoWilkes]

Keeping you in the loop, I modified the LF_TRIGGER setting as suggested and then reenabled CSF 2 days ago. So far so good! My server load is constantly higher than before I enabled CSF, but I don't think that's related.

Thanks for the suggestions, I really appreciate it!