CSF Blocking Access to All Accounts (WHM/CPANEL)

[joegold]

I have reported this issue in a different thread about 4 months ago but we still haven't solved the issue. We have two WHM/CPANEL servers that are running CentOS v7.9.2009. Both servers are running csf v14.12. Both servers were running fine for over 3 years until we migrated from Centos 6 to 7. Since then we have been experiencing and issue with CSF that blocks HTTP traffic to all accounts on the server on an intermittent basis. When this happens we can still access WHM/Cpanel. When we disable CSF HTTP access is restored. It is not an IP block because it happens from all IPs. Flushing all blocks doesn't help either. The only way to get back access with the CSF enabled again is to do a graceful reboot.

I do see in CSF log messages that: " kernel: Firewall: *TCP_IN Blocked*

I held off from posting about this again until I was sure that #1 the issue wasn't caused by Pyxsoft AntiMalware (of which was removed from the server last month) and #2 the issue doesn't occur after cPanel updates occurred and the server needs rebooting. Now I can say for sure that it's neither of those issues.

This is issue happens randomly at least once per week. When it does no site on the server is accessible.

Has anyone experienced this problem? What logs should I be looking at to try and figure out what is triggering this?

Thank you!

[d265]

Have you tried a clean install?

Code: Select all

./etc/csf/uninstall.sh

[Sergio]

@joegold
When this happens, what are the lines reported in CSF under
Watch System Logs -> /var/log/lfd.log

In there it should tell you what happened.

[joegold]

So the firewall was off... I enabled it, and then tried to go to a few sites on the server for which I received "Forbidden Screen"... here is what the log looks like after restart:

Nov 19 14:42:13 zeus lfd[29395]: daemon started on zeus.xxxxxxxxx.com - csf v14.12 (cPanel)
Nov 19 14:42:13 zeus lfd[29395]: LF_APACHE_ERRPORT: Set to [2]
Nov 19 14:42:13 zeus lfd[29395]: Restricting syslog/rsyslog socket acccess to group [mysyslog]...
Nov 19 14:42:13 zeus lfd[29395]: EasyApache4, using /etc/apache2/logs/error_log instead of /usr/local/apache/logs/error_log (Web Server)
Nov 19 14:42:13 zeus lfd[29395]: EasyApache4, using /etc/apache2/logs/error_log instead of /usr/local/apache/logs/error_log {ModSecurity}
Nov 19 14:42:13 zeus lfd[29395]: CSF Tracking...
Nov 19 14:42:13 zeus lfd[29395]: LOAD Tracking...
Nov 19 14:42:13 zeus lfd[29395]: Country Code Lookups...
Nov 19 14:42:13 zeus lfd[29395]: Country Code Filters...
Nov 19 14:42:13 zeus lfd[29395]: Country Code Ignores...
Nov 19 14:42:13 zeus lfd[29395]: System Integrity Tracking...
Nov 19 14:42:13 zeus lfd[29395]: Exploit Tracking...
Nov 19 14:42:13 zeus lfd[29395]: Directory Watching...
Nov 19 14:42:13 zeus lfd[29395]: Email Script Tracking...
Nov 19 14:42:13 zeus lfd[29395]: Email Queue Tracking...
Nov 19 14:42:13 zeus lfd[29395]: ModSecurity IP D/B Tracking...
Nov 19 14:42:13 zeus lfd[29395]: Email Relay Tracking...
Nov 19 14:42:13 zeus lfd[29395]: Temp to Perm Block Tracking...
Nov 19 14:42:13 zeus lfd[29395]: System Statistics...
Nov 19 14:42:13 zeus lfd[29395]: Process Tracking...
Nov 19 14:42:13 zeus lfd[29395]: Account Tracking...
Nov 19 14:42:13 zeus lfd[29395]: SSH Tracking...
Nov 19 14:42:13 zeus lfd[29395]: Webmin Tracking...
Nov 19 14:42:13 zeus lfd[29395]: SU Tracking...
Nov 19 14:42:13 zeus lfd[29395]: Console Tracking...
Nov 19 14:42:13 zeus lfd[29395]: WHM Tracking...
Nov 19 14:42:13 zeus lfd[29395]: Watching /var/log/messages...
Nov 19 14:42:13 zeus lfd[29395]: Watching /etc/apache2/logs/error_log...
Nov 19 14:42:13 zeus lfd[29395]: Watching /var/log/secure...
Nov 19 14:42:13 zeus lfd[29395]: Watching /var/log/maillog...
Nov 19 14:42:13 zeus lfd[29395]: Watching /var/log/customlog...
Nov 19 14:42:13 zeus lfd[29395]: Watching /usr/local/cpanel/logs/login_log...
Nov 19 14:42:13 zeus lfd[29395]: Watching /usr/local/cpanel/logs/access_log...
Nov 19 14:42:13 zeus lfd[29395]: Watching /var/log/exim_mainlog...

[joegold]

@d265

The firewall has been reinstalled fresh at least 8 times over the last 5 months. After a fresh install it usually works for about 2 months according to the notes I'm looking at... Its an odd problem because we have 5 servers that are basically clones. Out of the 5, only 3 keep having this issue over and over. I've even made sure the config files settings are the same on all servers. The servers that it happens on, are the 3 with a considerable amount more traffic though. I originally though the issue was related to Pyxsoft AntiMalware plugin but after removing it from 2 of the servers the issue still occurred. I just went and reinstalled CSF clean again, and copied over the config settings. Now the firewall is enabled and nothing is blocking. I did notice when I went to make a backup of the config settings that the last backup was just before CSF auto updated to v14.12. The date and time stamp of the backup shows that it was about 30 mins before the firewall started blocking all accounts again. However, the other 2 servers that usually have this issue also updated around the same time and neither of them had this issue yet...

[Sergio]

The logs don't show any block.

Try to do this, find out what is your Internet connection IP and then check all the logs if there is any block to that IP.

Just curious, Did Configserver installed CSF in your server or you installed yourself?
I ask this because if you have a payed installation you can contact them asking for help directly.

[joegold]

I ended up removing CSF completely, and then reinstalling it.. again. But this time instead of using the backup settings file, I manually adjusted and redid every setting (with the same exact settings as before) and saved it. Its been about 12 days now and no issues.... yet... However, one of the other servers that I did not yet do this to, locked all IP's out last night again. One common thing I noticed with all the servers but cPanel support won't confirm or acknowledge is this always seems to happen after a cPanel update. Last night was no exception either. When I got the notice the server was down, I logged into WHM and saw that there was a cPanel update about 45 mins earlier. Cpanel support wont, of course, support any third party applications like CSF so they won't help. I'm going to do the same manual reconfig of settings to the next server tomorrow and then we will just wait it out and see if that fixes the issue or not. I will update this thread again soon.

[Sergio]

It will nice if you compare both configurations and find out what line is different.

[joegold]

@Serigio - The configs were absolutely identical. I opened the config in a browser window before I reinstalled CSF in a different browser window. After installation I copied each setting, one by one...

[Sergio]

@joegold,
Do you have cPHulk installed? Have you checked in there if the IP is not blocked in there?
Also, have you checked in cPanel ModSecurity Tools if the IP was not blocked in there?

It could happen that maybe some of the other tools are blocking the IPs and not CSF.