IP Reputation Issues

[khimaira]

After enabling IP reputation integration, I had multiple occasions of cpanel monitoriog reporting lfd failures and subsequent restarts on 2 servers. From lfd.log I was able to determine that the URI::Escape module was not installed. I manually installed that and thought the lfd stops and starts would be corrected. Alas I am now seeing FASTART errors that seem to coincide with the failures. I currently have the integration disabled, but immediately after enabling it I see something like this:

Sep 19 13:58:43 host2 lfd[750613]: cxs Reputation Enabled...
Sep 19 13:58:43 host2 lfd[750613]: LOAD Tracking...
Sep 19 13:58:43 host2 lfd[750613]: Messenger HTTPS Service starting...
Sep 19 13:58:43 host2 lfd[750613]: Messenger HTML Service starting...
Sep 19 13:58:43 host2 lfd[750613]: Messenger TEXT Service starting...
Sep 19 13:58:43 host2 lfd[750613]: Cluster Service starting...
Sep 19 13:58:43 host2 lfd[750613]: Blocklist Tracking...
Sep 19 13:58:43 host2 lfd[750613]: Country Code Filters...
Sep 19 13:58:43 host2 lfd[750613]: Country Code Lookups...
Sep 19 13:58:43 host2 lfd[750613]: System Integrity Tracking...
Sep 19 13:58:43 host2 lfd[750613]: Exploit Tracking...
Sep 19 13:58:43 host2 lfd[750613]: Directory Watching...
Sep 19 13:58:43 host2 lfd[750613]: Email Script Tracking...
Sep 19 13:58:43 host2 lfd[750613]: Email Queue Tracking...
Sep 19 13:58:43 host2 lfd[750613]: Email Relay Tracking...
Sep 19 13:58:43 host2 lfd[750613]: System Statistics...
Sep 19 13:58:43 host2 lfd[750613]: Port Scan Tracking...
Sep 19 13:58:43 host2 lfd[750613]: Connection Tracking...
Sep 19 13:58:43 host2 lfd[750613]: Process Tracking...
Sep 19 13:58:43 host2 lfd[750613]: Account Tracking...
Sep 19 13:58:43 host2 lfd[750613]: SSH Tracking...
Sep 19 13:58:43 host2 lfd[750613]: SU Tracking...
Sep 19 13:58:43 host2 lfd[750613]: Console Tracking...
Sep 19 13:58:43 host2 lfd[750613]: WHM Tracking...
Sep 19 13:58:43 host2 lfd[750613]: Watching /var/log/maillog...
Sep 19 13:58:43 host2 lfd[750613]: Watching /var/log/exim_mainlog...
Sep 19 13:58:43 host2 lfd[750613]: Watching /usr/local/cpanel/logs/login_log...
Sep 19 13:58:43 host2 lfd[750613]: Watching /var/log/messages...
Sep 19 13:58:43 host2 lfd[750613]: Watching /usr/local/cpanel/logs/access_log...
Sep 19 13:58:43 host2 lfd[750613]: Watching /var/log/secure...
Sep 19 13:58:43 host2 lfd[750613]: Watching /etc/apache2/logs/error_log...
Sep 19 13:58:44 host2 lfd[750622]: *Error* FASTSTART: (Blocklist [CXS_ALL] IPv4) [] [iptables-restor
e: line 2 failed]
Sep 19 13:58:44 host2 lfd[750622]: Retrieved and blocking blocklist GREENSNOW IP address ranges
Sep 19 13:59:08 host2 lfd[750618]: Messenger HTTPS Service started for 1912 domains
Sep 19 13:59:08 host2 lfd[750618]: lfd HTTPS messenger using 2019232 kB of RSS memory at startup, adding up to 10 children = 22211552 kB
Sep 19 13:59:08 host2 lfd[750618]: lfd HTTPS messenger using 2195344 kB of VIRT memory at startup, adding up to 10 children = 24148784 kB

It appears this error may be related to the failures of the lfd daemon as there are no problems when the integration is disabled.. It's certainly possible there are some duplicate IP's but I'd like to assume these are handled by the scripts.

Any assistance would be appreciated.

[ForumAdmin]

Is there anything unusual in /var/lib/csf/csf.block.CXS_ALL?

[khimaira]

Yes, 1432 IP addresses.

[khimaira]

I misread the question. There are no headers,spurious characters, or blank lines

[ForumAdmin]

That error will have nothing to do with lfd failing as it is an informational error message. Given the huge size of lfd HTTPS messenger it's more likely a memory issue and you should consider enabling MESSENGERV2.

[khimaira]

Thank you for the suggestion. I have created the /home/csf/public_html directory and enabled messengerv2. I'll wait to be sure it is stable before retrying the blocklist integration.

[khimaira]

Presumably unrelated to my original issue, I've noticed a major change in the cxs notices regarding file uploads since creating /home/csf/public_html and enabling messengerv2. All of the script upload notices show a web upload script path of /home/csf/pubplic_html/wp-admin which of course does not exist.

Web upload script user : nobody (99)
Web upload script owner: ()
Web upload script path : /home/csf/public_html/wp-admin
Web upload script URL : http://XXXXX.org/wp-admin/admin-ajax.php
Remote IP : 69.80.72.153
Deleted : No
Quarantined : Yes [/home/quarantine/cxscgi/20170922-114516-WcUwDFLKuvLGlo7569wvfQAAAMI-file-oJScRD.1506095116_1]

NOTE: This alert may be a ModSecurity false-positive as /home/csf/public_html/wp-admin does not exist

Additionally /home/csf/public_html/error_log is filling with lines of this:

[22-Sep-2017 14:11:44 America/New_York] PHP Notice: Undefined index: HTTP_ACCEPT_LANGUAGE in /home/csf/public_html/index.php on line 12
[22-Sep-2017 14:11:56 America/New_York] PHP Notice: Undefined index: HTTP_ACCEPT_LANGUAGE in /home/csf/public_html/index.php on line 12

The directory permissions and ownership are:
drwxrwx--x 6 csf csf 4096 Sep 20 13:37 .cagefs/
drwxr-xr-x 2 csf csf 4096 Sep 20 13:35 .cl.selector/
-rw-r--r-- 1 csf csf 1108 Sep 20 13:36 en.php
drwx--x--x 2 csf nobody 4096 Sep 20 13:37 public_html/
-rw-r--r-- 1 csf csf 199 Sep 22 02:34 recaptcha.php
-rw------- 1 csf csf 0 Sep 21 13:25 unblock.txt

I don't have a clue where to start looking for the crossed wires.

[khimaira]

I tried enabling the IP Reputation with Messengerv2 running and it still causes LFD crashes. Incidentally, there are no where near the 19212 domains mentioned in the log on this server. Maybe 25% of that.

The service “lfd” appears to be down.


Server XXXXXXX.com
Primary IP Address XXXXXXX
Service Name lfd
Service Status failed
Notification The service “lfd” appears to be down.
Service Check Method The system’s command to check or to restart this service failed.
Number of Restart Attempts 1
Startup Log No startup log
Memory Information
Used 6.24 GB
Available 9.25 GB
Installed 15.49 GB
Load Information 0.92 0.78 0.64
Uptime 203 days, 15 hours, 51 minutes, and 40 seconds
IOStat Information avg-cpu: %user %nice %system %iowait %steal %idle 2.66 0.42 0.65 0.15 0.00 96.13 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sdb 61.65 1739.31 805.56 30605422982 14174857735 sda 99.49 4157.07 805.56 73149065601 14174857735 sdc 7.96 130.28 128.57 2292410370 2262375280 sdd 4.23 296.48 1407.48 5216998114 24766437424 md127 0.00 0.01 0.01 97922 158548 md126 3.25 11.08 14.92 194985304 262608800 md125 51.54 2225.68 201.19 39163707194 3540262672 md124 5.08 185.63 34.72 3266464178 610905704 md123 2.10 21.33 12.46 375258812 219320200 md122 40.16 234.03 515.42 4118048778 9069498088 md121 1.91 0.22 15.67 3870474 275797432
Top Processes
PID Owner CPU % Memory % Command
759334 root 15.45 0.08 /usr/local/cpanel/3rdparty/bin/perl /usr/sbin/lfd
969748 root 9.09 6.80 /usr/local/cpanel/3rdparty/bin/clamd
759284 root 6.39 0.20 tailwatchd - chkservd - ftpd check
46118 mysql 1.32 5.86 /usr/sbin/mysqld --basedir=/usr --datadir=/ssd1/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/log/mysqld.log --open-files-limit=50000 --pid-file=/ssd1/mysql/XXXXXX.com.pid --socket=/var/lib/mysql/mysql.sock --port=3306
6716 root 1.08 0.19 collectl-cloudlinux -D --pname cloudlinux -C /etc/cloudlinux-collect

I have disabled it. Also removing Messengerv2 and /home/csf/public_html has stopped the CXS file upload notices from showing the path as /home/csf.. The file /etc/apache2/conf.d/csf.messenger.conf was removed automatically.

I'm stuck without the enhancements.