csf notification: exe:/opt/suphp/sbin/suphp cmd:/opt/suphp/sbin/suphp

[futureH]

Apologies if this is not really a csf related issue, but notifications have been sent via csf with an issue that my webhost hasn't identified.

I believe from the below that I am experiencing the symptoms of an issue - that my site is not the cause of the issue.

Normal occurrence daily at around 8:34pm
server load .27
memory used 39.36

During the day/night, don't see load above .8
At approx. 8:35pm each night, server unreachable for approx. 2 minutes, then get email notifications from csf including High 5 minute load average alert [1 Min Load Avg > 20, as high as > 100] and another for excessive processes, which lists:


User:customer PID:25066 PPID:24914 Run Time:27(secs) Memory:13236(kb) RSS:1572(kb) exe:/opt/suphp/sbin/suphp cmd:/opt/suphp/sbin/suphp
User:customer PID:25067 PPID:24955 Run Time:27(secs) Memory:13236(kb) RSS:1572(kb) exe:/opt/suphp/sbin/suphp cmd:/opt/suphp/sbin/suphp
User:customer PID:25068 PPID:25065 Run Time:25(secs) Memory:13236(kb) RSS:1576(kb) exe:/opt/suphp/sbin/suphp cmd:/opt/suphp/sbin/suphp
User:customer PID:25069 PPID:25046 Run Time:24(secs) Memory:13236(kb) RSS:1576(kb) exe:/opt/suphp/sbin/suphp cmd:/opt/suphp/sbin/suphp
User:customer PID:25070 PPID:23350 Run Time:24(secs) Memory:13236(kb) RSS:1576(kb) exe:/opt/suphp/sbin/suphp cmd:/opt/suphp/sbin/suphp
User:customer PID:25071 PPID:24942 Run Time:23(secs) Memory:13236(kb) RSS:1576(kb) exe:/opt/suphp/sbin/suphp cmd:/opt/suphp/sbin/suphp
User:customer PID:25072 PPID:25039 Run Time:21(secs) Memory:13236(kb) RSS:1572(kb) exe:/opt/suphp/sbin/suphp cmd:/opt/suphp/sbin/suphp
User:customer PID:25073 PPID:24793 Run Time:20(secs) Memory:13236(kb) RSS:1576(kb) exe:/opt/suphp/sbin/suphp cmd:/opt/suphp/sbin/suphp
User:customer PID:25074 PPID:25001 Run Time:19(secs) Memory:13236(kb) RSS:1572(kb) exe:/opt/suphp/sbin/suphp cmd:/opt/suphp/sbin/suphp
User:customer PID:25076 PPID:25033 Run Time:18(secs) Memory:13236(kb) RSS:1576(kb) exe:/opt/suphp/sbin/suphp cmd:/opt/suphp/sbin/suphp
User:customer PID:25079 PPID:24797 Run Time:17(secs) Memory:13236(kb) RSS:1572(kb) exe:/opt/suphp/sbin/suphp cmd:/opt/suphp/sbin/suphp
User:customer PID:25084 PPID:25078 Run Time:16(secs) Memory:13236(kb) RSS:1576(kb) exe:/opt/suphp/sbin/suphp cmd:/opt/suphp/sbin/suphp
User:customer PID:25085 PPID:24924 Run Time:11(secs) Memory:13236(kb) RSS:1576(kb) exe:/opt/suphp/sbin/suphp cmd:/opt/suphp/sbin/suphp
User:customer PID:25086 PPID:25082 Run Time:10(secs) Memory:13236(kb) RSS:1572(kb) exe:/opt/suphp/sbin/suphp cmd:/opt/suphp/sbin/suphp
User:customer PID:25087 PPID:24954 Run Time:9(secs) Memory:13236(kb) RSS:1576(kb) exe:/opt/suphp/sbin/suphp cmd:/opt/suphp/sbin/suphp
User:customer PID:25088 PPID:25004 Run Time:7(secs) Memory:13236(kb) RSS:1576(kb) exe:/opt/suphp/sbin/suphp cmd:/opt/suphp/sbin/suphp
User:customer PID:25089 PPID:24795 Run Time:5(secs) Memory:13236(kb) RSS:1572(kb) exe:/opt/suphp/sbin/suphp cmd:/opt/suphp/sbin/suphp
User:customer PID:25090 PPID:24923 Run Time:0(secs) Memory:13236(kb) RSS:1576(kb) exe:/opt/suphp/sbin/suphp cmd:/opt/suphp/sbin/suphp

I say again: At approx. 8:35pm each night, server unreachable for approx. 2 minutes. Server load any other time < .8

Webhost support claims the following, in descending order, could be the cause, even though the issue occurs at a set time and not at a peak useage time for visitors:

- support claim an issue with two databases. "...due to the databases "xxxyyy" and "yyyxxx". Now I have repaired the database from my end."
There were no issues with these databases and didn't require 'fixing'. Issue still exists.
- support now inform me "I have now added the executables in csf pignore list".
Not sure how not being notified of an issue helps. Issue still exists.
- support now state they will monitor the server, findings are "We have experienced the load spike now. We found that the mysql process is consuming more resources on that time. "
- support then say "I have checked the cron log for the mentioned time and found the user "customer" was run the following cron on that time."
I have a cron set up to run every 15 minutes. I reschedule cron so would not run during 8-9pm. Issue still exists.
- support now say "The server load appears to be normal now. There were lots of Apache connections from the IP "185.25.35.x"."
- support also say "You may need to optimize the scripts with the help of your developer. Else, you may need to change the Cron intervals."
Already did the cron change, no difference. Added 185.25.35.x to deny file in csf. Issue still exists.
- support then say "The website script "xxxyyy.php" was using high process usage. "
Top Process | 17.0 - this is the most used script so logically it will show up as the top process.
- support "I have shared the slow query log's of 24 hours below. Please discuss with MYSQL developer and see if the database can be optimized."
Have made many optimizations, Issue still exists.

Given the issue only occurs at one specific time, is it possible the webhost is doing a server backup or using the server as a backup that could cause this issue, or could another customer on the 'node' or physical server be causing the issue?
Any other possible explanations?

I've looked at all crons with
cat /var/spool/cron/*

crons around the same time:
this one likely set up by webhost to look at logs
35 * * * * /usr/bin/test -x /usr/local/cpanel/bin/tail-check && /usr/local/cpanel/bin/tail-check

this one unlikely to be the cause as no load issue at any other time
5,20,35,50 * * * * /usr/local/cpanel/scripts/eximstats_spam_check 2>&1

[Sergio]

You have not said what type of server it is like if it is Dedicated or VPS/CloudSERVER.
How many processors and memory it has.
Have you checked the /var partition if it has plenty or space?.
All the above is good to know to find why the load is kind of high.

Do you have all the suit of CSF installed?

[futureH]

Ah, yes, sorry about that

You have not said what type of server it is like if it is Dedicated or VPS/CloudSERVER.

VPS

How many processors and memory it has.

Intel(R) Xeon(R) CPU E5-2430 v2 @ 2.50GHz
Total processors: 24

total used free shared buff/cache available
Mem: 2097152 831220 24900 1664 1241032 1142090
Swap: 0 0 0
Total: 2097152 831220 24900

Have you checked the /var partition if it has plenty or space?.
All the above is good to know to find why the load is kind of high.

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/vzfs 41943040 22323544 19619496 54% /
/dev/simfs 41943040 22323544 19619496 54% /tmp
/dev/simfs 41943040 22323544 19619496 54% /var/tmp
devtmpfs 1048576 0 1048576 0% /dev
tmpfs 1048576 0 1048576 0% /dev/shm
tmpfs 1048576 328 1048248 1% /run
tmpfs 1048576 0 1048576 0% /sys/fs/cgroup


Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/vzfs 734003 294242 439761 41% /
/dev/simfs 734003 294242 439761 41% /tmp
/dev/simfs 734003 294242 439761 41% /var/tmp
devtmpfs 262144 54 262090 1% /dev
tmpfs 262144 1 262143 1% /dev/shm
tmpfs 262144 250 261894 1% /run
tmpfs 262144 10 262134 1% /sys/fs/cgroup

Do you have all the suit of CSF installed?

Just csf and lfd
ConfigServer Security & Firewall - csf v13.06

[Sergio]

Does the VPS has 24 cpu cores assigned to it or just 2,?
Does the total memory RAM assigned to the VPS is 2, 3 or 4 GB?
Do you have installed ModSecurity in your VPS?

[futureH]

Does the VPS has 24 cpu cores assigned to it or just 2,?

Note: These figures are averages since 0000 hours today.
You have 24 CPUs; therefore, these CPU percentages are divided by 24 to indicate the true percentage of all CPU power used.

User Domain % CPU % MEM MySQL Processes
mysql 0.30 23.70 0.0
customer customer. com 0.02 0.11 0.4
nobody 0.01 7.62 0.0
mailman 0.00 0.00 0.0
unauthenticated 0.00 0.00 0.0
root 0.00 0.00 1.0

Does the total memory RAM assigned to the VPS is 2, 3 or 4 GB?

2GB

Do you have installed ModSecurity in your VPS?

It's on the VPS but as far as I can tell via WHM, no vendors rules have been enabled.


I did notice that an old WP blog is consuming most [60%] of the CPU

Top Processes
customer customer .com 60.0 /usr/bin/php /home/customer/public_html/wp-blog/index.php
customer customer .com 9.0 /usr/bin/php /home/customer/public_html/customer/lists.php

so I might rename that folder temporarily or see if there is a maintenance mode for WP and see if that makes any difference, but given CPU load only jumps once a day at a rather specific time, not holding my breath!

[Sergio]

Have you checked the CRONs on that specific user?
Is the WP version new or old?

[futureH]

Have you checked the CRONs on that specific user?

Mentioned in first post.

Is the WP version new or old?

Old. Removed. Issue occurred again tonight, same time as usual.