Csf cluster ping not working

[titans]

Hi,

We have a master server and many slave servers which together form a csf cluster. When we try to use the option cluster ping, we are not getting reply from few servers. Please see the details below.

csf --cping
---
Sent request to 213.xxxxx, no reply
Sent request to 213.xxxxxx, no reply
Sent request to 213.xxxxx, no reply
Sent request to 213.xxxxxxx, no reply
Sent request to 213.xxxxxxx, no reply
Sent request to 198.xxxxxxx, no reply
Sent request to 5.xxxxxxx, replied: [PONG!]
Sent request to 213.xxxxxx, replied: [PONG!]
Sent request to 213.xxxxxx, replied: [PONG!]
Sent request to 213.xxxxxxx, no reply
Sent request to 213.xxxxxxxxx, no reply
Sent request to 213.xxxxxxx, no reply
Sent request to 213.xxxxxx, replied: [PONG!]
---
Prompt response on this matter is highly appreciated.

[ForumAdmin]

The main reasons for it not working would be:

• Not all of the servers are running csf v9.29

• The servers responses are timing out (4 seconds)

• Issues with the cluster setup

You should start by checking the remote servers lfd.log just after running the ping to see if there are any messages.

[titans]

Hello,
Thank you for the reply
The traffic flows ok between master and all other slave servers, but the responding "PONG!" is never sent from the replying server (the ones with "no reply"). Please let us know is it caused due to any update with software or any issues in CSF, it was not like this before.
Kindly assist.

[marcele]

Did you read his reply ? Make sure to update all the slave servers to csf v9.29. They are probably on older versions.

[titans]

Yes, all installed csf versions are v9.29. Actually the issues started to happen after this latest update. Before applying the update, it was working fine.

As has already been mentioned, when a cluster PING command is sent from one server to all servers in the cluster, some servers that answer (respondend: PONG) and some servers that do not (no reply).

The traffic flows ok between sender and all other servers, but the responding "PONG!" is never sent from the replying server (the ones with "no reply").

[titans]

Hello,

Any updates? Actually the clustering setup is correct and able to set the rules. But the issue is some of slave servers is not replying for the requests. Please see the result from my tests.

CSF Rule sent from Slave:

root@cpanel [~]# csf -cd xxx.xxx.xxx.xxx PHP Hack
Sent request to xxx.xxx.xxx.xx1, replied: [Received]
Sent request to xxx.xxx.xxx.xx2, no reply
Sent request to xxx.xxx.xxx.xx3, replied: [Received]

root@xxx.xxx.xxx.xx1 [~]# csf -g xxx.xxx.xxx.xxx
DENYIN 134 0 0 DROP all -- !lo * xxx.xxx.xxx.xxx 0.0.0.0/0
DENYOUT 120 0 0 DROP all -- * !lo 0.0.0.0/0 xxx.xxx.xxx.xxx

root@xxx.xxx.xxx.xx2 [~]# csf -g xxx.xxx.xxx.xxx
DENYIN 134 0 0 DROP all -- !lo * xxx.xxx.xxx.xxx 0.0.0.0/0
DENYOUT 120 0 0 DROP all -- * !lo 0.0.0.0/0 xxx.xxx.xxx.xxx

Please let me know why some servers not replying back even its applying the rule correctly.

[ForumAdmin]

As requested before - You need to post whether the non-replying cluster members actually get the requests (from those servers lfd.log) so we know whether it is responding that there is an issue with or whether it is not actually receiving the instruction in the first place.

[titans]

Hello,

Yes, the non-replying cluster members are receiving the responses from master server. See the lfd.log snippet below.

+++++++++++++
Jan 28 16:22:15 slave1 lfd[426243]: Cluster member xxx.xxx.xxx.xxx said restart csf and lfd
Jan 28 16:22:15 slave1 lfd[426243]: Cluster - csf restarting...
Jan 28 16:22:31 slave1 lfd[426243]: Cluster - lfd restarting...
Jan 28 16:22:32 slave1 lfd[428704]: Cluster Service starting...
Jan 28 16:47:29 slave1 lfd[456283]: Cluster member xxx.xxx.xxx.xxx said PING!
+++++++++++++

But one strange thing I noticed here is in the replying servers the CLUSTER_SENDTO and CLUSTER_RECVFROM sections in the csf.conf are not listing the entire server IPs. But the nonreplying servers have the entire lists in it.

[ForumAdmin]

Thank you, that was the information we needed. We've found an unforeseen issue with the encrypted cluster traffic which was almost certainly the reason for this. We will shortly release v9.30 which you will need to have an all cluster members which should hopefully help with this.

[ForumAdmin]

v9.30 has been released:
https://blog.configserver.com/

Note: There are still circumstances when you may not get a reply in time if the response takes longer than the wait interval (5 seconds in v9.30). This is expected and needed to prevent client/server hanging.