CSF/LFD Very Slow with Alma, Repeated Error Messages

[cwbob]

I've notice AlmaLinux in general being slow, especially websites, with down detector something reporting a website down. I suspect server issues as it's always been a WordPress website with the redis and lightspeed caching modules. So much for a speed increase!

[warmwhisky]

Just wanted to drop a follow-up for anyone seeing this error:

Code: Select all

Error: Failed to detect code [xyz] in SYSLOG_LOG [/var/log/messages]

I ran into it again this year and was going mad trying to figure it out until I realised I had posted about it last year. The cause turned out to be unrelated to kernel or csf.conf settings.

In my case, I had Meilisearch running as a systemd service with this in its config:

Code: Select all

log_level = "DEBUG"

That log level was flooding journald with thousands of entries per second, which caused rsyslog (via imjournal) to rate-limit and drop messages, including the tiny test line CSF/LFD injects with logger.

You’ll see messages like this in journalctl -u rsyslog or systemctl status rsyslog:

Code: Select all

imjournal: messages lost due to rate-limiting (20000 allowed within 600 seconds)

The Fix:
Change the Meilisearch config to reduce logging:

Code: Select all

log_level = "WARN"

Then restart Meilisearch:

Code: Select all

systemctl restart meilisearch

Within a few minutes, the logging pressure eases, and /var/log/messages starts behaving again. CSF/LFD’s test messages make it through, and the errors stop.

Hope this helps someone (or future-me) avoid the same loop.