LFD causing kernel panic

[inertz]

Hi,
Refer to above matter,
Recently our server have kernel panic.
We notice it is related with LFD.

Code: Select all

CPU: 28 PID: 12607 Comm: lfd - (child) c Kdump: loaded Not tainted 3.10.0-1160.62.1.el7.x86_64 #1
[21600.068683] RIP  [<ffffffffb91a6549>] __list_del_entry+0x29/0xd0
[21600.068186] RAX: 0000000000000000 RBX: ffff8a929d9712a0 RCX: dead000000000200


      MEMORY: 64 GB
       PANIC: "BUG: unable to handle kernel NULL pointer dereference at           (null)"
         PID: 12607
     COMMAND: "lfd - (child) c"
        TASK: ffff8a9a9e420000  [THREAD_INFO: ffff8a9a78b84000]
         CPU: 28
       STATE: TASK_RUNNING (PANIC)

Crash kernel debug;

Code: Select all

crash> files
PID: 12607  TASK: ffff8a9a9e420000  CPU: 28  COMMAND: "lfd - (child) c"
ROOT: /    CWD: /etc/csf
 FD       FILE            DENTRY           INODE       TYPE PATH
  0 ffff8a9a9d1fb400 ffff8a932c80c240 ffff8a9a9f3e8850 CHR  /dev/null
  1 ffff8a9a9d1fa700 ffff8a932c80c240 ffff8a9a9f3e8850 CHR  /dev/null
  2 ffff8a9a9d1fb500 ffff8a932c80c240 ffff8a9a9f3e8850 CHR  /dev/null
  3 ffff8a9a9cf0a400 ffff8a929a073200 ffff8a9a9cd42038 REG  /run/lfd.pid
  4 ffff8a9a9d870200 ffff8a929a39ce40 ffff8a9a7bc5d0d0 REG  /var/log/maillog
  5 ffff8a9a9d871c00 ffff8a9a9cb723c0 ffff8a9a908084d0 REG  /var/log/messages
  6 ffff8a9a9d871f00 ffff8a8bbfb078c0 ffff8a9a9b6910d0 REG  /var/log/secure
  7 ffff8a9a9d871b00 ffff8a9a90a4a600 ffff8a9a90a59cd0 REG  /var/log/dovecot-info.log
  8 ffff8a9a9d870300 ffff8a9a7bde9740 ffff8a9a7bf76cd0 REG  /usr/local/apache/logs/error_log
  9 ffff8a9a9d871400 ffff8a9a90942cc0 ffff8a9a909538d0 REG  /var/log/cwp_client_login.log
 10 ffff8a9a78185f00 ffff8a9a90a48300 ffff8a9a90a464d0 REG  /var/lib/csf/lock/CT_INTERVAL.lock
 11 ffff8a9a78185700 ffff8a983bbbefc0 ffff8a9540390ad0 REG  /proc/12607/net/udp

crash> bt
PID: 12607  TASK: ffff8a9a9e420000  CPU: 28  COMMAND: "lfd - (child) c"
 #0 [ffff8a9a78b878c8] machine_kexec at ffffffffb8e662f4
 #1 [ffff8a9a78b87928] __crash_kexec at ffffffffb8f22b82
 #2 [ffff8a9a78b879f8] crash_kexec at ffffffffb8f22c70
 #3 [ffff8a9a78b87a10] oops_end at ffffffffb9591798
 #4 [ffff8a9a78b87a38] no_context at ffffffffb8e75d14
 #5 [ffff8a9a78b87a88] __bad_area_nosemaphore at ffffffffb8e75fe2
 #6 [ffff8a9a78b87ad8] bad_area at ffffffffb957fa71
 #7 [ffff8a9a78b87b00] __do_page_fault at ffffffffb95948b7
 #8 [ffff8a9a78b87b70] do_page_fault at ffffffffb9594975
 #9 [ffff8a9a78b87ba0] page_fault at ffffffffb9590778
    [exception RIP: __list_del_entry+41]
    RIP: ffffffffb91a6549  RSP: ffff8a9a78b87c50  RFLAGS: 00010207
    RAX: 0000000000000000  RBX: ffff8a929d9712a0  RCX: dead000000000200
    RDX: ffff8a929d971700  RSI: 0000000000000000  RDI: ffff8a929d9712a0
    RBP: ffff8a9a78b87c50   R8: 0000000000000001   R9: ffffffffc05fd16d
    R10: ffff8a8bbfc03e00  R11: fffff4fda1766900  R12: ffff8a9a78185700
    R13: 0000000000000000  R14: ffff8a9a78b87f18  R15: ffff8a9a9e02a900
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
#10 [ffff8a9a78b87c58] list_del at ffffffffb91a65fd
#11 [ffff8a9a78b87e48] seq_read at ffffffffb9076f93
#12 [ffff8a9a78b87eb8] proc_reg_read at ffffffffb90c75c0
#13 [ffff8a9a78b87ed8] vfs_read at ffffffffb904e2ff
#14 [ffff8a9a78b87f08] sys_read at ffffffffb904f145
#15 [ffff8a9a78b87f50] system_call_fastpath at ffffffffb9599f92
    RIP: 00007fb6752d6740  RSP: 00007ffc0544a208  RFLAGS: 00010202
    RAX: 0000000000000000  RBX: 0000000000000000  RCX: ffffffffb9599ed5
    RDX: 0000000000002000  RSI: 0000000002a1a190  RDI: 000000000000000b
    RBP: 0000000000002000   R8: 0000000000000000   R9: 0000000000002000
    R10: 0000000002a1a190  R11: 0000000000000246  R12: 0000000002a1a190
    R13: 000000000000000b  R14: 0000000001303010  R15: 00000000029a8a30
    ORIG_RAX: 0000000000000000  CS: 0033  SS: 002b
crash>

[inertz]

It happens again this early morning.
Reverting to v14.15 and will try to update.

[inertz]

Reverting the older version not causing panic. No reboot since last reboot.

root pts/1 xxx.xxx.xxx.2 Tue May 10 11:34 still logged in
root pts/0 xxx.xxx.xxx.2 Thu Apr 28 16:31 - 16:32 (00:00)
root pts/0 xxx.xxx.xxx.2 Thu Apr 28 16:18 - 16:18 (00:00)
root pts/0 xxx.xxx.xxx.2 Wed Apr 27 14:48 - 14:49 (00:00)
root pts/0 xxx.xxx.xxx.2 Tue Apr 26 10:16 - 10:16 (00:00)
root pts/0 xxx.xxx.xxx.2 Mon Apr 25 09:52 - 09:53 (00:00)
reboot system boot 3.10.0-1160.62.1 Sun Apr 24 16:50 - 11:34 (15+18:43)
reboot system boot 3.10.0-1160.62.1 Sat Apr 23 17:08 - 11:34 (16+18:25)

[sahostking]

yea on all our servers I am noticing lfd keeps crashing and appears on our monitoring. We keep having to start it on random servers every few hours.

Must be new update causing this as it has been running for years fine