Page 1 of 1
Global ignore list seems to cause high CPU usage
Posted: 02 Jul 2012, 18:49
by Domineaux
Hello Chirpy,
I have recently been seeing frequent high CPU usage reported for LFD processes that appears to occur during the retrieval of the global ignore list we use which contains around 100 specific IP addresses and 15 class C blocks like 123.123.123.0/24 .
Is there any reason that you know of for this to be contributing excessively to high CPU usage?
ps. the master global ignore list does have comments for each IP or range in the following format:
# IP ranges for monitoring systems
123.123.122.1 #Monitor one
123.123.123.0/24 #Monitoring network one
Re: Global ignore list seems to cause high CPU usage
Posted: 02 Jul 2012, 19:36
by Doobla
The problem is that LFD uses upwards of 100% cpu for up to 3 seconds when checking the (locally cached) global ignore list. Since this is done continually in LFD then LFD becomes a cpu hog. The loop which is responsible for the "continual" part begins with
Code: Select all
while (1) {
$0 = "lfd - processing";
The actual code block that is generating all of the cpu usage is
Code: Select all
if (-e "/etc/csf/csf.gignore") {
open (IN, "</etc/csf/csf.gignore") or &cleanup(__LINE__,$!);
flock (IN, LOCK_SH);
my @ignore = <IN>;
close (IN);
chomp @ignore;
foreach my $line (@ignore) {
if ($line =~ /^(\#|\n|\r|\s)/ or $line eq "") {next}
my ($first,undef) = split(/\s/,$line);
my ($ip,$iscidr) = split(/\//,$first);
if (&checkip($first)) {
if ($iscidr) {push @cidrs,$first} else {$ignoreips{$ip} = 1}
}
elsif ($ip ne "127.0.0.1") {&logfile("Invalid CIDR in csf.gignore: [$first]")}
}
foreach my $entry (@cidrs) {
if (&checkip($entry) == 6) {
eval {$cidr6->add($entry)};
} else {
eval {$cidr->add($entry)};
}
if ($@) {&logfile("Invalid CIDR in csf.gignore: $entry")}
}
}
Oddly, after restarting LFD and allowing it to complete its initial sanity/security checks against itself, the cpu usage is low. It only increases gradually over time.
I've tried changing the content of the /etc/csf/csf.gignore file even to the point of emptying it completely and the high cpu usage still occurs. The only way to address this is to set GLOBAL_IGNORE = "" in /etc/csf/csf.conf and then remove the /etc/csf/csf.gignore file completely. Once that is done and csf+lfd are restarted, then cpu usage stays sane.
We're seeing this on a variety of server types from the single core lowly xen vps servers to high end 16 core dells. LFD, when acting problematic, will simply use a full core for 2-3 seconds at a time and then since LF_PARSE defaults to 5 seconds then you have an almost continual cpu hog.
Re: Global ignore list seems to cause high CPU usage
Posted: 02 Jul 2012, 21:38
by ForumAdmin
If you modify that code to read:
Code: Select all
if (-e "/etc/csf/csf.gignore") {
open (IN, "</etc/csf/csf.gignore") or &cleanup(__LINE__,$!);
flock (IN, LOCK_SH);
my @ignore = <IN>;
close (IN);
chomp @ignore;
foreach my $line (@ignore) {
if ($line =~ /^(\#|\n|\r|\s)/ or $line eq "") {next}
my ($first,undef) = split(/\s/,$line);
my ($ip,$iscidr) = split(/\//,$first);
if (&checkip($first)) {
if ($iscidr) {push @cidrs,$first} else {$ignoreips{$ip} = 1}
}
elsif ($ip ne "127.0.0.1") {&logfile("Invalid CIDR in csf.gignore: [$first]")}
}
foreach my $entry (@cidrs) {
if (&checkip($entry) == 6) {
eval {$cidr6->add($entry)};
} else {
eval {$cidr->add($entry)};
}
if ($@) {&logfile("Invalid CIDR in csf.gignore: $entry")}
}
unlink "/etc/csf/csf.gignore";
}
Then restart lfd it should resolve the issue. We'll look into it in more depth if anything else is needed and then release a new version.
Re: Global ignore list seems to cause high CPU usage
Posted: 02 Jul 2012, 21:59
by Domineaux
Thanks that should help as a quick and dirty improvement.
Hope you can get a more refined way of handling into a future release.
Re: Global ignore list seems to cause high CPU usage
Posted: 02 Jul 2012, 22:11
by Doobla
The problem I can see with that solution is that if lfd is restarted then it will lose any knowledge of our global ignore list. Or am I misreading the code?
Thanks for the quick reply. If I can get verification on the above then I'll work on implementing this tonight.
Re: Global ignore list seems to cause high CPU usage
Posted: 02 Jul 2012, 22:38
by ForumAdmin
No, that file should be deleted. It is created when the latest GLOBAL_IGNORE file is downloaded which happens when lfd restarts and every LF_GLOBAL seconds. This triggers lfd to reload the list of ignored entries that were just downloaded. We should have a new release out in a few minutes that resolves this issue and actually improves the load time.
Re: Global ignore list seems to cause high CPU usage
Posted: 02 Jul 2012, 22:42
by Doobla
Perfect. Appreciate the quick turnaround.
Re: Global ignore list seems to cause high CPU usage
Posted: 02 Jul 2012, 22:53
by ForumAdmin
v5.59 has been released to resolve this issue. Thank you for reporting the issue and in particular Doobla for the detailed response.
Re: Global ignore list seems to cause high CPU usage
Posted: 02 Jul 2012, 23:11
by Doobla
Awesome. Thank you.
Re: Global ignore list seems to cause high CPU usage
Posted: 02 Jul 2012, 23:43
by Domineaux
Thanks for the quick resolution Chirpy.