Troubleshooting guide
Chapter 7 Troubleshooting Security Servers and Content Security
How to Improve HTTP Security Server performance in a High Performance Environment
Advanced Technical Reference Guide 4.1 • June 2000 58
Excessive Log Grace period to 30 sec (See the SecureKnowledge Solution in the Check Point Technical
Services site (ID 110022.0.1679268.2471760)), and then re-installed the policy.
9. Test ended approx. 1:15 p.m., and after change no. 8, it appeared that there were no more log buffer
messages on the console. Number of connections at this time dropped to less than 10,000.
10. Note that many drops are logged and most appear to be return packets from web servers. These packets will
continue for up to 10 min (default) as web server is still trying to close connection.
11. All fw and httpss processes are stable through the entire test, (no processes hanging, no core dumps etc.)
.
Discussion
Re: observation no.4 and no. 7:
It was deemed necessary by the test team to increase the resources required for this environment. With the CPU
at 99% during peak and sometime at 100%, there appeared to be no room for higher loads. Also, with the high
number of “Cannot connect to www server”entriesintheahttpd.elg log file, it was determined
that the box was out of resources periodically, even though this message could appear for other reasons
including servers that does not respond, etc.
Re: observation no. 8:
After reducing the Excessive Log Grace Period to 30 sec, (half the default of 60 sec.), the messages to the
console “Log buffer message queue full” stopped. This message occurs when the kernel
process responsible for the logging is filling the buffer for the log messages faster than the user mode process
can empty this buffer This is a normal message identifying the potential loss of important log messages. A
better remedy is a faster CPU and /or to increase the size of the log queue, which is a system parameter. The
latter may in some cases not resolve the problem.
Re: observation no. 10:
These are mostly late packets from the web server(s) as determined by a network sniffer. Because the
connection has already been removed from the connections table, (i.e. client browser has already closed or reset
its connection to the transparent proxy) these packets are dropped by the clean-up rule. Possible remedy is to
increase :tcpendtimeout from 50 sec to some higher value. This will allow the connection to stay in the
connection table longer and therefore allow the packets to get through and therefore get ack'd and the
connection to be closed in an orderly fashion. This has a negative side effect of drastically increasing the size of
the connections table.
Another solution is to add a rule to filter these return packets, from any, source port 80, to the external IP
address of VPN-1/FireWall-1 on port gt. 1023, reject, no track. This will at least eliminate these from the log
viewer. This was determined to be the preferred corrective action for this environment.
Another solution is to make a code change to enable Check Point gateways to drop non-first TCP packets
instead of matching the rule base. It should be noted that this INSPECT fix will cause a change of behavior
from the existing Check Point gateway behavior in the following way. Following a reboot, policy unload or
stopping the FireWall, all active TCP connections will be blocked, and any timed-out TCP connections (i.e.,
connections that have been inactive longer than the TCP timeout) will be disconnected. The ability of
VPN-1/FireWall-1 to maintain connections after policy reload will not be affected by this change.
For the changes, see http://www.checkpoint.com/techsupport/alerts/ackdos_update.html
Once these connections have been removed from the connections table, these packets will be dropped by rule 0
– so this might explain these kind of log messages.
Action Plan
Following this test, the following actions were taken. They are presented for the purpose of illustration, and may
be a useful guide for your own environment.
1. Obtained an E450, 4 CPU machine with a total of 1 GB of RAM.