Host Intrusion Detection System Administrator's Guide Release 3.1
Templates and Alerts
Buffer Overflow Template
Appendix A
136
Buffer Overflow Template
The vulnerability
addressed by this
template
All buffer overflow attacks (for example, stack smashing, return-into-libc, execute on
heap) attempt to overflow a buffer. The buffer can be a local variable residing on the
stack, a dynamically allocated buffer residing on the heap, or a global variable residing
in the process data segment. Refer to the white paper Stack Buffer Overflow Protection
in HP-UX 11i, available at http://www.docs.hp.com, for a description of buffer
overflow attacks on HP-UX. Unusually long program arguments are carefully modified
by an attacker to overflow a buffer for which the program does not perform bounds
checking. By overflowing the buffer, an attacker can modify the program’s execution flow
to execute malicious code and thereby hijack a privileged program. Modifying a
program’s execution flow can be accomplished in several ways, including the following:
• Overflowing a buffer on the stack to modify the return address in an activation
record.
• Overflowing a buffer on the heap to modify a free memory header so that the heap
memory allocation code subsequently overwrites a function’s return address.
• Overflowing a buffer in the data segment in order to overwrite an adjacent variable
containing a function pointer so that a subsequent dereferencing of the variable
results in the execution of malicious code.
How this template
addresses the
vulnerability
The Buffer Overflow (BO) template monitors attack patterns that are indicative of
various types of buffer overflow attacks and reports execute-on-stack buffer overflow
attacks detected by the HP-UX kernel (starting with HP-UX 11i). The template monitors
privileged setuid programs where the effective user ID (euid) is not equal to the real
user ID (ruid) and the euid is one of the user IDS specified in the template’s property
list of privileged users (for example, root).
Specifically, the template monitors privileged setuid programs for the following:
• The privileged setuid program was invoked with an unusually long program
argument.
• The privileged setuid program was invoked with program arguments that contain
nonprintable characters (for example, could be CPU opcodes).
The template also reports when the kernel detects that a program has attempted to
execute on its stack (perhaps as part of a stack buffer overflow attack).
NOTE In HP-UX 11i v1 and later, comprehensive stack buffer overflow protection, which uses a
combination of highly efficient software and existing memory management hardware,
protects against both known and unknown buffer overflow attacks without sacrificing
system performance. This protection is managed with the executable_stack tunable
kernel parameter. You can allow selected programs to execute from the stack by marking
them with the -es option of the chatr command. Refer to the executable_stack (5) and
chatr (1) manpages and the Stack Buffer Overflow Protection in HP-UX 11i white paper,
available at http://docs.hp.com.