HP-UX IPFilter V17.05 Administrator Guide HP-UX 11i v2 and HP-UX 11i v3

14 HP-UX IPFilter and IPSec
14.1 IPFilter and IPSec basics
IPSec and IPFilter will not panic or corrupt each other. However, there are situations in which one
product might block traffic for the other. The following figure shows the positions of IPFilter and
IPSec in the network stack:
Figure 1 IPFilter and IPSec
IPFilter, which is below IPSec in the networking stack, filters network packets before they reach
IPSec. You can have both IPFilter and IPSec configured and running on a system without them
negatively affecting each other.
Figure 2 Scenario one
In the Scenario one, you have IPFilter and IPSec on system A with IPFilter blocking packets from
system B and IPSec encrypting packets from system C. When a packet arrives at system A, IPFilter
checks to see if it is from system B, and, if so, blocks the packet. If not, the packet continues up
the stack to IPSec. IPSec checks to see if it is from system C. If so, the packet arrives encrypted.
No overlap is in the configurations of IPFilter and IPSec in this network topology, so there are no
conflicts in Scenario One.
CAUTION: HP-UX IPSec does not support NAT traversal. If you are using HP-UX IPFilter with
HP-UX IPSec, do not use NAT functionality. However, it is possible that IPFilter and NAT can be
used in network configurations containing other vendors’ IPSec products that do support NAT
traversal.
14.2 IPSec UDP negotiation
You can configure IPSec and IPFilter so that there is some overlap in the configurations. However,
you must be sure the overlapping configurations do not block each other.
Before exchanging IPSec-encrypted or authenticated packets, IPSec negotiates security parameters
using the Internet Key Exchange (IKE) protocol. The IKE protocol exchanges messages using UDP
protocol port 500, or port 4500 if IPSec NAT traversal is used.
If the IPFilter configuration is so broad that it blocks all UDP traffic, IPSec cannot complete IKE
negotiations and packets that are configured to be secured by IPSec are dropped. The IPSec log
on the initiating side will show the error MM negotiation timeout or Phase 1 negotiation
timeout.
84 HP-UX IPFilter and IPSec