HP-UX IPFilter Version 15.01 Administrator's Guide

Before exchanging IPSec-encrypted or authenticated packets, IPSec negotiates security parameters
using the Internet Key Exchange (IKE) protocol. The IKE protocol exchanges messages using
UDP protocol port 500, or port 4500 if IPSec NAT traversal is used.
If the IPFilter configuration is so broad that it is blocking all UDP traffic, then IPSec cannot
complete IKE negotiations. When an IKE negotiation is not completed, the encrypted packets
are not received. If this happens, the IPSec log on the initiating side will show the error MM
negotiation timeout or Phase 1 negotiation timeout.
To let IPSec complete IKE negotiations, configure IPFilter to let the IKE negotiation packets
through.
Figure 13-3 Scenario Two
IPSec <---------------> TCP <-----------------> IPSec
A
B
10.10.10.10
15.15.15.15
IPFilter
-----UDP-----
In Scenario Two, IPFilter is configured to block UDP traffic on system A, you want all TCP traffic
to pass through . From system B on the network, you want all TCP traffic encrypted. System A
has IP address 10.10.10.10 and system B has IP address 15.15.15.15.
You configure IPSec on each system to encrypt packets between two systems.
When TCP traffic is initiated from A to B or from B to A, IPSec first negotiates security parameters
using the IKE protocol (UDP port 500). You must configure IPFilter on system A to let IKE packets
through. To do so, add the following rules to your configuration:
pass in quick proto UDP from 15.15.15.15 port = 500 to 10.10.10.10 port = 500
pass out quick proto UDP from 10.10.10.10 port = 500 to 15.15.15.15 port = 500
block in proto UDP
block out proto UDP
These rules allow IKE packets to pass correctly.
NOTE: You must configure IPFilter to pass traffic both in and out on UDP port 500 for IPSec
to work properly. If IPFilter is used with IPSec requiring the NAT traversal function, UDP port
4500 must be set to pass for in and out traffic.
When Traffic Appears to Be Blocked
In the following scenario there is overlap in the configurations of IPFilter and IPSec. To get this
negotiation through, you must configure IPFilter rules to let TCP traffic through.
Figure 13-4 Scenario Three
IPSec <---------------> TCP <-----------------> IPSec
A
B
10.10.10.10
15.15.15.15
IPFilter
---TCP-----
In Scenario Three, IPSec is configured to encrypt TCP traffic between system A and system B
and IPFilter is configured to block all TCP traffic with the following rules:
block in proto TCP
block out proto TCP
112 HP-UX IPFilter and IPSec