HP-UX IPFilter Version 17 Administrator's Guide

Before exchanging IPSec-encrypted or authenticated packets, IPSec negotiates security parameters
using the Internet Key Exchange (IKE) protocol. The IKE protocol exchanges messages using
UDP protocol port 500, or port 4500 if IPSec NAT traversal is used.
If the IPFilter configuration is so broad that it blocks all UDP traffic, IPSec cannot complete IKE
negotiations and packets that are configured to be secured by IPSec are dropped. The IPSec log
on the initiating side will show the error MM negotiation timeout or Phase 1
negotiation timeout.
To enable IPSec to complete IKE negotiations, configure IPFilter to allow the IKE negotiation
packets through.
Figure 14-3 Scenario Two
IPSec <---------------> TCP <-----------------> IPSec
A
B
10.10.10.10
15.15.15.15
IPFilter
-----UDP-----
In Scenario Two, IPFilter is configured to block UDP traffic on system A, you want all TCP traffic
to pass through . From system B on the network, you want all TCP traffic encrypted. System A
has IP address 10.10.10.10 and system B has IP address 15.15.15.15.
You configure IPSec on each system to encrypt packets between two systems.
When TCP traffic is initiated from A to B or from B to A, IPSec first negotiates security parameters
using the IKE protocol (UDP port 500). You must configure IPFilter on system A to pass IKE
packets. To do so, add the following rules to your configuration:
pass in quick proto UDP from 15.15.15.15 port = 500 to 10.10.10.10 port = 500
pass out quick proto UDP from 10.10.10.10 port = 500 to 15.15.15.15 port = 500
block in proto UDP
block out proto UDP
These rules allow IKE packets to pass correctly.
NOTE: You must configure IPFilter to pass traffic both in and out on UDP port 500 for IPSec
to work properly. If IPFilter is used with IPSec requiring the NAT traversal function, UDP port
4500 must be set to pass for in and out traffic.
14.3 When Traffic Appears to Be Blocked
In the following scenario there is overlap in the configurations of IPFilter and IPSec. To get this
negotiation through, you must configure IPFilter rules to let TCP traffic through.
Figure 14-4 Scenario Three
IPSec <---------------> TCP <-----------------> IPSec
A
B
10.10.10.10
15.15.15.15
IPFilter
---TCP-----
In Scenario Three, IPSec is configured to encrypt TCP traffic between system A and system B
and IPFilter is configured to block all TCP traffic with the following rules:
block in proto TCP
block out proto TCP
118 HP-UX IPFilter and IPSec