HP-UX IPFilter V18.0 Administrator Guide for HP-UX 11i v3

To enable IPSec to complete IKE negotiations, configure IPFilter to allow the IKE negotiation packets
through.
Figure 3 Scenario two
In Scenario two, IPFilter is configured to block UDP traffic on system A, you want all TCP traffic to
pass through . From system B on the network, you want all TCP traffic encrypted. System A has IP
address 10.10.10.10 and system B has IP address 15.15.15.15.
You configure IPSec on each system to encrypt packets between two systems.
When TCP traffic is initiated from A to B or from B to A, IPSec first negotiates security parameters
using the IKE protocol (UDP port 500). You must configure IPFilter on system A to pass IKE packets.
To do so, add the following rules to your configuration:
pass in quick proto UDP from 15.15.15.15 port = 500 to 10.10.10.10 port = 500
pass out quick proto UDP from 10.10.10.10 port = 500 to 15.15.15.15 port = 500
block in proto UDP
block out proto UDP
These rules allow IKE packets to pass correctly.
NOTE: You must configure IPFilter to pass traffic both in and out on UDP port 500 for IPSec to
work properly. If IPFilter is used with IPSec requiring the NAT traversal function, UDP port 4500
must be set to pass for in and out traffic.
14.3 When traffic appears to be blocked
In the following scenario there is overlap in the configurations of IPFilter and IPSec. To get this
negotiation through, you must configure IPFilter rules to let TCP traffic through.
Figure 4 Scenario three
In Scenario three, IPSec is configured to encrypt TCP traffic between system A and system B and
IPFilter is configured to block all TCP traffic with the following rules:
block in proto TCP
block out proto TCP
14.4 Allowing protocol 50 and protocol 51 traffic
IPSec uses Encapsulating Security Payload (ESP) to provide data confidentiality and Authentication
Header (AH) to provide data integrity at the IP layer. Depending on a user’s IPSec traffic policy
configuration, IPSec inserts ESP, AH, or both as protocol headers into an IP datagram that
immediately follows an IP header. The protocol field of that IP header will be 50 (ESP) or 51 (AH)
to indicate the next protocol.
86 HP-UX IPFilter and IPSec