HP-UX IPSec version A.02.00 Administrator's Guide

Troubleshooting HP-UX IPSec
IPSec Operation
Chapter 5150
On an end system (the local system is the source for the outbound
packet), the Policy Manager sequentially searches the host IPSec
policies in priority order for the first policy with an IP packet filter
that matches the packet. If no match is found, HP-UX IPSec uses the
default host IPSec policy.
On a gateway system (the local system is forwarding the outbound
packet), the Policy Manager sequentially searches the gateway IPSec
policies in priority order. If no match is found, HP-UX IPSec uses the
default gateway IPSec policy.
If the transform (action) specified in the matching IPSec policy is to
encrypt or authenticate the IP packets using AH or ESP, IPSec SAs
may already exist for the policy. The new packet can use the existing
IPSec/QM SAs if the IP addresses, ports and protocols match. The
new packet can also use the existing IPSec/QM SAs if both IP
addresses match and host-based keying is enabled (the Exclusive bit
is not set for the policy). Otherwise, new IPSec/QM SAs are
established.
3. Establish an ISAKMP/MM SA
Before establishing an IPSec/QM SA, the IKE daemon must
establish an ISAKMP/MM SA with the remote system, if none exists.
To establish an ISAKMP/MM SA, the Policy Manager daemon is
queried for an IKE policy, based on the remote system’s IP address.
As part of the ISAKMP/MM SA negotiations, each system
establishes a Diffie-Hellman value using the Oakley group specified
in the IKE policy. Because the Diffie-Hellman algorithm is
vulnerable to third-party attacks, each system must also
authenticate the identity of the other system using the primary
authentication method specified in the IKE policy (preshared keys or
RSA signatures with security certificates), and verify the remote
system’s ID type and ID value (HP-UX IPSec uses IP addresses by
default). The two systems also negotiate the authentication and
encryption algorithms for the ISAKMP/MM SA, and SA lifetime
according to values specified in the IKE policies.
4. Establish IPSec/QM SAs
Once the ISAKMP/MM SA is established, the IKE daemon uses the
secured channel to establish IPSec/QM SAs with its peers. Two
IPSec/QM SAs are established: one for packets from the local system
to the remote system, and one for packets from the remote system to
the local system.