HP-UX IPSec Configuring Microsoft Windows IP Security to Operate with HP-UX IPSec HP Part Number: J4256-90025 Published: June 2007 Edition: 1.
Table of Contents About This Document.........................................................................................................9 Typographic Conventions......................................................................................................................9 Introduction..........................................................................................................................................11 Testing Environment.....................................................
glossary.............................................................................................................................
List of Figures 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 IP Security Policy Wizard..............................................................................................................16 Rules Tab.......................................................................................................................................17 Rule Properties Dialog Box...........................................................................................................
List of Tables 1 IPsec Parameters on Windows and HP-UX .................................................................................
About This Document This document describes how to configure Microsoft Windows IP Security to operate with the HP-UX IPSec product. Typographic Conventions This document uses the following typographical conventions: %, $, or # audit(5) Command Computer output Ctrl+x ENVIRONMENT VARIABLE [ERROR NAME] Key Term User input Variable [] {} ... | WARNING CAUTION IMPORTANT NOTE A percent sign represents the C shell system prompt.
Introduction This document contains the following sections: • “Windows IP Security Configuration Overview” (page 13) This section contains a brief overview of the Windows IPsec configuration parameters and the terminology used in the Windows IPsec configuration utilities. • “Configuring a Windows Host-to-Host Policy” (page 14) This section describes how to configure IP Security (IPsec) on a Windows client to secure IP packets sent to and received from an HP-UX system in a host-to-host topology.
The above problem typically occurs with ESP-encrypted UDP or ICMP packets that are fragmented by IP. HP-UX 11i systems minimize IP fragmentation of ESP-encrypted TCP packets. You may still experience problems with ESP-encrypted TCP packets sent from an HP-UX system to a Windows 2000 system if an intermediate IP gateway fragments the ESP packet. Protocol Implementation Differences HP-UX and Microsoft Windows both implement the IP Security protocol suite.
Windows IP Security Configuration Overview On Microsoft Windows systems, all IP Security (IPsec) configuration data resides in a single IP Security policy. You can create multiple IP Security policies, but only one local policy can be active on the system. If the system is a member of a Windows Active Directory domain, you can use an IP Security policy from a Group Policy defined for the domain.
— Tunnel Settings The tunnel settings specify if the rule is a tunnel rule. If it is a tunnel rule, the settings also specify the tunnel destination endpoint. — Connection Type The connection type specifies the connection (link) types for the rule, such as LAN. • General The general parameters for a policy specify IKE SA parameters, such as the IKE encryption algorithm, IKE hash (integrity algorithm), Diffie-Hellman Group, and IKE SA key lifetimes. The parameters correspond to IKE SA proposals.
Step 1: Starting the IP Security Policies Snap-in Configuration Utility Use the following procedure to start the IP Security Policies configuration utility. This utility is a snap-in module for the Microsoft Management Console (MMC). 1. 2. Start the Microsoft Management Console (MMC). From the Microsoft Start menu, click Run and type MMC. Click OK. If the IP Security Policies snap-in configuration utility is not loaded, use the following procedure to add it: a.
Figure 1 IP Security Policy Wizard 6. The Policy Wizard opens the Completing the IP Security policy wizard window. Select the Edit properties check box if it is not already selected. Click Finish. The IP Security configuration utility opens the Policy Properties dialog box. The title of the window will be name Policy, where name is the policy name. Step 3: Adding a Rule The primary purpose of a rule is to assign actions to filters. A rule also specifies IKE authentication methods.
Figure 2 Rules Tab 2. The IP Security configuration utility opens the Rule Properties dialog box, which has a tab for each category of rule configuration data: IP Filter List, Filter Action, Authentication Methods, Tunnel Setting, and Connection Type ( Figure 3).
TIP: After you have created a rule, you can open the Rules Properties dialog box by right clicking the rule and selecting Properties. Step 4: Creating the IP Filter List and Filters for the Rule An IP filter list can contain one or more filters. IPsec uses the filters to determine which rule to apply to an IP packet. The IP Security configuration utility displays the rules for a policy in reverse alphabetical order based on the name of the IP filter list for the rule.
The IP Security configuration utility opens a Filter Properties dialog box. 2. Select the Addressing tab in the Filter Properties dialog box. Use the drop-down menus to specify the address types for the source and destination addresses. The selections are: • My IP Address1 • Any IP Address • A specific DNS Name • A specific IP Address • A specific IP Subnet Enter the source and destination IP addresses or DNS names for the filter. If you selected A specific IP Subnet, enter the subnet mask.
In Figure 6, the administrator specifies protocol information for a Windows system that will be a telnet client. The protocol type is TCP, the source port is a wildcard (any port), and the destination port is the IANA registered TCP port number for the telnet service, 23.
4. From the IP Filter List dialog box, you can add another filter to the filter list by clicking the Add button. Click OK in the IP Filter List dialog box to return to the IP Filter List tab in the Rule Properties dialog box. 5. Add the filter list to the rule by selecting the option button for the filter list you just created. In Figure 7, the administrator added the filter list foo for the rule.
• • Security Methods General Select the Security Methods tab, then select Negotiate security. Verify that the following check boxes are not selected:2 • Accept unsecured communication, but always respond using IPSec. • Allow unsecured communication with non-IPSec-aware computer. In addition, verify that the Session key perfect forward secrecy (PFS) check box is not selected. (HP-UX does not support session key PFS, also referred to as PFS for keys only.
Figure 9 Security Method Dialog Box The Encryption and Integrity and Integrity only methods each correspond to a set of predefined parameters for an IPsec SA proposal, including an IPsec transform type (such as ESP). The transforms and additional SA parameters defined for these methods may vary according to the Windows release installed.
Figure 10 Custom Security Methods Settings Dialog Box c. Click OK to return to the Security Methods tab in the Filter Actions dialog box. If the parameters you configured for a custom method match a predefined method, the configuration utility will display an informative message and select the matching predefined method. 3. 4. 5. From the Security Methods tab, you can add more methods (IPsec SA proposals) by clicking the Add button.
Figure 11 Selecting the Filter Action Step 6: Configuring the IKE Authentication Method and Preshared Key for the Rule When configuring a rule to be compatible with HP-UX IPSec, the authentication method specifies the IKE authentication method (preshared key or certificates) for IPsec. The authentication method must match the value specified for the -authentication argument in the ipsec_config add ike command.
Figure 12 Configuring A Preshared Key To use IKE authentication with certificates, select Use a certificate from this certification authority (CA). Click Browse. The IP Security configuration utility opens a Select Certificate box with a list of CA certificates stored on your system. Select the CA for the appropriate CA and click OK. (For additional information about configuring Microsoft Windows certificates, see Using Microsoft Windows Certificates with HP-UX IPSec, available at http://docs.hp.com. 4. 5.
• • • • • Encryption algorithm: 3DES Hash algorithm: MD5 Diffie-Hellman Group: 2 Maximum lifetime: 28,800 seconds (8 hours) Maximum Quick Modes: 100 You can specify alternative values for the above parameters in the ipsec_config add ike command. On Windows XP systems with SP2, IP Security policies are pre-configured with four IKE SA proposals. The second IKE proposal matches the default HP-UX IPSec IKE proposal3, and will be used by the two systems if no changes are made to the default configuration data.
Figure 14 Key Exchange Settings Dialog Box Configure the fields as follows: • Master key perfect forward secrecy (PFS) Selecting this check box sets the maximum number of IPsec or Quick Mode (QM) negotiations that IKE can perform using an IKE SA to 1. It is equivalent to specifying -maxqm 1 in the ipsec_config add ike command. PFS is computationally expensive and HP recommends that you enable it only in hostile environments.
Figure 15 IKE Security Algorithms Dialog Box Use the drop-down menus to select the appropriate integrity algorithm, encryption algorithm, and Diffie-Hellman Group (these are equivalent to the -hash, -encryption, and -group arguments of the ipsec_config add ike command). 3. Click OK to return to the Key Exchange Security Methods dialog box. Click OK to return to the Key Exchange Settings dialog box. Click Close to close the Policy Properties dialog box.
Figure 16 IPSEC Services Properties Dialog Box Alternatively, you can manually start the IP Security service by entering the following Windows command: net start policyagent You can also use the following sequence of commands to manually stop and restart the IP Security service. This also clears any existing IPsec SAs,: net stop policyagent net start policyagent Step 10: Assigning the IP Security Policy The IP Security subsystem will not use the new policy until you assign (activate) it.
Figure 17 Assigning the IP Security Policy Step 11: Verifying the Configuration To verify your configuration, generate traffic that matches the address filter. On the HP-UX system, enter the following command to verify that the IKE SA and IPsec SAs are established: ipsec_report -sa Example In this example, IPsec secures telnet connections from the Windows system to the HP-UX system, using authenticated ESP. The Windows system's address is 10.1.1.1 The HP-UX system's address is 10.2.2.2.
HP-UX Configuration On the HP-UX system, the administrator configures the following policies and records: ipsec_config add host telnet_from_foo1 \ -source 10.2.2.2/32/TELNET -destination 10.1.1.1 \ -action ESP_3DES_HMAC_SHA1 ipsec_config add ike foo1 -remote 10.1.1.1 -auth PSK ipsec_config add auth foo1 -remote 10.1.1.1 \ -psk my_preshared_key If the HP-UX IPSec subsystem is not already started, the administrator starts it using the ipsec_admin -start command.
Configuring a Windows End-to-End Tunnel Policy The only IPsec tunnel topology supported between an HP-UX system and a Windows system is an end-to-end tunnel.5The procedure for configuring an end-to-end tunnel policy on Windows system is the same as procedure for configuring a host policy, except that you must configure two, non-mirrored rules: one rule for outbound packets and one rule for inbound packets, as described in the sections that follow.
TIP: The tunnel setting is used by all packets selected using the address filters for the rule. Do not include any filters for host-to-host (non-tunneled) packets in the filter list for a rule with a tunnel. 1. 2. 3. 4. Start the IP Security Policies snap-in if necessary. Create an IP Security policy or modify an existing policy. To modify an existing policy, select the policy in the right navigation pane and right click the policy. Select Properties.
Figure 18 Outbound Rule Filter Figure 19 Outbound Rule Tunnel Settings Inbound Rule The inbound rule is for packets to the Windows system (destination address 10.1.1.1) from the HP-UX system (source address 10.2.2.2).
Figure 20 Inbound Rule Filter Figure 21 Inbound Rule Tunnel Settings Additional Parameters You must configure the remaining rule parameters (filter action, authentication methods, and connection type) to be compatible with the HP-UX configuration. In addition, the general parameters for the rule (the IKE SA parameters) must be compatible with the HP-UX configuration.
HP-UX Configuration On the HP-UX system, the host and tunnel policies are bi-directional (mirrored), so you configure only one host policy and only one tunnel policy. Since this is an end-to-end tunnel, the tunnel policy does not have to specify the tunnel endpoints. HP-UX IPSec will use the end source and end destination addresses as the tunnel addresses (the tsource and tdestination values default to the source and destination values). ipsec_config add host foo1 -source 10.2.2.2 \ -destination 10.1.1.
Troubleshooting Tips Most interoperability problems occur during IKE negotiations, so examining IKE log events is useful. You can use the following procedures to enable and view IKE log events: Using IKE Logging on HP-UX Systems Use the following procedure to view detailed IKE log events on HP-UX systems: 1.
5. Disable IKE logging. On Windows XP systems, set the HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\PolicyAgent\Oakley\EnableLogging REG_DWORD value to 0. On Windows 2003 systems, enter the following command: netsh ipsec dynamic set config ikelogging 0 6. Stop and restart the IP Security service.
Comparing HP-UX and Windows IPsec Configuration Parameters This section contains Table 1, which compares how HP-UX and Windows systems configure and store IPsec parameters.
Table 1 IPsec Parameters on Windows and HP-UX (continued) Parameter Windows Configuration HP-UX Configuration IKE Preshared Key Specify it in the Specify it using the Authentication Methods for -preshared argument of a rule. the ipsec_config add auth command. IKE Exchange Type Windows supports only Main Mode exchanges. Notes Specify it using the -exchange argument of the ipsec_config add auth command. The default value is MM (Main Mode).
The filter matches packets with the following addresses: Source address: 10.1.1.1 Destination address: 10.2.2.2 If the filter is mirrored, it also matches packets with the following addresses: Source address: 10.2.2.2 Destination address: 10.1.1.1 The mirror setting only affects Windows IP Security behavior before IPsec SAs are established.
proposed value sent by the remote system if it is within the range specified by the IPsec protocol suite. Windows IKE SA Lifetime Values By default, Windows XP systems use the following values for preferred IKE key lifetime values: 480 minutes (eight hours) 0 (infinite) IPsec SA negotiations (sessions) In testing with HP-UX IPSec, HP configured a shorter IKE SA lifetime value on the Windows system. When the Windows system was the initiator, it sent the configured lifetime value to the remote system.
If the HP-UX system initiates IPsec SA negotiations, the HP-UX IKE daemon proposes the preferred lifetime values to the remote system. The remote system may process these values in any manner according to the IPsec protocol suite.
Related Publications The following documents are available at http://docs.hp.com: • • • HP-UX IPSec Administrator's Guide Using Microsoft Windows Certificates with HP-UX IPSec HP-UX IPSec manpages The following documents are available at http://microsoft.
glossary 3DES Triple Data Encryption Standard. A symmetric key block encryption algorithm that encrypts data three times, using a different 56-bit key each time (168 bits are used for keys). 3DES is suitable for bulk data encryption. AES Advanced Encryption Standard. Uses a symmetric key block encryption. HP-UX IPSec supports AES with a 128-bit key. AES is suitable for encrypting large amounts of data.
transform 48 glossary A transform defines the IPsec action(s) to be taken on the IP data, such as passing the data in clear text, discarding the data, authenticating and encrypting the data using ESP, or authenticating the data using AH.
