Reference Architecture: Consolidating Oracle Databases with Secure Resource Partitions in a Serviceguard Cluster Whitepaper

ManualsBrandsHP ManualsSoftwareHP-UX Containers (SRP)

Reference Architecture: Consolidating

Oracle Databases with Secure Resource

Partitions in a Serviceguard Cluster

Workload Consolidation with SRP

Technical white paper

Table of contents

Executive summary .......................................................................................................................... 3

Benefits of database consolidation ................................................................................................. 3

Introduction to Secure Resource Partitions ............................................................................................ 4

Conventions used in the document ..................................................................................................... 4

Oracle deployment choices .............................................................................................................. 4

General considerations ................................................................................................................ 5

Start/stop .................................................................................................................................. 5

CPU entitlements ......................................................................................................................... 5

Software installation .................................................................................................................... 5

Database storage ........................................................................................................................ 6

Software used ............................................................................................................................. 7

Step 1: Creating the SRP .................................................................................................................. 7

Task 1: Preparing the system for SRP .............................................................................................. 8

Task 2: Adding a new users group ............................................................................................... 11

Task 3: Creating a base SRP ....................................................................................................... 11

Task 4: Applying the sshd template .............................................................................................. 12

Task 5: Applying the custom template ........................................................................................... 12

Step 2: Preparing the system ........................................................................................................... 13

Task 1: Creating the Oracle user ................................................................................................. 13

Task 2: Creating the Oracle file systems ........................................................................................ 14

Task 3: Modifying compartment rules............................................................................................ 14

Task 4: Starting the SRP .............................................................................................................. 14

Task 5: Logging in to the SRP ...................................................................................................... 15

Step 3: Deploying Oracle .............................................................................................................. 16

Task 1: Logging in to the MKTPRD SRP .......................................................................................... 16

Task 2: Installing Oracle software ................................................................................................ 16

Task 3: Creating a database ....................................................................................................... 19

Task 4: Modifying listener.ora ..................................................................................................... 22

Task 5: Automatically starting up and shutting down the database ..................................................... 22

Step 4: Completing the Serviceguard configuration (only required for Serviceguard integration) ................. 25

Task 1: Stopping the SRP ............................................................................................................ 25

Task 2: Exporting the SRP configuration ........................................................................................ 26

Task 3: Import SRP configuration on additional cluster nodes ............................................................ 26

Task 4: Exporting the SLVM configuration ...................................................................................... 28

Task 5: Importing SLVM on additional cluster nodes ........................................................................ 28

Task 6: Creating SRP Serviceguard package configuration ............................................................... 28

Task 7: Verifying and applying package Serviceguard package configuration..................................... 29

Step 5: Extending Serviceguard monitor script (only use for Serviceguard integration) .............................. 32

Summary of content (47 pages)

PAGE 1
Reference Architecture: Consolidating Oracle Databases with Secure Resource Partitions in a Serviceguard Cluster Workload Consolidation with SRP Technical white paper Table of contents Executive summary .......................................................................................................................... 3 Benefits of database consolidation ................................................................................................. 3 Introduction to Secure Resource Partitions .....
PAGE 2
Conclusion .................................................................................................................................. 34 Appendix A: SRP setup (srp_sys –s output)......................................................................................... 35 Appendix B: Oracle RC script ......................................................................................................... 39 Appendix C: Creating shared logical volumes ...................................................
PAGE 3
Executive summary The HP Virtual Server Environment Reference Architecture represents a simple solution for database consolidation using HP Secure Resource Partitions version 2 (SRPv2).
PAGE 4
Introduction to Secure Resource Partitions HP-UX Secure Resource Partitions version 2 (SRPv2) provides an environment for securely consolidating multiple workloads within a single image of the HP-UX 11i operating system. HP SRPv2 uses Security Containment to provide secure isolation between SRPs. HP Process Resource Manager (PRM) enforces CPU and memory resource entitlements for each SRP.
PAGE 5
General considerations It is common to have a single Oracle administrative domain for systems running multiple Oracle database servers. With SRPv2, you can create an isolated execution and administration domain, enabling the creation of multiple Oracle administrative domains, one for each SRP. By creating individual administrative domains, you can restrict which databases database administrators can access.
PAGE 6
binaries and libraries but might run with different configuration/profiles, usually by specifying a different configuration file for each instance Pros: – Single installation to patch – Little extra disk space consumption for additional instances Cons: – Single version of application software – Inflexible installation – Inflexible update path when updating binaries all applications will be affected  Multi-installation/Multi-instance—You can install multiple copies (and potentially multiple versions) of the
PAGE 7
same level of security and flexibility of the configuration described in this white paper. To keep the configuration modular and flexible, the Oracle file systems are mounted when the SRP starts. Each SRP configuration can include a private fstab file (/var/hpsrp//etc/fstab) consisting of file systems that are mounted and unmounted at SRP stop or start time. The SRP private fstab uses the same format as the system fstab file. Therefore, any type up file system can be mounted during SRP start.
PAGE 8
Data needed Value used in this deployment SRP name MKTPRD SRP administrator root SRP login group mkt_dba IP address 192.1.1.11 Netmask 255.255.255.0 Default gateway 192.1.1.
PAGE 9
NDD_VALUE[2]=1 TRANSPORT_NAME[3]=ip6 NDD_NAME[3]=ip6_ire_cmpt_route_lookup_policy NDD_VALUE[3]=1 The required patches previously listed add functionality to the underlying HP-UX subsystems used by SRP (Security Containment and ARPA Transport). Use the following kctune commands to enable use of the new features: # kctune cmpt_net_enhs=3 # kctune cmpt_fs_enhs=1 New features available:  Enable all cross SRP network connections: Enabling this feature allows network connections between all SRPs by default.
PAGE 10
See Appendix A: SRP setup (srp_sys –s output) for full listing. # srp_sys -setup ############################## # # Setup SRP default template # ############################## Loading SRP default template ...
PAGE 11
Task 2: Adding a new users group 1. Authorize this group of users to log in to the SRP: # groupadd –g 200 mkt_dba For Serviceguard integration, the SRP login group must exist on all nodes of the cluster and have the same gid. If the systems are not configured to use a central user/group repository such as LDAP or NIS, then you must create the group on all cluster nodes the package may run on.
PAGE 12
If you are not using Serviceguard: # srp -batch -a MKTPRD ip_address= 192.1.1.11 iface=lan0 ip_mask=255.255.255.0 gw_ip_address=192.1.1.1 login_group=mkt_dba prm_group_type=PSET prm_cores=1 add compartment rules succeeded add compartment directory succeeded add RBAC admin role for compartment succeeded add RBAC compartment login role succeeded add startup directories succeeded add prm rules succeeded add compartment network service rules succeeded add ipaddress 192.1.1.
PAGE 13
2. (Optional) If your processor supports Hyper-Threading and you want to enable it for the SRP, modify the PRM record for the SRP in the /etc/prmconf file and run prmconfig –i: # grep ^MKTPRD /etc/prmconf MKTPRD:PSET:::1::LCPU=ON NOTE: Hyper-Threading is only available with PSETs and must be enabled in the kernel. (See lcpu_attr(5) for details.) Step 2: Preparing the system Task 1: Creating the Oracle user Using the useradd command, create the user that the database instance uses.
PAGE 14
Task 2: Creating the Oracle file systems Before logging in to the SRP, installing the Oracle software, and creating an Oracle database, you must create the file systems that the Oracle installation will use.
PAGE 15
For Serviceguard integration, you must manually configure the SRP LAN interface and mount the file systems for the SRP before executing srp –start. You only need to perform these additional steps during the initial configuration/setup process. These steps are not required after the Serviceguard package is created and used to start or stop SRP. 1. Enable the SRP IP address, and create a host route to the default gateway: # ifconfig lan0:10 192.1.1.11 netmask 255.255.255.0 # /usr/sbin/route add default 192.
PAGE 16
ora_mkt 28399 28397 $ 4 14:30:22 ? 0:00 sshd: ora_mkt@pts/0 Step 3: Deploying Oracle The following example uses a new installation of Oracle 11g and creates a sample database, using the Oracle Universal Installer. After logging in to the SRP, the procedures for installing the Oracle software and creating the database are no different than on a standalone system. If migrating an existing database, use the standard tools and procedures to move the configuration and data over.
PAGE 17
Figure 1 Figure 2 17
PAGE 18
Figure 3 18
PAGE 19
Figure 4 Task 3: Creating a database You do not have to run the orainstRoot.sh script because the inventory pointer file is already created: /var/hpsrp/MKTPRD/home/ora_mkt/oraInst.loc When using the runInstaller in the future use the “ –invPtr/var/hpsrp/MKTPRD/home/ora_mkt/oraInst.loc” option to point to the SRP inventory pointer. Using the dbca tool create an Oracle database $ /var/hpsrp/MKTPRD/apps/oracle/product/11.1.
PAGE 20
Figure 5 20
PAGE 21
Figure 6 Figure 7 21
PAGE 22
Task 4: Modifying listener.ora Although a process running in an SRP only has access to the SRP IP address (by default), it is a best practice to restrict processes to listen only on the SRP IP address. 1. Modify the Oracle listener configuration to bind only to the SRP IP address by setting the HOST parameter to the IP address of the SRP: $ cat $ORACLE_HOME/network/admin/listener.ora LISTENER = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = 192.1.1.
PAGE 23
ORACLE_HOME=/var/hpsrp/MKTPRD/u01/app/product/11.1.0/db_1 DB_START=/var/hpsrp/MKTPRD/home/ora_mkt/dbstart DB_SHUT=/var/hpsrp/MKTPRD/home/ora_mkt/dbshut DB_USER=ora_mkt 5. Create RC symbolic links for automatic start/stop. # ln -s /var/hpsrp/MKTPRD/sbin/init.d/mktprd /var/hpsrp/MKTPRD/sbin/rc1.d/K401mktprd # ln -s /var/hpsrp/MKTPRD/sbin/init.d/mktprd /var/hpsrp/MKTPRD/sbin/rc2.d/S376mktprd 6. Create start/stop scripts.
PAGE 24
# ps -fR MKTPRD UID PID PPID C STIME TTY TIME COMMAND ora_mkt 6708 1 6 10:52:25 ? 0:00 ora_vktm_MKTPRD ora_mkt 6737 1 3 10:52:26 ? 0:00 ora_d000_MKTPRD ora_mkt 6719 1 6 10:52:26 ? 0:00 ora_dia0_MKTPRD ora_mkt 6727 1 2 10:52:26 ? 0:00 ora_ckpt_MKTPRD ora_mkt 6699 1 0 10:52:25 ? 0:00 /var/hpsrp/MKTPRD/u01/app/product/11.1.
PAGE 25
SQL> exit Disconnected from Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options # Step 4: Completing the Serviceguard configuration (only required for Serviceguard integration) The following section details the steps needed to complete the Serviceguard configuration for the SRP.
PAGE 26
# ifconfig lan0:10 0 # /usr/sbin/route delete default 192.1.1.1 1 source 192.1.1.11 delete net default: gateway 192.1.1.1 Task 2: Exporting the SRP configuration 1. Using the srp –export command, export the SRP configuration to an exchange file. This exchange file is copied to all nodes in the cluster. Do not use the ok_export_dir option to include the SRP private directory because Serviceguard mounts this filesystem on the node running the package. # srp -batch -export MKTPRD -xfile /tmp/MKTPRD.
PAGE 27
X11MotifDevKit CDEDevKit C-ANSI-C C-Dev-Tools CodeAdvisor Auxiliary-Opt ACXX B.11.31.01 B.11.31.01 C.06.25 C.06.25 C.02.15 C.06.25 C.06.25 The following products have different versions: product source system target system ------------------------------Jre60 1.6.0.06.00 1.6.0.05.01 SRP MKTPRD exported from atcuxvm7 on Tue Apr 13 11:02:02 PDT 2010 import compartment rules succeeded x cmpt/apache.srp_incl, 394 bytes, 1 tape blocks x cmpt/base.srp_incl, 3351 bytes, 7 tape blocks x cmpt/oracledb.
PAGE 28
----------------- Status for SRP:MKTPRD ---------------------Status:STOPPED IP: 192.1.1.11 Interface:(not managed by SRP) MEM Entitle:90.91% CPU Entitle:50.00% MEM Max:(none) CPU Max:(none) Usage:0.00% Usage:0.00% Task 4: Exporting the SLVM configuration See Appendix C: Creating shared logical volumes for examples of exporting logical volume configuration.
PAGE 29
# Add SLVM entries for all shared lvols fs_name /dev/vgMKTPRD/MKTPRD fs_directory /var/hpsrp/MKTPRD fs_type "vxfs" fs_mount_opt "-o rw" fs_umount_opt "" fs_fsck_opt "" fs_name /dev/vgMKTPRD/MKTPRD_SW fs_directory /var/hpsrp/MKTPRD/u01 fs_type "vxfs" fs_mount_opt "-o rw" fs_umount_opt "" fs_fsck_opt "" fs_name /dev/vgMKTPRD/MKTPRD_DB fs_directory /var/hpsrp/MKTPRD/u02 fs_type "vxfs" fs_mount_opt "-o rw" fs_umount_opt "" fs_fsck_opt "" vg vgMKTPRD Task 7: Verifying and applying package Serviceguard packag
PAGE 30
Validation for package MKTPRD succeeded via /etc/cmcluster/scripts/mscripts/master_control_script.sh. Modify the package configuration ([y]/n)? y Completed the cluster update 4. Start the package: # cmrunpkg -v MKTPRD Running package MKTPRD on node atcuxvm7 Successfully started package MKTPRD on node atcuxvm7 cmrunpkg: All specified packages are running 5.
PAGE 31
ster/MKTPRD/srp_monitor_script ora_mkt MKTPRD 12419 ora_psp0_MKTPRD ora_mkt MKTPRD 12458 ora_qmnc_MKTPRD ora_mkt MKTPRD 12431 ora_ckpt_MKTPRD ora_mkt MKTPRD 12415 ora_diag_MKTPRD ora_mkt MKTPRD 12423 ora_dia0_MKTPRD ora_mkt MKTPRD 12429 ora_lgwr_MKTPRD ora_mkt MKTPRD 12413 ora_vktm_MKTPRD # 1 0 14:15:33 ? 0:00 1 0 14:15:43 ? 0:00 1 0 14:15:33 ? 0:00 1 0 14:15:33 ? 0:00 1 0 14:15:33 ? 0:00 1 0 14:15:33 ? 0:00 1 0 14:15:33 ? 0:00 6.
PAGE 32
mktprd.rose.hp.com)(PORT = 1521))(CONNECT_DATA = (SERVER = DEDICATED)(SERVICE_NAME = MKTPRD)))" EMPNO ENAME JOB MGR HIREDATE SAL COMM ---------- ---------- --------- ---------- --------- ---------- --------DEPTNO ---------7902 FORD ANALYST 7566 03-DEC-81 3000 20 Step 5: Extending Serviceguard monitor script (only use for Serviceguard integration) SRP provides a basic Serviceguard monitor script (srp_monitor_script) that can monitor critical application functionality.
PAGE 33
} function db_active { if [ -f "$MAINTENANCE_FILE" ]; then return 0 fi active=`su $ORACLE_USER -c "echo 'set head off\nconn / as sysdba\n select DATABASE_STATUS from v\\\$instance;' | ${ORACLE_HOME}/bin/sqlplus -S /nolog"` echo $active | grep ACTIVE >/dev/null return $? } In the main block of the script add the code between the Begin/End added code comments: ################ # main routine ################ while [ 0 ] do sleep $SLEEP_TIME # Begin added code listener_up if (($? != 0)); then echo "ERROR: List
PAGE 34
Conclusion Consolidation is extremely important in current IT infrastructures and provides IT managers opportunities to reduce costs, while increasing usage of their IT resources. As important as these benefits are, it is also important to maintain service levels agreements in a secure operating environment when consolidating multiple workloads on to a single system.
PAGE 35
Appendix A: SRP setup (srp_sys –s output) # srp_sys -s ############################## # # Setup SRP default template # ############################## Loading SRP default template ...
PAGE 36
To enable additional users, execute the following command: roleadm assign SRPlogin-init Would you like to enable Compartment Login feature? [y] Adding RBAC role (SRPlogin-init) for INIT cmpt login ... [ OK ] Any service monitored by pam_hpsec account management module is enabled with compartment login enabled. Below is a list of such services in /etc/pam.conf. Refer to pam_hpsec(5) on how to manage compartment login feature.
PAGE 37
############################## # # sshd configuration # ############################## Checking sshd configuration ... Enter sshd configuration file: [/opt/ssh/etc/sshd_config] [ OK ] Saving SRP default template ... [ OK ] Detected Init Compartment Secure Shell daemon listening on all IP addresses. Will conflict with any SRP Secure Shell daemons. Would you like to restrict the Init compartment's sshd IP addresses? [y] Enter IP addresses, separated by comma ',': [10.10.10.
PAGE 38
Note: compartment enablement will not take effect until system is rebooted. Reboot system now (cd /; shutdown -r now)? [n] y SHUTDOWN PROGRAM 01/07/10 16:18:16 PST Do you want to send your own message? (You must respond with 'y' or 'n'.): n Broadcast Message from root (ttyp3) Thu Jan 7 16:18:40... PLEASE LOG OFF NOW ! ! ! System maintenance about to begin. All processes will be terminated in 0 seconds. Broadcast Message from root (ttyp3) Thu Jan SYSTEM BEING BROUGHT DOWN NOW ! ! ! … 38 7 16:18:40...
PAGE 39
Appendix B: Oracle RC script #!/sbin/sh # --- Set variables WHAT='Oracle DB' ORACLE_HOME= DB_START= DB_SHUT= DB_USER= # --case $1 in 'start_msg') echo "Starting $WHAT" rval=0 ;; 'stop_msg') echo "Stopping $WHAT" rval=0 ;; 'start') su - $DB_USER -c "$DB_START $ORACLE_HOME" rval=$? ;; 'stop') su - $DB_USER -c "$DB_SHUT $ORACLE_HOME" rval=$? ;; *) echo "usage: $0 {start|stop|start_msg|stop_msg}" rval=1 ;; esac exit $rval 39
PAGE 40
Appendix C: Creating shared logical volumes Creating volume group vgMKTPRD # /usr/sbin/pvcreate /dev/rdsk/c2t0d0 Physical volume "/dev/rdsk/c2t0d0" has been successfully created. # /usr/bin/mkdir -p -m u=rwx,g=rx,o=rx /dev/vgMKTPRD # /sbin/mknod /dev/vgMKTPRD/group c 64 0x010000 # /usr/bin/chmod u=rw-,g=r,o=--- /dev/vgMKTPRD/group # /usr/sbin/vgcreate -A y -x y -l 255 -p 16 -s 4 /dev/vgMKTPRD /dev/dsk/c2t0d0 Increased the number of physical extents per physical volume to 12799.
PAGE 41
# /usr/sbin/cmexec atcuxvm8 /usr/sbin/vgchange -a n /dev/vgMKTPRD Volume group "/dev/vgMKTPRD" has been successfully changed. # /usr/sbin/cmexec atcuxvm8 /usr/sbin/vgexport /dev/vgMKTPRD vgexport: Volume group "/dev/vgMKTPRD" has been successfully removed. # /usr/sbin/vgexport -s -p -v -m /var/tmp/fsweb.atcuxvm7.vgMKTPRD.map /dev/vgMKTPRD Beginning the export process on Volume Group "/dev/vgMKTPRD". vgexport: Volume group "/dev/vgMKTPRD" is still active.
PAGE 42
Please remember to take a backup using the vgcfgbackup command after activating the volume group. # /usr/sbin/cmexec atcuxvm8 /usr/bin/rm /var/tmp/fsweb.atcuxvm7.vgMKTPRD.map # /usr/bin/rm /var/tmp/fsweb.atcuxvm7.vgMKTPRD.map Creating logical volume MKTPRD_SW for Oracle application software # /usr/sbin/lvcreate -A y -n MKTPRD_SW -L 10240 -p w -s y vgMKTPRD Logical volume "/dev/vgMKTPRD/MKTPRD_SW" has been successfully created with character device "/dev/vgMKTPRD/rMKTPRD_SW".
PAGE 43
version 7 layout 10485760 sectors, 10485760 blocks of size 1024, log size 16384 blocks largefiles supported # /usr/sbin/mount -F vxfs -e /dev/vgMKTPRD/MKTPRD_DB /var/hpsrp/MKTPRD/u02 mount: mounted /dev/vgMKTPRD/MKTPRD_DB on /var/hpsrp/MKTPRD/u02 # chown ora_mkt:mkt_dba /var/hpsrp/MKTPRD/u02 Creating file system /var/hpsrp/MKTPRD/u01 for Oracle software # /sbin/mkdir -p -m u=rwx,g=rx,o=rx /var/hpsrp/MKTPRD/u01 # /usr/sbin/mkfs -F vxfs /dev/vgMKTPRD/rMKTPRD_SW version 7 layout 10485760 sectors, 10485760 blo
PAGE 44
Appendix D: SRP Serviceguard package configuration file package_name package_description MKTPRD "SRP MKTPRD Serviceguard Package" module_name module_version module_name module_version module_name module_version module_name module_version module_name module_version module_name module_version module_name module_version module_name module_version module_name module_version sg/basic 1 sg/failover 1 sg/priority 1 sg/service 1 sg/external 1 sg/volume_group 1 sg/filesystem 1 sg/package_ip 1 sg/acp 1 package_ty
PAGE 45
ip_subnet ip_address 192.1.1.0 192.1.1.
PAGE 46
Appendix E: SRP start-up sequence Figure 8 Figure 9 46
PAGE 47
For more information For more information on the HP Virtual Server Environment and HP VSE Reference Architectures, see: http://www.hp.com/go/vse http://www.hp.com/go/vsera For more information on the HP Secure Resource Partitions, see: http://www.hp.com/go/hpux-security-docs For more information on the HP-UX 11i Security, see: http://www.hp.com/go/hpuxsecurity For HP-UX 11i Security training from HP (including Secure Resource Partitions), see: http://www.hp.com/education/courses/h3541s.