Database Serving Solution Pack for Oracle HiAv on Linux Installation and Administration Guide Database Serving Solution Pack for Oracle HiAv on Linux 3.1.
Copyright © 2004-2006 PolyServe, Inc. Use, reproduction and distribution of this document and the software it describes are subject to the terms of the software license agreement distributed with the product (“License Agreement”). Any use, reproduction, or distribution of this document or the described software not explicitly permitted pursuant to the License Agreement is strictly prohibited unless prior written permission from PolyServe has been received.
Contents 1 Introduction 2 Installation and Configuration New Installations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Software Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Install MxDB-Oracle-HiAv . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Upgrade MxDB-Oracle-HiAv from the 3.1.0 Release . . . . . . . . . . . . . Upgrade MxDB-Oracle-HiAv from a Release Earlier than 3.1.0 . . . . 1.
Contents iv Maintenance Mode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Delete a Virtual Oracle Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . The mxdb Command Line Interface . . . . . . . . . . . . . . . . . . . . . . . Multiple Oracle Homes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Additional Monitoring Capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . Supplemental Startup Scripts . . . . . . . . . . . . . .
1 Introduction The Database Serving Solution Pack for Oracle HiAv on Linux (MxDB-Oracle-HiAv) provides high availability for database instances and Net Services listener processes. MxDB-Oracle-HiAv uses a Virtual Oracle Service to provide connectivity to the database. A service monitor is associated with each Virtual Oracle Service. The monitor periodically checks the health of the matrix servers and the database instances and listeners via a probe action.
2 Installation and Configuration This chapter describes how to install the Database Serving Solution Pack for Oracle HiAv on Linux (MxDB-Oracle-HiAv) and lists prerequisites for the product. New Installations Software Prerequisites Before installing MxDB-Oracle-HiAv, you will need to install the following on each node in the matrix, as described in the PolyServe Matrix Server Installation Guide. • A supported operating system and kernel as specified in the MxDB-Oracle-HiAv release notes.
Chapter 2: Installation and Configuration 2. Install Oracle into a PSFS filesystem. 3. Install the PolyServe MxS Oracle Database Solution Pack (MxODM). 4. Set up the matrix configuration file on each node. 5. Perform network configuration for Virtual Oracle Services. 1. Create PSFS Filesystems The PolyServe Matrix Server Cluster filesystem supports a specialized mount option called DBOPTIMIZE that enables direct I/O. It is recommended that all database files (e.g.
Chapter 2: Installation and Configuration 4 See “Configure PSFS Filesystems” in the PolyServe Matrix Server Administration Guide for details about creating and mounting filesystems. 2. Install Oracle For MxDB-Oracle-HiAv to function properly, Oracle Home must be installed into a PSFS filesystem. MxDB-Oracle-HiAv depends on the standard Optimal Flexible Architecture (OFA) layout of the Oracle software.
Chapter 2: Installation and Configuration 5 The MxODM software is located in the ODM subdirectory of the MxDB-Oracle-HiAv distribution. See the PolyServe MxS Oracle Database Solution Pack Release Notes for a description of the software and installation information. 4. Set Up the Matrix Configuration File The root user’s matrix configuration file (/root/.matrixrc) must be set up on each node in the matrix.
Chapter 2: Installation and Configuration 6 Upgrade MxDB-Oracle-HiAv from the 3.1.0 Release If MxDB-Oracle-HiAv 3.1.0 is currently installed, you can use a rolling upgrade to go to the 3.1.1 release. During the upgrade, some servers can be configured with MxDB-Oracle-HiAv 3.1.0, while others are configured with MxDB-Oracle-HiAv 3.1.1. To perform a rolling upgrade, complete the following steps: 1. Rehost all databases from a Matrix Server node to other servers using the MxDB-Oracle-HiAv GUI or CLI. 2.
Chapter 2: Installation and Configuration 7 4. Install MxODM 3.1.0 into only one node of the matrix. 5. Install MxDB-Oracle-HiAv 3.1.1 on each node of the matrix. 6. Use the MxDB-Oracle-HiAv GUI to create new Virtual Oracle Services. Each step is described in detail below. 1. Stop Existing Databases The -S option to the MxDB-Oracle CLI command stops a database currently under the control of MxDB-Oracle-HiAv. Repeat the command for each database.
Chapter 2: Installation and Configuration 8 4. Install MxODM 3.1.0 MxODM 3.1.0 must be installed into the Shared Oracle Home on only one node in the matrix. Refer to the PolyServe MxS Oracle Database Solution Pack Release Notes for details. 5. Install MxDB-Oracle-HiAv 3.1.1 MxDB-Oracle-HiAv must be installed on each node in the matrix. It can be installed from either the product CD or the location where you have downloaded the software. Issue the following command: # rpm -ivh /mxdb_oracle_ha-3.
3 Operations Management Applications The following management applications are used to perform MxDB-Oracle-HiAv operations: MxDB-Oracle-HiAv Graphical User Interface. This GUI supports both configuration and operations activities. MxDB-Oracle Command Line Interface. This CLI supports operations, but does not support Virtual Oracle Service configuration actions.
Chapter 3: Operations 10 use of Oracle Net Services tools such as netca and netmgr is also supported. • Creates a virtual host that SQL*Net clients use to access the database. • Creates a service monitor that checks the health of servers, database instances, and listeners. Once a VHOST has an associated service monitor, it is known as a Virtual Oracle Service. • Performs the initial boot of the database instance and the Net Services listener.
Chapter 3: Operations 11 Naming Conventions The associations among the Virtual Oracle Server, service monitor, Database Instance, and Database Name can become confusing if certain conventions are not followed. It is generally best practice to use the following naming scheme: • The names of the database and the ORACLE_SID should be the same, including case sensitivity. • The names of the service monitor and the database should be the same.
Chapter 3: Operations 12 The following parameters can be set in the configuration file. Parameter Value Action log A full pathname to a PSFS directory MxDB-Oracle-HiAv logs messages into this directory lsnr_password A legal SQL*Net Listener password, or NONE to disable listener passwords for the database Overrides the default Solution Pack Listener password Solution Pack Logging The probes and monitors provided with the Solution Pack send logging output to two locations.
Chapter 3: Operations 13 For example: $ cd $ORACLE_HOME/dbs $ ls -l *PROD*conf -rw-r--r-- 1 root root 75 Feb 8 17:12 mxdbha_PROD.conf $ cat mxdbha_PROD.conf log /u01/app/oracle/product/9.2.0/rdbms/log lsnr_password 8B1323CCD6F199D5 In this case, the listener.ora file will need to have a matching assignment, such as: $ cd $ORACLE_HOME/dbs $ ls -l *PROD*conf -rw-r--r-1 root root 75 Feb 8 17:12 mxdbha_PROD.conf $ cat mxdbha_PROD.conf log /u01/app/oracle/product/9.2.
Chapter 3: Operations 14 Place Databases Under MxDB-Oracle-HiAv Control The Solution Pack includes a demonstration database called PILOT that you can use to become familiar with the process of placing a database under MxDB-Oracle-HiAv control and with managing the database via MxDB-Oracle-HiAv. Create the Demonstration Database The Configuration GUI is used to create the PILOT demonstration database. To invoke the GUI, execute the following command from an xterm session.
Chapter 3: Operations 15 The database name will be filled in as PILOT by the GUI. Continue to populate the Virtual Host Name box with the DNS entry given by the System Administrator for this service. Proceed to move the desired nodes from the Available Nodes box to the Selected Nodes box. Configure Net Services (SQL*Net) Select how Net services should be configured in the Net services (SQL*Net) section of the Configuration GUI. MxDB-Oracle-HiAv will configure listener.ora and tnsnames.
Chapter 3: Operations 16 If you want MxDB-Oracle-HiAv to configure the files, click the “Allow solution pack to configure the default Net Services configuration” button, and then examine the tunable location ($ORACLE_HOME/network/admin, by default) and port (1521, by default) specified for TNS_ADMIN. If the values are acceptable, click the Configure button.
Chapter 3: Operations 17 Execute the PILOT_create.sh Script When you have completed the Net Services configuration and clicked the Apply button, a dialog box appears directing you to connect to the system as the Oracle user through an xterm session. After you are logged in, go to the /tmp directory and execute the PILOT_create.sh script. It takes approximately five minutes to create the PILOT database.
Chapter 3: Operations 18 created. Be sure to examine this file. Additional logging is provided in the /tmp/MxDB-Oracle-HiAv.log file. Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 3: Operations 19 The status bar at the bottom of the GUI indicates the tasks it is performing as shown on the following screen. Configure an Existing Database To configure High Availability for an existing database, first shut the database down, then start the configuration GUI by executing the following commands from an xterm session. You must be the Oracle user or a member of group DBA, and ORACLE_HOME must be set in your environment.
Chapter 3: Operations 20 As was the case with the PILOT demonstration database creation, select the Oracle Home and verify/correct the oracle owner and DGA group information. Next, click on the Database Name pull-down and choose Create New Database Service. Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 3: Operations 21 Next, enter the name of the database in the Database Name box. In the following screens, an existing Oracle9i database called PROD is placed under Solution Pack control. The Matrix Server VHOST created in this sample Virtual Oracle Service setup is called vnprod.pdx.polyserve.com. Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 3: Operations 22 Move the nodes you want to act as Primary and Backups into the Selected Nodes box. Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 3: Operations 23 Next, either allow the Solution Pack to configure Net Services (or do so with netca). Be sure to supply the TNS_ADMIN path. In the example, the PROD database uses the default TNS_ADMIN directory and the listener will listen on port 1621. Click Apply. Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 3: Operations 24 The status bar indicates the various steps and progress. The Solution Pack will report that it is verifying the database as meeting the Solution Pack criteria (e.g., $ORACLE_HOME/dba/init.ora, and so on). Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 3: Operations 25 The GUI then reports that the operation succeeded by displaying a green status bar. Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 3: Operations 26 The GUI now refreshes its information regarding the new Virtual Oracle Service. In the example case, the GUI has placed PROD in the Database Name field and illuminated a green Running indicator suggesting that the database is up and is being probed by the Solution Pack. Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 3: Operations 27 Database Operations Database Administration changes slightly when a database is monitored by MxDB-Oracle-HiAv. If a database fails the monitor probe, the database will be restarted. Therefore, if a DBA connects to a database through SQL*Plus or Enterprise Manager and shuts the database down, it will be restarted by the Solution Pack.
Chapter 3: Operations On this screen, the Virtual Oracle Service is being stopped. Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 3: Operations On this screen, the Virtual Oracle Service is stopped and the Stop Database button is now labeled “Start Database.” To restart the Virtual Oracle Service, click the Start Database button. Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 3: Operations 30 Rehost a Database MxDB-Oracle-HiAv makes it easy to move a database service within the matrix. The following three screens show the GUI actions necessary to move the PROD database service from the Primary to the 1st Backup server. Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 3: Operations 31 Reorder the “Selected Nodes” list as desired to change the Primary and Backup assignments. Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 3: Operations 32 After reordering the nodes in the Selected Nodes list, click the Apply button. The Solution Pack then shuts down the database on the old Primary and starts it on the newly assigned Primary. Add or Remove Nodes On the Application window, move the nodes between the Available Nodes and Selected Nodes boxes to effect adding or removing nodes from the Virtual Oracle Service. Adding nodes to a Virtual Oracle Service does not impact the database, as it is already running.
Chapter 3: Operations 33 NOTE: Performing any re-ordering of the Primary causes the database to be stopped and restarted. On the following screen, the 1st Backup is removed, 10.10.60.7 is added as the 2nd Backup, and 10.10.80.8 is promoted from 2nd Backup to 1st Backup, all in one operation. The database is running on node 10.10.60.6 and will not be impacted by this operation. Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 3: Operations 34 Maintenance Mode Maintenance Mode is an important state of the Virtual Oracle Service. Once a database has been placed under Solution Pack control, all Net Services access is through the VHOST as established in the listener.ora and tnsnames.ora files. When a Virtual Oracle Service is in the Stopped state, access through the VHOST is disabled. After all, it is a Virtual Oracle Service with no service.
Chapter 3: Operations 35 Without Maintenance Mode, this normal scenario might pose a problem. Consider administrative tasks that the DBA needs to conduct while the database is executing, but the Solution Pack probes are disabled. For example, you may need to apply a patch where manual database startup and shutdown need to occur but without action being taken by the Solution Pack due to probe failures.
Chapter 3: Operations On this screen, the sample PROD Virtual Oracle Service is placed into Maintenance Mode. Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 3: Operations On this screen, sqlplus is used to shut down the database. Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 3: Operations On this screen, sqlplus is used to restart the database. Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 3: Operations 39 On this screen, a connection is made through the VHOST via SQL*Net. To stop Maintenance Mode for a database, simply click the Stop Maintenance button. If the database is down when Stop Maintenance is clicked, it will be started on the Primary. If the database is already running, probing resumes. Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 3: Operations 40 Delete a Virtual Oracle Service To delete a Virtual Oracle Service, click on Delete Oracle Service. A dialogue will appear for verification. Deleting a Virtual Oracle Service does not affect the database itself. The VHOST and service monitor are removed, but the database remains intact. It will be shut down during this operation. The entries for the service will remain in the SQL*Net configuration files.
Chapter 3: Operations 41 The mxdb Command Line Interface In addition to the -c option, which invokes the GUI, the mxdb command can be used for operations.The command is invoked as follows: $ /opt/polyserve/mxdb_oracle_ha/bin/mxdb The mxdb command includes options that enable Database Administrators to perform the same operations available in the Solution Pack GUI. The following table describes the command arguments. Option Procedure -d This argument is required.
Chapter 3: Operations 42 Multiple Oracle Homes The GUI and CLI can perform operations on any Oracle Home of either Oracle9i or Oracle10g. The oracle user can login and execute the GUI and select an Oracle9i or Oracle10g home to perform operations on without care for environment variables. Likewise, with the mxdb command, the -d and -h options can be used to operate upon any database in the system as long as the user executing the command is in the dba group.
Chapter 3: Operations 43 Supplemental Startup Scripts MxDB-Oracle-HiAv supports site-customizable startup and shutdown scripts. The Solution Pack ensures that the database instance and SQL*Net listener processes are available in the matrix. You might also want supplemental processes to be started and stopped each time the database is restarted or failed over to another node.
Chapter 3: Operations 44 • Naming Convention. The naming convention for a supplemental script is as follows: – Path. Supplemental startup scripts are located in the etc directory of the Solution Pack (i.e., /opt/polyserve/mxdb_oracle_ha/etc). – Script Name. There is only one supplemental script per database. The script is named custom_${ORACLE_SID}.sh. • Permissions. The execution bit must be set on supplemental scripts.
4 Oracle Enterprise Manager Configuration Issues with Failover High Availability Special configuration is required to take advantage of advanced Enterprise Manager features such as Automatic Discovery and the Enterprise Manager Job System in a failover High Availability environment. Incidentally, configuring a shared Oracle Home offers an increased level of availability with regard to submitted jobs. In a traditional Failover HA scenario, submitted jobs are lost during transition.
Chapter 4: Oracle Enterprise Manager 46 Configuration Tutorial The following eight steps provide a tutorial and example of setting up Autodiscovery and persistent Enterprise Manager Jobs. Step 1. In accordance with Metalink Note 190026.1, establish a fixed value for the Agent dbsnmpkey. Place this value in a file of your choosing in the share Oracle Home (e.g., $ORACLE_HOME/network/agent/dbsnmpkey.txt). Without this key file, the agent file objects created by job submission (*.
Chapter 4: Oracle Enterprise Manager 47 Step 2. To function with a VHOST (Virtual Oracle Service), the Intelligent Agent config file snmp_rw.ora needs to be modified. The following screen shows the contents of the snmp_rw.ora file on a Matrix Server with an MxDB-Oracle-HiAv Virtual Oracle Service called Q9DEV.pdx.polyserve.com. When originally created, the sole content of the file was the assignment of the DBSNMP.HOSTNAME parameter. Intelligent Agent inserts the other content once it executes.
Chapter 4: Oracle Enterprise Manager 48 Verify that the Net Services files contains a HOST assignment that precisely matches the value given to the DBSNMP.HOSTNAME in the snmp_rw.ora file. The following screen shows an example of using the grep(1) command to validate file contents. Step 3.
Chapter 4: Oracle Enterprise Manager 49 Step 4. It is best to set up Oracle Management Server on a different tier than the database. Once the OMS server has been configured with a repository, it is time to perform initial discovery of the MxDB-Oracle-HiAv Virtual Oracle Services. The following seven screenshots taken from the example system show the following: • The Virtual Oracle Service called Q9DEV.pdx.polyserve.com is active on host 10.10.180.
Chapter 4: Oracle Enterprise Manager Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 4: Oracle Enterprise Manager Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 4: Oracle Enterprise Manager Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 4: Oracle Enterprise Manager Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 4: Oracle Enterprise Manager Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 4: Oracle Enterprise Manager 55 Step 5. Once the Virtual Oracle Service has been discovered, the Enterprise Manager Job System can be tested to ensure that the setup is correct. The following seven screen shots depict a test job being submitted to the sample MxDB-Oracle-HiAv Virtual Oracle Service called Q9DEV.pdx.polyserve.com. Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 4: Oracle Enterprise Manager Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 4: Oracle Enterprise Manager Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 4: Oracle Enterprise Manager Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 4: Oracle Enterprise Manager 59 Step 6. Once the job has been submitted, the MxDB-Oracle-HiAv failover will preserve the integrity of the submitted job since the architecture is based upon a cluster filesystem. The next step is to use the MxDB-Oracle-HiAv GUI to rehost the database to another node. In the example, the Q9DEV.pdx.polyserve.com Virtual Oracle Service will be rehosted from 10.10.180.1 to 10.10.180.2. Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 4: Oracle Enterprise Manager When the green “Running” indicator appears, the ping(1) command is used to establish that the Intelligent Agent is running on the newly rehosted server, in this case 10.10.180.2 Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 4: Oracle Enterprise Manager 61 Step 7. After rehosting (or failover), Enterprise Manager Agent ping can be used to ensure that auto discovery was successful. The following Agent ping establishes that auto discovery is working in your MxDB-Oracle-HiAv environment as seen in the following screen shots. Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 4: Oracle Enterprise Manager Copyright © 1999-2006 PolyServe, Inc. All rights reserved.
Chapter 4: Oracle Enterprise Manager 63 Step 8. Autodiscovery is only part of the value add, however. The most important issue is the persistence of Enterprise Manager jobs. Complying with Metalink Note 190026.1 and combined with the Matrix Server cluster filesystem, the job submitted earlier in the example is still scheduled to run even though the Q9DEV.pdx.polyserve.com Virtual Oracle Service was failed over from its primary to its secondary server. Copyright © 1999-2006 PolyServe, Inc.
