MxDB-Oracle-HiAv Installation and Administration Guide MxDB-Oracle-HiAv 3.5 for Red Hat Enterprise Linux AS/ES 4.
Copyright © 2004-2007 PolyServe, Inc. Use, reproduction and distribution of this document and the software it describes are subject to the terms of the software license agreement distributed with the product (“License Agreement”). Any use, reproduction, or distribution of this document or the described software not explicitly permitted pursuant to the License Agreement is strictly prohibited unless prior written permission from PolyServe has been received.
Contents 1 Install MxDB-Oracle-HiAv Software Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Installation Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1. Create PSFS Filesystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2. Install Oracle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3. Install MxODM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Contents iv Create the Demonstration Database . . . . . . . . . . . . . . . . . . . . . . . Configure an Existing Database . . . . . . . . . . . . . . . . . . . . . . . . . . . Database Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Start or Stop a Virtual Oracle Service . . . . . . . . . . . . . . . . . . . . . . Rehost a Database. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Add or Remove Nodes . . . . . . . . . . . . . . . . . . .
1 Install MxDB-Oracle-HiAv This chapter describes how to install or upgrade the Database Serving Solution Pack for Oracle HiAv on Linux (MxDB-Oracle-HiAv) and lists prerequisites for the product. Software Prerequisites Before installing MxDB-Oracle-HiAv, you will need to install the following on each node in the matrix, as described in the PolyServe Matrix Server Installation Guide. • A supported operating system and kernel as specified in the MxDB-Oracle-HiAv release notes.
Chapter 1: Install MxDB-Oracle-HiAv 2 5. Perform network configuration for Virtual Oracle Services. 6. Install MxDB-Oracle-HiAv. 1. Create PSFS Filesystems The Matrix Server PSFS filesystem supports a specialized mount option called DBOPTIMIZE that enables direct I/O. It is recommended that all database files (e.g., datafiles, online redo logs, archived redo logs) be stored in filesystems mounted with the DBOPTIMIZE mount option.
Chapter 1: Install MxDB-Oracle-HiAv 3 MxDB-Oracle-HiAv depends on the standard Optimal Flexible Architecture (OFA) layout of the Oracle software. Databases created with the Database Configuration Assistant (DBCA) will generally possess the layout required for MxDB-Oracle-HiAv. You will need to comply with two configuration requirements: • ORACLE_HOME/dbs. Appropriate database configuration files (either an init.ora or a spfile.ora) must exist in this directory for any database managed by MxDB-Oracle-HiAv.
Chapter 1: Install MxDB-Oracle-HiAv 4 Before starting the installation, be sure to obtain the following information. The installation procedure will ask for these values if it cannot locate them in the environment. • The value of $ORACLE_HOME (e.g., /u01/app/oracle/product/9204). • The Oracle user ID (e.g., oracle). • The Oracle DBA group (e.g., dba). MxODM can be installed from the product CD or the location where you have downloaded the software. Type the following command: # rpm -i /odm-3.
Chapter 1: Install MxDB-Oracle-HiAv 5 5. Perform Network Configuration for Virtual Oracle Services MxDB-Oracle-HiAv offers connectivity for a Virtual Oracle Service via a VHOST. You will need to create a DNS entry for the Virtual Oracle Service before you place the database under the control of MxDB-Oracle-HiAv. NOTE: For testing purposes, it is sufficient to simply define the Virtual Oracle Service definition in each /etc/hosts file.
Chapter 1: Install MxDB-Oracle-HiAv 6 3. Install MxDB-Oracle-HiAv 3.5.1 on the server: # rpm -ivh /mxdb_oracle_ha-3.5.1-release.i386.rpm 4. Repeat steps 2 and 3 on the remaining servers in the matrix. 5. Restart Oracle services: mxdb -d -O Upgrade MxDB-Oracle-HiAv from a Release Earlier than 3.1.0 NOTE: If you have a large number of databases and are currently using MxDB-Oracle-HiAv 2.7.
Chapter 1: Install MxDB-Oracle-HiAv 7 for each database. You will need to be the Oracle user when running the command. The following example stops a service monitor for a database instance called PROD: $ /opt/polyserve/mxdb_oracle_ha/bin/MxDB-Oracle -S PROD Oracle User is authenticated. Proceeding... Validating Service Monitor for PROD Preparing to stop the PROD Service on all nodes. Command successfully submitted to the High Availability engine. Please note, the database will change state asynchronously.
Chapter 1: Install MxDB-Oracle-HiAv 8 5. Install MxODM 3.5.1 MxODM 3.1.0 must be installed into the Shared Oracle Home on only one node in the matrix. Refer to the PolyServe MxS Oracle Database Solution Pack Release Notes for details. 6. Install MxDB-Oracle-HiAv 3.5.1 MxDB-Oracle-HiAv must be installed on each node in the matrix. It can be installed from either the product CD or the location where you have downloaded the software. Issue the following command: # rpm -ivh /mxdb_oracle_ha-3.5.
2 Operations Introduction MxDB-Oracle-HiAv provides high availability for database instances and Net Services listener processes. MxDB-Oracle-HiAv uses a Virtual Oracle Service, which is a type of Matrix Server virtual host, to provide connectivity to the database. A service monitor is associated with each Virtual Oracle Service. The monitor periodically checks the health of the matrix servers and the database instances and listeners via a probe action.
Chapter 2: Operations 10 Service Loss MxDB-Oracle-HiAv considers a Virtual Oracle Service to be in a down state if the probe returns a failure, or if the probe times out. If a database cannot satisfy the very rudimentary checks performed by the default MxDB-Oracle-HiAv probe script (mxdb_probe.sql), there is likely something very wrong with the instance. The default action taken by MxDB-Oracle-HiAv is to reboot the instance.
Chapter 2: Operations 11 • Creates a virtual host that SQL*Net clients use to access the database. • Creates a service monitor that checks the health of servers, database instances, and listeners. When a virtual host has an associated service monitor, it is known as a Virtual Oracle Service. • Performs the initial boot of the database instance and the Net Services listener. Operations The GUI supports a rich set of functionality for managing Virtual Oracle Services.
Chapter 2: Operations 12 Naming Conventions The associations among the Virtual Oracle Server, service monitor, Database Instance, and Database Name can become confusing if certain conventions are not followed. It is generally best practice to use the following naming scheme: • The names of the database and the ORACLE_SID should be the same, including case sensitivity. • The names of the service monitor and the database should be the same.
Chapter 2: Operations 13 The following parameters can be set in the configuration file. Parameter Value Action log A full pathname to a PSFS directory. MxDB-Oracle-HiAv logs messages into this directory. lsnr_password A legal SQL*Net Listener password, or NONE to disable listener passwords for the database. Overrides the default MxDB-Oracle-HiAv Listener password. MxDB_PROBE_CORE _TIMEOUT A value (in seconds) less than the total Service Monitor timeout.
Chapter 2: Operations 14 SQL*Net Security with Virtual Oracle Services MxDB-Oracle-HiAv implements listener security. By default, the listener password is taken from the following file in the product installation directory: /opt/polyserve/mxdb_oracle_ha/lib/lsnr_passwd_def.txt. The DBA can override this default by assigning a legal value to the lsnr_password parameter in the MxDB-Oracle-HiAv configuration file.
Chapter 2: Operations 15 ) (ADDRESS_LIST = (ADDRESS = (PROTOCOL = IPC)(KEY = IPCPROD)) ) ) ) PASSWORDS_LISTENER_PROD = 8B1323CCD6F199D5 Place Databases Under MxDB-Oracle-HiAv Control MxDB-Oracle-HiAv includes a demonstration database called PILOT that you can use to become familiar with the process of placing a database under MxDB-Oracle-HiAv control and with managing the database via MxDB-Oracle-HiAv. It is not necessary to configure this demonstration database.
Chapter 2: Operations 16 The database name will be filled in as PILOT by the GUI. In the Virtual Host Name field, enter the DNS entry that the System Administrator specified for this service. Then move the desired nodes from the Available Nodes column to the Selected Nodes column. Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 2: Operations 17 Configure Net Services (SQL*Net) Select how Net services should be configured in the Net services (SQL*Net) section of the Configuration GUI. MxDB-Oracle-HiAv will configure listener.ora and tnsnames.ora only if there is not a current set of these files generated by the Oracle Net Services configuration utilities (e.g., netca, netmgr).
Chapter 2: Operations 18 Execute the PILOT_create.sh Script When you have completed the Net Services configuration and clicked the Apply button, a dialog box appears directing you to connect to the system as the Oracle user through an xterm session. After logging in, go to the /tmp directory and run the PILOT_create.sh script. It takes about five minutes to create the PILOT database.
Chapter 2: Operations Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 2: Operations 20 The status bar at the bottom of the GUI indicates the tasks it is performing as shown on the following screen. Configure an Existing Database To configure High Availability for an existing database, first shut the database down and then start the configuration GUI by executing the following command from an xterm session. You must be the Oracle user or a member of group DBA, and ORACLE_HOME must be set in your environment.
Chapter 2: Operations 21 When the Application window appears, select the Oracle Home and verify/correct the Oracle owner and DBA Group information. (The Oracle Home pull-down list is populated with contents from the /etc/oratab file. You may need to copy the /etc/oratab file to each node in the cluster.) Next, click on the Database Name pull-down and choose Create New Database Service. Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 2: Operations 22 Next, enter the name of the database in the Database Name field. In the following screens, an existing Oracle9i database called PROD is placed under MxDB-Oracle-HiAv control. The Matrix Server Virtual Host created in this example is called vnprod.pdx.polyserve.com. Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 2: Operations 23 Move the nodes you want to act as Primary and Backups for the virtual host into the Selected Nodes box. Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 2: Operations 24 Next, either allow MxDB-Oracle-HiAv to configure Net Services or do so with netca. (For additional information about SQL*Net requirements, refer to “Configure Net Services (SQL*Net)” on page 17.) NOTE: Note, if the listener.ora and/or tnsnames.ora files have been manipulated with netca or netmgr, the solution pack will not modify them as per the text these tools embed at the beginning of the file. Be sure to verify the TNS_ADMIN path.
Chapter 2: Operations 25 The status bar indicates the various steps and progress. MxDB-Oracle-HiAv will report that it is verifying the database as meeting the necessary criteria (e.g., $ORACLE_HOME/dba/init.ora, and so on). Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 2: Operations 26 The GUI then reports that the operation succeeded by displaying a green status bar. Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 2: Operations 27 The GUI now refreshes its information regarding the new Virtual Oracle Service. In the following example, the GUI has placed PROD in the Database Name field and illuminated a green Running indicator suggesting that the database is up and is being probed by MxDB-Oracle-HiAv. Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 2: Operations 28 Database Operations Database Administration changes slightly when a database is monitored by MxDB-Oracle-HiAv. If a database fails the monitor probe, the database will be restarted. Therefore, if a DBA connects to a database through SQL*Plus or Enterprise Manager and shuts the database down, it will be restarted by MxDB-Oracle-HiAv.
Chapter 2: Operations On this screen, the Virtual Oracle Service is being stopped. Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 2: Operations On this screen, the Virtual Oracle Service is stopped and the Stop Database button is now labeled “Start Database.” To restart the Virtual Oracle Service, click the Start Database button. Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 2: Operations 31 Rehost a Database MxDB-Oracle-HiAv makes it easy to move a database service within the matrix. The following three screens show the GUI actions necessary to move the PROD database service from the Primary to the 1st Backup server. Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 2: Operations 32 Reorder the “Selected Nodes” list as desired to change the Primary and Backup assignments. Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 2: Operations 33 After reordering the nodes in the Selected Nodes list, click the Apply button. MxDB-Oracle-HiAv then shuts down the database on the old Primary and starts it on the newly assigned Primary. Add or Remove Nodes On the Application window, move the nodes between the Available Nodes and Selected Nodes columns to effect adding or removing nodes from the Virtual Oracle Service. Adding nodes to a Virtual Oracle Service does not impact the database, as it is already running.
Chapter 2: Operations 34 NOTE: Performing any re-ordering of the Primary causes the database to be stopped and restarted. On the following screen, the 1st Backup is removed, 10.10.60.7 is added as the 2nd Backup, and 10.10.80.8 is promoted from 2nd Backup to 1st Backup, all in one operation. The database is running on node 10.10.60.6 and will not be impacted by this operation. Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 2: Operations 35 Maintenance Mode Maintenance Mode is an important state of the Virtual Oracle Service. When a database is under MxDB-Oracle-HiAv control, all Net Services access is through the virtual host as established in the listener.ora and tnsnames.ora files. When a Virtual Oracle Service is in the Stopped state, access through the virtual host is disabled. After all, it is a Virtual Oracle Service with no service.
Chapter 2: Operations 36 Without Maintenance Mode, this normal scenario might pose a problem. Consider administrative tasks that the DBA needs to conduct while the database is executing, but MxDB-Oracle-HiAv probes are disabled. For example, you may need to apply a patch where manual database startup and shutdown need to occur but without action being taken by MxDB-Oracle-HiAv due to probe failures.
Chapter 2: Operations On this screen, the sample PROD Virtual Oracle Service is placed into Maintenance Mode. Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 2: Operations On this screen, SQL*Plus is used to shut down the database. Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 2: Operations On this screen, SQL*Plus is used to restart the database. Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 2: Operations 40 On this screen, a connection is made through the virtual host via SQL*Net. To stop Maintenance Mode for a database, simply click the Stop Maintenance button. If the database is down when Stop Maintenance is clicked, it will be started on the Primary. If the database is already running, probing resumes. Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 2: Operations 41 Delete a Virtual Oracle Service To delete a Virtual Oracle Service, click on Delete Oracle Service. A dialogue will appear for verification. Deleting a Virtual Oracle Service does not affect the database itself. The virtual host and service monitor are removed, but the database remains intact. It will be shut down during this operation. The entries for the service will remain in the SQL*Net configuration files.
Chapter 2: Operations 42 The mxdb Command Line Interface In addition to the -c option, which invokes the GUI, the mxdb command can be used for operations. The command is invoked as follows: $ /opt/polyserve/mxdb_oracle_ha/bin/mxdb The mxdb command includes options that enable Database Administrators to perform the same operations that are available in the MxDB-Oracle-HiAv GUI. The following table describes the command arguments. Option Procedure -d This argument is required.
Chapter 2: Operations 43 Multiple Oracle Homes The GUI and CLI can perform operations on any Oracle Home for either Oracle9i or Oracle10g. The oracle user can login, execute the GUI, and select an Oracle9i or Oracle10g home to perform operations on without care for environment variables. Likewise, with the mxdb command, the -d and -h options can be used to operate upon any database in the system as long as the user executing the command is in the dba group.
Chapter 2: Operations 44 Custom Virtual Oracle Service Start/Stop and Probe Scripts MxDB-Oracle-HiAv ships with default probe, start and stop scripts that generally suffice for production usage. However, Oracle is a complex product and Oracle IT shops vary widely in their usage of the product. To that end, it is nearly impossible to provide a default set of start, stop and probe scripts that meet the needs of all sites.
Chapter 2: Operations 45 values are naturally returned from select statements executed against the internal V$ views of the database. The default probe script (mxdb_probe.sql) already includes this infrastructure. Be advised that copying the mxdb_probe.sql script to the ${ORACLE_BASE}/admin//scripts directory will cause the scripts to go into immediate use if there is a Virtual Oracle Service for that database executing.
Chapter 2: Operations 46 tolerance for a long-running SQL*Plus session executing as a part of the probe. It is, in essence, the amount of time allotted to your custom probe actions before the solution pack considers your probe actions stalled. The default probe actions are very light so the default setting of this parameter is 20 seconds. If significant actions are added to a custom probe, this value must be set to accommodate the extra processing time.
Chapter 2: Operations 47 should not be a concern because any process spawned by the supplemental script will likely be daemon in nature (e.g., Intelligent Agents). That is, spawning daemon processes such as the Intelligent Agent is a nearly immediate task. • SHELL. Supplemental scripts will be interpreted by the bash shell. • Execution Environment.
Chapter 2: Operations 48 After implementing these steps, the Intelligent Agent will be started each time the PILOT database is started. The sample script can be renamed to support starting the Intelligent Agent for any database under MxDB-Oracle-HiAv control. Add Code to the Supplemental Script It is completely up to the administrator to determine any other supplemental startup code that should be implemented in the script.
Chapter 2: Operations 49 To configure Data Guard Standby start/stop and probe scripts, change directories to ${ORACLE_BASE}/admin//scripts and copy the reference scripts from the /opt/polyserve/mxdb_oracle_ha/lib directory to your current directory. Next, create symbolic links as shown in the following example: $ cd $ORACLE_BASE/admin/${ORACLE_SID}/scripts $ cp /opt/polyserve/mxdb_oracle_ha/lib/DG*sql . $ ls -lgG total 16 -rwxr-xr-x 1 93 Apr 13 15:01 DG_PHYS_STANDBY_mxdb_abort.
Chapter 2: Operations 50 solution pack that the SQL*Net entries already exist. The following example shows a portion of a listener.ora file for a Data Guard Standby database with DB_UNIQUE_NAME and SERVICE_NAMES assignments of SEATTLE. The HOST assignment of seattle.pdx.polyserve.com is a Virtual Oracle Service VHOST. SEATTLE = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = seattle.pdx.polyserve.
3 Oracle Enterprise Manager Configuration Issues with Failover High Availability Special configuration is required to take advantage of advanced Enterprise Manager features such as Automatic Discovery and the Enterprise Manager Job System in a failover High Availability environment. Incidentally, configuring a shared Oracle Home offers an increased level of availability with regard to submitted jobs. In a traditional Failover HA scenario, submitted jobs are lost during transition.
Chapter 3: Oracle Enterprise Manager 52 Configuration Tutorial The following eight steps provide a tutorial and example of setting up Autodiscovery and persistent Enterprise Manager Jobs. Step 1. In accordance with Metalink Note 190026.1, establish a fixed value for the Agent dbsnmpkey. Place this value in a file of your choosing in the shared Oracle Home (e.g., $ORACLE_HOME/network/agent/dbsnmpkey.txt). Without this key file, the agent file objects created by job submission (*.
Chapter 3: Oracle Enterprise Manager the file was the assignment of the DBSNMP.HOSTNAME parameter. Intelligent Agent inserts the other content once it executes. Verify that the Net Services files contains a HOST assignment that precisely matches the value given to the DBSNMP.HOSTNAME in the snmp_rw.ora file. The following screen shows an example of using the grep(1) command to validate file contents. Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 3: Oracle Enterprise Manager 54 Step 3. Start the Intelligent Agents using the following syntax: $ agentctl start agent password_file= NOTE: The Supplemental Script that ships with MxDB-Oracle-HiAv performs this step during Virtual Oracle Server startup. It is also a good source of reference for manually testing the infrastructure. The following screen shows the output on the sample system. Step 4.
Chapter 3: Oracle Enterprise Manager Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 3: Oracle Enterprise Manager Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 3: Oracle Enterprise Manager 57 • Step 2 of 2 shows “node” Q9DEV.pdx.polyserve.com being discovered. • A green check mark signifies that Q9DEV.pdx.polyserve.com has been discovered. Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 3: Oracle Enterprise Manager Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 3: Oracle Enterprise Manager 59 Step 5. Once the Virtual Oracle Service has been discovered, the Enterprise Manager Job System can be tested to ensure that the setup is correct. The following seven screen shots depict a test job being submitted to the sample MxDB-Oracle-HiAv Virtual Oracle Service called Q9DEV.pdx.polyserve.com. Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 3: Oracle Enterprise Manager Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 3: Oracle Enterprise Manager Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 3: Oracle Enterprise Manager Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 3: Oracle Enterprise Manager 63 Step 6. Once the job has been submitted, the MxDB-Oracle-HiAv failover will preserve the integrity of the submitted job since the architecture is based upon a cluster filesystem. The next step is to use the MxDB-Oracle-HiAv GUI to rehost the database to another node. In the example, the Q9DEV.pdx.polyserve.com Virtual Oracle Service will be rehosted from 10.10.180.1 to 10.10.180.2. Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 3: Oracle Enterprise Manager When the green “Running” indicator appears, the ping(1) command is used to establish that the Intelligent Agent is running on the newly rehosted server, in this case 10.10.180.2 Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 3: Oracle Enterprise Manager 65 Step 7. After rehosting (or failover), an Enterprise Manager Agent ping can be used to ensure that auto discovery was successful. The following Agent ping establishes that auto discovery is working in the MxDB-Oracle-HiAv environment as seen in the following screen shots. Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
Chapter 3: Oracle Enterprise Manager 66 Step 8. Autodiscovery is only part of the value add, however. The most important issue is the persistence of Enterprise Manager jobs. Complying with Metalink Note 190026.1 and combined with the Matrix Server cluster filesystem, the job submitted earlier in the example is still scheduled to run even though the Q9DEV.pdx.polyserve.com Virtual Oracle Service was failed over from its primary to its secondary server. Copyright © 1999-2007 PolyServe, Inc.
Chapter 3: Oracle Enterprise Manager Copyright © 1999-2007 PolyServe, Inc. All rights reserved.
