Red Hat Enterprise Linux 5 Configuration Example - Oracle HA on Cluster Suite Configuring Oracle for High Availability (HA) on Red Hat Cluster Suite Edition 1 Landmann
Red Hat Enterprise Linux 5 Configuration Example - Oracle HA on Cluster Suite Configuring Oracle for High Availability (HA) on Red Hat Cluster Suite Edition 1 Landmann rlandmann@redhat.
Legal Notice Copyright © 2010 Red Hat Inc.. T his document is licensed by Red Hat under the Creative Commons Attribution-ShareAlike 3.0 Unported License. If you distribute this document, or a modified version of it, you must provide attribution to Red Hat, Inc. and provide a link to the original. If the document is modified, all Red Hat trademarks must be removed.
Table of Contents Table of Contents .Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4. . . . . . . . . . 1. About T his Guide 4 2. Audience 4 3. Related Documentation 4 4. Feedback 4 5. Document Conventions 5 5.1. T ypographic Conventions 5 5.2. Pull-quote Conventions 6 5.3. Notes and Warnings 7 .Chapter . . . . . . . . 1. . . .Overview . . . . . . . . . . . . .
Red Hat Enterprise Linux 5 Configuration Example - Oracle HA on Cluster Suite 5.1. Red Hat Cluster Suite HA 5.2. Red Hat Cluster Suite T imers 5.3. RGManager Configuration 5.3.1. Customizing oracledb.sh Environment Variables 5.3.1.1. DB_PROCNAMES 5.3.1.2. LSNR_PROCNAME 5.3.2. Network VIP for Oracle Listeners 5.3.2.1. listener.ora Configuration 5.3.3. Files System Entries 37 37 38 38 38 39 39 40 41 . . . . . . . . .cluster.conf Sample . . . . . . . . . . . . File . . . . . . . . . . . . . . . . . . . . .
Table of Contents 3
Red Hat Enterprise Linux 5 Configuration Example - Oracle HA on Cluster Suite Introduction 1. About This Guide T his manual provides a step-by-step installation of Oracle for High Availability (HA) using the Red Hat Advanced Platform product, Cluster Suite. T his manual presents both the simple RDBMS Enterprise Edition failover scenario, as well at Oracle RDBMS Real Applications Clusters (RAC) on Shared GFS file systems.
Introduction If you have a suggestion for improving the documentation, try to be as specific as possible. If you have found an error, please include the section number and some of the surrounding text so we can find it easily. 5. Document Conventions T his manual uses several conventions to highlight certain words and phrases and draw attention to specific pieces of information. In PDF and paper editions, this manual uses typefaces drawn from the Liberation Fonts set.
Red Hat Enterprise Linux 5 Configuration Example - Oracle HA on Cluster Suite T o insert a special character into a gedit file, choose Applications → Accessories → Character Map from the main menu bar. Next, choose Search → Find… from the Character Map menu bar, type the name of the character in the Search field and click Next. T he character you sought will be highlighted in the Character T able. Double-click this highlighted character to place it in the T ext to copy field and then click the Copy button.
Introduction static int kvm_vm_ioctl_deassign_device(struct kvm *kvm, struct kvm_assigned_pci_dev *assigned_dev) { int r = 0; struct kvm_assigned_dev_kernel *match; mutex_lock(&kvm->lock); match = kvm_find_assigned_dev(&kvm->arch.
Red Hat Enterprise Linux 5 Configuration Example - Oracle HA on Cluster Suite Chapter 1. Overview T his manual provides a step-by-step installation of Oracle for High Availability (HA) using the Red Hat Advanced Platform product, Cluster Suite. T his manual provides installation instructions for the following two scenarios: Simple RDBMS Enterprise Edition failover Oracle RDMBS Real Applications Cluster (RAC) on shared GFS file systems A sample two-node cluster is provided for both installation types.
Chapter 1. Overview see the kbase article “Red Hat Support for Oracle Enterprise Edition and Cold Failover Cluster Suite configurations”: http://kbase.redhat.com/faq/docs/DOC-21631. 1.1.2. Oracle Real Application Clusters for Red Hat Cluster Suite and GFS Oracle Enterprise Edition has a separately priced option called Real Application Clusters (RAC), and this does provide for shared access, or multi-instance, shared operation.
Red Hat Enterprise Linux 5 Configuration Example - Oracle HA on Cluster Suite Figure 1.1. Sample T wo-Node Oracle Cluster Figure 1.2, “Cluster Node Connections” shows a generalized summary of the connections for each node in the configuration. Each node is connected to a public network, to a private network, and to shared storage. In addition, each node is configured with a fencing device that is also connected to the private network. Figure 1.2.
Chapter 1. Overview Asymmetrical RAC topologies RAC clusters are often configured to be symmetrical; the type of workload presented to the nodes is similar. In this topology, the servers are also of the same relative computing strength. T ypically, the servers are over-configured by 50% in order for the failover node to handle the work of both nodes. However, this assumes that the business requirements for degraded operation are identical to normal operation, which is not always the case.
Red Hat Enterprise Linux 5 Configuration Example - Oracle HA on Cluster Suite RAID technology T here seem to be more RAID options than ever before. A simple thing to remember for databases is that, on average, a 144GB 15K drive is the same speed as a 36GB 15K, so if you factor for IOPS throughput, you don’t need to worry about space. RAID5 is often used as a speed/space compromise, but is very slow especially for random writes, which databases do a lot.
Chapter 1. Overview zones to regulate the use of the array. It is recommended that the all the FCP ports from the servers be isolated into a zone with the access ports (or HCAs) on the storage array. Although many customers use large storage arrays and then sub-divide their use amongst many consumers in the datacenter, we would not recommend this for Oracle clusters. Production Oracle databases tend to place great demands on storage arrays, and sharing them will only complicate the deployment.
Red Hat Enterprise Linux 5 Configuration Example - Oracle HA on Cluster Suite Note T he Oracle Cache Fusion links can run over standard UDP Ethernet (GbE links, at least), but can also run over a proprietary RDMA Infiniband network, and is fully supported by Oracle on RHEL, starting with RDBMS version 10.2.0.3. T he Oracle GCS protocol functions like many other conventional cache protocols and must broadcast to all nodes when requesting a cache block. On a hardware bus, this is called Snooping.
Chapter 2. Hardware Installation and Configuration Chapter 2. Hardware Installation and Configuration A Cluster is a complex arrangement of bits and pieces that, once combined with the software configuration, produces a highly available platform for mission critical Oracle databases. T he hardware configuration requires some knowledge of the application, or at a minimum, its expectation of performance.
Red Hat Enterprise Linux 5 Configuration Example - Oracle HA on Cluster Suite and load balancing. Host-based RAID operations compete for resources and could compromise throughput on the database server. Many storage topologies include FCP switch infrastructure and this can be used to isolate the I/O traffic to the array. We recommend that the storage array HCAs and the four ports of the two HBAs all be placed in one zone. For more information on HBA configuration see Section 3.2.
Chapter 2. Hardware Installation and Configuration Tip During a log switch, the previously closed log is copied to the archive destination, and is usually not throttled. T his can impact transaction commit response times. One of the simplest ways to mitigate this effect is to place the Archive Log destination on DIO-enabled NFS mount, and the network connection be forced to 100T X. T his is the easiest way to throttle archive log copies.
Red Hat Enterprise Linux 5 Configuration Example - Oracle HA on Cluster Suite within application servers must be configured correctly, so failover delay is minimized. T he backside network is a private, dedicated network that should be configured as a four-port VLAN, if a non-private switch is used. Most customers buy dual-ported NICs, which are not as reliable as two single-ported NICs.
Chapter 2. Hardware Installation and Configuration Oracle Clusterware. T he fencing methods that we recommend for use with Oracle databases, are all power-based, and have been in the Red Hat Cluster Suite for several releases. Mature, power-based fencing methods are, indeed, the foundation of any robust cluster. Most T ier 1 server vendors provide built-in baseboard management controllers (BMC), but they are called many things (HP iLO, Dell DRAC, Sun ILOM).
Red Hat Enterprise Linux 5 Configuration Example - Oracle HA on Cluster Suite Chapter 3. Software Installation and Configuration A Cluster is a complex arrangement of bits and pieces that, once combined with the software configuration, produces a highly available platform for mission critical Oracle databases. We probably can’t repeat that often enough, but complexity is public enemy #1. Clusters, by definition, are complex.
Chapter 3. Software Installation and Configuration device scsi cciss device scsi qla2300 install nfs --server=192.168.1.212 --dir=/vol/ed/jneedham/ISO/RHEL5/U3/64 reboot yes lang en_US.UTF-8 keyboard us network --device eth0 --bootproto=static --device=eth0 --gateway=192.168.1.1 -ip=192.168.1.1 --ip=192.168.1.114 --nameserver=139.95.251.1 -netmask=255.255.255.
Red Hat Enterprise Linux 5 Configuration Example - Oracle HA on Cluster Suite 3.2. Storage Topology Once the server has all the required software installed, configuration can commence. Configuring the server node to map the external LUN will require some incremental configuration on the server, some incremental configuration on the array and then back to the server to verify that all LUNs were mapped.
Chapter 3. Software Installation and Configuration previous method for gathering WWIDs required the execution of the scsi_id command on each LUN. $ scsi_id -g -s /block/sdc #External LUN, returns 360a9800056724671684a514137392d65 $ scsi_id -g -s /block/sdd #External LUN, returns 360a9800056724671684a502d34555579 T he following example of a multipath configuration file shows the Red Hat Cluster Suite quorum disk and, for the RAC/GFS node, the first of three Oracle Clusterware Voting Disks.
Red Hat Enterprise Linux 5 Configuration Example - Oracle HA on Cluster Suite defaults { user_friendly_names udev_dir polling_interval selector path_grouping_policy getuid_callout prio_callout path_checker rr_min_io rr_weight failback no_path_retry user_friendly_name } yes /dev 10 "round-robin 0" failover "/sbin/scsi_id -g -u -s /block/%n" /bin/true tur 100 priorities immediate fail yes Now that the m ultipath.conf file is complete, try restarting the m ultipath service.
Chapter 3. Software Installation and Configuration Tip You may need to change the maximum journal size for a partition. T he following procedure provides an example of changing the maximum journal size of an existing partition to 400MB.
Red Hat Enterprise Linux 5 Configuration Example - Oracle HA on Cluster Suite T he public network is the pathway used by the application tier to access the database. T he failure scenario is the loss of an entire node, so although bonding does provide protection in the event of the public interface failure, this is not as likely. Bonded public interfaces complicate application tier network configuration and failover sequencing. T his network is not bonded in our example.
Chapter 3. Software Installation and Configuration ifcfg-eth2 # Intel Corporation 82546GB Gigabit Ethernet Controller DEVICE=eth2 HWADDR=00:04:23:D4:88:BE MASTER=bond0 SLAVE=yes BOOTPROTO=none TYPE=Ethernet ONBOOT=no ifcfg-eth3 # Intel Corporation 82546GB Gigabit Ethernet Controller DEVICE=eth3 HWADDR=00:04:23:D4:88:BF MASTER=bond0 SLAVE=yes BOOTPROTO=none TYPE=Ethernet ONBOOT=no ifcfg-bond0 DEVICE=bond0 IPADDR=192.168.2.162 NETMASK=255.255.255.0 NETWORK=192.168.2.0 BROADCAST=192.168.2.
Red Hat Enterprise Linux 5 Configuration Example - Oracle HA on Cluster Suite Note You can test the fencing configuration manually with the fence_node command. T est early and often. 3.3.4. Red Hat Cluster Suite services T here are now enough hardware and software pieces in place that the cluster.conf file can be completed and parts of the cluster can be initialized. Red Hat Cluster Suite consists of a set of services (cm an, qdisk, fenced) that ensure cluster integrity.
Chapter 3. Software Installation and Configuration 192.168.1.7 192.168.1.8 rac7-priv.example.com rac8-priv.example.com rac7-priv rac8-priv Tip When doing initial testing, set the init level to 2 in the /etc/inittab file, to aid node testing. If the configuration is broken and the node reboots back into init 3, the startup will hang, and this impedes debugging. Open a window and tail the /var/log/m essages file to track your progress.
Red Hat Enterprise Linux 5 Configuration Example - Oracle HA on Cluster Suite Chapter 4. RAC/GFS Cluster Configuration T his chapter provides information on a configuring RAC/GFS cluster. For information on configuring a cold failover HA cluster, see Chapter 5, Cold Failover Cluster Configuration. Preparing a cluster for RAC requires additional package installation and configuration. Deploying Oracle RAC on a certified GFS cluster requires additional software and configuration.
Chapter 4. RAC/GFS Cluster Configuration Tip T he time a node takes to reboot depends on several factors, including BIOS settings. Many servers scan all of memory and then scan PCI buses for boot candidates from NICs or HBAs (of which there should only be one). Disabling these scans and any other steps in the BIOS that take time, will improve recovery performance. T he grub.conf file often continues a built-in 5-second delay for screen hold. Sometimes, every second counts. 4.1.1.
Red Hat Enterprise Linux 5 Configuration Example - Oracle HA on Cluster Suite PAGE 36Chapter 4. RAC/GFS Cluster Configuration Warning GFS volumes can be grown if the file system requires more capacity. T he gfs_grow command is used to expand the file system, once the LUN has been expanded. By keeping the filesystem mapping to single LUNs, it reduces an errors (or bugs) that might arise during gfs_grow operations. T here is no performance difference between using the DDM inode, or subsequent CLVMD created logical volumes, built on these inodes.
Red Hat Enterprise Linux 5 Configuration Example - Oracle HA on Cluster Suite 2. Create directories that correspond to the hostnames: $ mkdir rac7 $ mkdir rac8 3. Create the admin directory in each directory: $ mkdir rac7/admin $ mkdir rac8/admin 4. Create the CPDN link (from each host). ON RAC7, in $OH/network: $ ln –s @hostname admin On RAC8, in $OH/network: $ ln –s @hostname admin 4.4.
Chapter 4. RAC/GFS Cluster Configuration T he Oracle SGA (Shared Global Area) contains several memory structures that used to improve the performance of the executing SQL. T he largest, and most critical is the db block buffer cache. T his cache typically consumes over 80% of the SGA. Several SQL pools used for results and complex parallel operations consume the next largest block. T he advent of x64 systems make it possible to SGA ranges in the 8-1024GB range.
Red Hat Enterprise Linux 5 Configuration Example - Oracle HA on Cluster Suite Chapter 5. Cold Failover Cluster Configuration T his chapter provides information on configuration a cold failover HA cluster. For information on configuring a RAC/GFS cluster, see Chapter 4, RAC/GFS Cluster Configuration. Long before RAC (and its progenitor, OPS) was suitable for high availability, customers still needed Oracle databases to be more reliable.
Chapter 5. Cold Failover Cluster Configuration members. T his ensures the cluster is viable and that Oracle can continue to operate on the primary node. Some failure cases can cause the heartbeat to become erratic or unreliable, so modern clustering products provide a second check-in mechanism, which insures that quorum is maintained. Quorum voting causes each cluster member to identify itself by voting, in the form of a simple write to a shared vote, or quorum disk.
Red Hat Enterprise Linux 5 Configuration Example - Oracle HA on Cluster Suite 5.3. RGManager Configuration Note At this point in the process, we have installed Oracle on a shared volume disk (i.e. SAN). T he Resource manager is required only in the HA configuration and is responsible for ensuring that the selected node is capable of supporting an Oracle instance.
Chapter 5. Cold Failover Cluster Configuration T here has been some debate over what constitutes a running instance and RGManager uses a list of Oracle background processes (daemons in unix) to test for their existence. Historically, PMON and SMON are the most frequently selected and they are a good set. PMON is the Process Monitor part of the instance, and mostly cleans out latches and locks left by dead processes.
Red Hat Enterprise Linux 5 Configuration Example - Oracle HA on Cluster Suite # Cold Failover VIP 192.168.1.60 hacf-vip 192.168.1.160 192.168.2.160 192.168.2.5 rac5 rac5-priv rac5-jlo 192.168.1.161 192.168.2.161 192.168.2.6 rac6 rac6-priv rac6-jlo 5.3.2.1. listener.ora Configuration T he listener is managed by the rgm anager package, but the functionality is determined by the Oracle configuration file, listener.ora. T he bolded LIST ENER tag in the file is the specific name of this listener instance.
Chapter 5. Cold Failover Cluster Configuration 5.3.3. Files System Entries Oracle single-instance can run on any RHEL-supported filesystem type. T hs is unlike RAC, where only GFS is specifically certified for use. Most customers use EXT 3, but EXT 4, GFS and NFS are supported as well.
Red Hat Enterprise Linux 5 Configuration Example - Oracle HA on Cluster Suite Sample cluster.conf File T his appendix provides a sample cluster.conf file for a two node cold failover configuration with power fencing via an APC power strip.
Revision History Revision History Revision 1-24 .4 00 Rebuild with publican 4.0.0 2013-10-31 Rüdiger Landmann Revision 1-24 Rebuild for Publican 3.0 2012-07-18 Anthony T owns Revision 1.
Red Hat Enterprise Linux 5 Configuration Example - Oracle HA on Cluster Suite F FCP switch infrastructure, Storage T opology feedback, Feedback fenced service, Red Hat Cluster Suite services, GFS Configuration fence_node command, Fencing Network fencing, Fencing T opology - configuration, Fencing Configuration, Fencing Network - power-managed, Cold Failover Cluster Configuration - technologies, Fencing Configuration file system - blocks-based, Storage Considerations - files-based, Storage Considerations -
Revision History IP routing, virtual, RAC/GFS Considerations iSCSI technology, Storage T opology and DM-Multipath J journal size, file system, qdisk Configuration K kickstart file, RHEL Server Base L license - Enterprise Edition, RAC/GFS Considerations - Real Application Clusters (RAC), RAC/GFS Considerations Link Aggregation Control Protocol (LACP), RAC/GFS Considerations listener.ora configuration file, listener.
Red Hat Enterprise Linux 5 Configuration Example - Oracle HA on Cluster Suite Oracle Enterprise Manager (OEM) console, Customizing oracledb.sh Environment Variables oracledb.
Revision History Reliable Data Sockets (RDS) protocol, RAC/GFS Considerations rgmanager package, Red Hat Cluster Suite HA RHEL server bse, RHEL Server Base RPM groups, installation, RHEL Server Base S scsi_id command, Multipath Configuration SELINUX, RHEL Server Base Serial Access SCSI (SAS) drives, Storage Considerations Serial AT A (SAT A), Storage Considerations, Storage T opology Serviceguard, HP, Oracle Enterprise Edition HA for Red Hat Cluster Suite SGS_T ARGET tuning variable, Oracle Settings and S
