Deployment Guide: Oracle on Microsoft Windows and the Dell PowerEdge 6850 Server By: Bryan Thomas and Larry Pedigo Performance Tuning Corporation
Introduction Migrating traditional Oracle databases to Oracle Real Application Clusters has become an emerging trend in the IT field. The reasons are clear: Oracle RAC offers a way to obtain high availability and high performance benefits at a relatively low cost. Dell PowerEdge servers are one of the most popular platforms for Oracle RACimplementations. Dell is now introducing the PowerEdge 6850 server, which should soon become the platform of choice for mission-critical Oracle RAC implementations.
Of course, any solution with this much cutting-edge technology is going to have a fair amount of inherent complexity. This can be discouraging to someone attempting a new Oracle RAC deployment. However, this paper focuses on the areas where IT staff needs the most help: configuring the cluster hardware, configuring the shared storage, and navigating through the maze of software and patches that need to be installed.
Oracle Data Provider for .NET (ODP.NET) – .NET data provider designed exclusively by and for Oracle for optimal data access from .NET to Oracle 9i Database and Oracle Database 10g. Can be used from any .NET language, including C# and Visual Basic .NET. Microsoft Windows Server offers a powerful platform for deploying an Oracle database, particularly an Oracle RAC database. Windows Server Editions offer all of the scalability and flexibility that you need for developing a robust production database.
However, preparing the Windows OS for Oracle involves more than simply popping the CD in and clicking through a few menus. One of the key problems is all of the drivers and hardware support required for an Oracle RAC installation. For example, all RAC implementations require shared external storage. A variety of BIOS settings, drivers, and support software installations are required to implement shared storage on a SAN. If any of the software versions are out of synch, the San may simply not function.
The order that the server’s NIC cards (and IP addresses) are defined to Windows is important. Windows must recognize the public IP address as the first address. Otherwise, any Oracle utility that uses “gethostbyname” (such as Oracle Enterprise Manager) might identify the wrong host. In Windows 2000, to check (and, if necessary, fix) the order that NICs are defined in: 1. Choose Start -> Settings -> Network and Dial-up Connections. 2. Choose Advanced -> Advanced Settings. 3.
Dell Minimum Hardware List and System Requirements Oracle provides some general guidelines for minimum hardware and software requirements. These include: • External shared storage • At least 256 MB of RAM on each node (512 MB preferred) • Oracle supported clusterware. In Oracle9i, Oracle supports a variety of OS dependent clusterware on different platform However, for Windows Server, Oracle provides the clusterware.
Cluster Hardware Considerations for Oracle RAC This section covers some of the issues that you should consider when choosing hardware for your Oracle RAC system. Two of the key issues are how to choose the right RAID level for your shared storage, and how to choose the right amount of memory for your RAC servers. RAID Systems Managing a large number of individual disk drives can be very difficult, since you must balance the database files across all of these disk drives in order to spread out the I/O load.
RAID 5 RAID 5 uses parity for fault tolerance. The advantage of using parity is that instead of having to double the number of disk drives in the system, you only have to add one disk drive to store the parity. RAID 5 uses parity, but distributes the parity among all of the disk drives in the RAID volume. RAID 5 is very popular because it provides a fault tolerant solution at a relatively low cost.
Recommendations for Oracle Component Placement Oracle is sensitive to read performance and sensitive to write performance on the redo log files and on the archive log files. Thus, the following recommendations are given. OS Volume - The OS should be installed on a RAID 1 disk volume. It is important that you do not need to restore/rebuild the OS in the event of a disk failure. This can be very time consuming and expensive.
Another technique is available to address memory above 4 GB. The Address Windowing Extensions interface (AWE) allows access to RAM up to 64 GB. AWE is implemented through Physical Address Extensions, which are enabled by setting the /PAE flag in the boot. ini file. In addition, the account that runs Oracle must have the “Lock memory” privilege set under Local Security. The “USE_INDIRECT_BUFFERS=TRUE” must also be set in the Oracle initialization file.
But in general, please take into consideration that memory requirements per instance are reduced when the same user population is distributed over multiple nodes. In this case: Assuming the same user population N number of nodes M buffer cache for a single system then (M / N) + ((M / N )*0.10) [ + extra memory to compensate for failed-over users ] Thus for example with a M=2G & N=2 & no extra memory for failed-over users =( 2G / 2 ) + (( 2G / 2 )) *0.
Reliability, Availability and Serviceability Features The Dell PowerEdge 6850 server is designed from top to bottom for High Availability. There are a number of features of the PowerEdge 6850 server that enhance Reliability, Availability and Serviceability.
The Dell PowerEdge 6850 server is capable of running in several different modes, due to the EM64T processor. There two major modes: IA32e mode and Legacy mode. IA32e mode is automatically invoked when a 64-bit OS is booted. Legacy mode is automatically invoked when either a 32-bit OS or 16-bit OS is booted. IA32e mode is based on the existing IA32 mode, with 64-bit extensions. Under IA32e mode, two different sub-modes exist: 64-bit mode and Compatibility mode.
Installing and Configuring the Dell | EMC SAN or PowerVault Shared Disk Array Installing and configuring a Dell | EMC SAN is a discipline unto itself. It requires training and specific storage domain knowledge. Fortunately, most of the “heavy lifting” is usually performed by Dell Professional Services. After the initial configuration, management becomes much more straightforward, and it is relatively easy to manage these storage arrays with the Navisphere Manager interface.
For a two-node cluster, you may directly attach to a CXx00 storage array with fiber cables rather than using a switch. This requires at least two HBAs and associated cables per server. In addition, PowerPath software is required for these implementations. Dell PowerVault Installation Steps A more common option for Direct Attached Storage on two-node clusters is a Dell PowerVault 22xS SCSI storage array. This provides a relatively low-cost option for simple clusters.
Oracle10g and ASM All of the recommendations listed above apply to Oracle9i or Oracle10g utilizing the Oracle Cluster File System. However, if the Automatic Storage Management option is used for Oracle10g, the rules may need to be modified. With ASM, Oracle software can take on the responsibility for managing disk mirroring and striping. You could go as far as giving Oracle a group of individual raw disks without any hardware RAID.
It is even more important to properly configure the private network. The private network is used as an interconnect between each cluster node. This network consists of Ethernet cables and one or more network switches. The major role of the private interconnect is to implement Oracle’s “cache fusion” technology. This is not a simple heartbeat mechanism, as in other types of clusters.
If you choose to utilize Intel NICs for teaming, perform the following configuration steps: 1. Install Intel PROSet II (if necessary) 2. Start Intel PROSet II as described in Using PROSet II 3. Run the Teaming Wizard to add adapters to each team. 4. See the PROSet II Help for additional information. Be aware that Oracle 10g requires the user to select the correct network interface during installation of the database software.
To configure shared disks, do the following: 1. On the first node only, log in to windows as the Local Administrator. 2. Navigate to Start -> Administrative Tools -> Computer Management. 3. Expand the Storage folder to Disk management. 4. Right-click the unallocated region of a basic disk and choose Create Partition (dynamic disks are not supported). 5. In the Create Partition wizard, choose Next -> Extended Partition (Primary partitions are not supported for Logical Drives). 6. Choose Extended Partition.
To configure shared disks, do the following: 1. On the first node only, log in to windows as the Local Administrator. 2. Navigate to Start -> Administrative Tools -> Computer Management. 3. Expand the Storage folder to Disk management. 4. Right-click the unallocated region of a basic disk and choose Create Partition (dynamic disks are not supported). 5. In the Create Partition wizard, choose Next -> Extended Partition (Primary partitions are not supported for Logical Drives). 6. Choose Extended Partition.
Shared Disk Configuration On windows 2003 (32 and 64 bit versions), the user must enable the automatic mounting of new volumes on each node in the cluster. • The default value for auto mounting was changed from enabled in Windows 2000 to disabled in Windows 2003, so this additional step is required. • Auto mount should be enabled before creating the new partitions and must be done on all nodes in the cluster. A reboot is required for the value to take hold. diskpart.
Note: As of October 2004, the 9.2.0.6 patchset is available for Windows. However, the 9.2.0.6 clusterware patch is not yet available as of this publication. In the interest of offering a fully tested RAC configuration, this paper will cover a consistent upgrade to all 9.2.0.5 components. Those wishing to upgrade the database components to 9.2.0.6 instead of 9.2.0.5 should read the release notes for the 9.2.0.6 patchset for exceptions to this procedure.
Note: As of October 2004, the 9.2.0.6 patchset is available for Windows. However, the 9.2.0.6 clusterware patch is not yet available as of this publication. In the interest of offering a fully tested RAC configuration, this paper will cover a consistent upgrade to all 9.2.0.5 components. Those wishing to upgrade the database components to 9.2.0.6 instead of 9.2.0.5 should read the release notes for the 9.2.0.6 patchset for exceptions to this procedure.
Run the 9.2.0.5 Cluster Setup Wizard You are now ready to run the first step in the install process. Since this step does not use the Oracle Universal Installer, you may install from one node without difficulty. You have the option of choosing a shared directory for software installation or local directories. If local directories are chosen, the Cluster Setup wizard will automaticallycopy the files to all other nodes. Use the following procedure: 1. Open a command prompt window. 2.
The Cluster Wizard will create only one OCFS drive for data files. If you would like to create additional OCFS drives from the logical partitions you created earlier, do the following: 1. Use Windows Disk Management tool to assign the unformatted logical partition a driver letter. • Start disk Manager, locate the logical partition you would like to format for OCFS, and right click inside the partition. Select “Change Drive Letter and Paths...” In the “Change Drive Letter and Paths for..
Installing Oracle10g Cluster Ready Services In Oracle10g, the Clusterware has been placed on a separate CD which installs Cluster Ready Services. You must install from this disk before installing from the Oracle10g Database installation CD. In addition, you must complete some pre-installation steps before starting the Cluster Ready Services installation. Pre-installation Steps Before installing Cluster Ready Services, you must create logical volumes on shared disk.
3. In the “Language Selection Page”, select the language or languages for your CRS installation. Click “Next”. 4. In the “Cluster Configuration Page”, enter a globally unique cluster name. Do not use special characters such as (, ), !, @, #, %, ^, &, and *. Also enter a public and a private node name for each node, without any domain qualifiers. Use the primary and private entered in the hosts file in an earlier step. Click “Next”. 5.
Cluster Ready Services are now installed, and you are ready to install Oracle10g software. Install Oracle Database Software Whether you are installing Oracle9i or Oracle10g, completion of the Cluster Software installation step makes it possible to perform an Oracle database software installation that is RAC enabled. Since Cluster Services are started by default at the end of Cluster Software installation, the OUI will be aware of all cluster nodes.
11. Software installation may take an hour or more. To successfully complete the software installation, perform the following additional steps: a. Installation progress may appear to be hung while operations are occurring on remote nodes. To check progress, check for I/O activity on each node, or check activity in Task Manager on each node.
11. Software installation may take an hour or more. To successfully complete the software installation, perform the following additional steps: a. Installation progress may appear to be hung while operations are occurring on remote nodes. To check progress, check for I/O activity on each node, or check activity in Task Manager on each node.
Oracle 9.2.0.5 Patch Installation Steps To install the Oracle 9.2.0.5 patchset, re-launch the OUI: 1. Open a command prompt window. 2. Navigate to the E:\oracle_stage\9205\Disk 1 directory. 3. Run setup.exe. 4. In the “OUI Welcome screen”, click “Next”. 5. In the next screen, check that the “Source Path” is correct. Set the “Destination” field to the Oracle 9.2 home directory. 6. In the “Cluster Node Selection” screen, highlight all nodes {if installing on individual nodes, only select the current node). 7.
4. Patch the OLM files by copying all of the following files from the E:\oracle_stage\ocfs9205\3575688\Disk1\preinstall_rac\olm directory to both of the following directories: a. Directories: %ORACLE_HOME%\bin C:\WINNT\System32\osd9i\olm b. Files: crlogdr.exe DeleteDisk.exe ExportSYMLinks.exe GUIOracleOBJManager.exe ImportSYMLinks.exe LetterDelete.exe LogPartFormat.exe OracleObjManager.exe OracleObjService.exe oraoobjlib.dll readme.txt 5.
Installing Oracle10g Database Software While installing Oracle10g database software, you have the option of creating a Starter Database. In this example, creation of the Starter Database will be delayed, in order to explain the technique of creating a database at any time outside of the software installation routine. Use the following procedure to perform a software-only install: 1. Insert the Oracle Database10g Release 1 (10.1.0.2) CD-ROM. Either the autorun.
Configuring the Oracle Listener Before creating a database, it is necessary to first create the Listener service for Oracle Net. To create a database in Cluster Mode, both local and remote entries in the listener.ora and tnsnames.ora file must be pre-created. The remote listener must be named LISTENERS_SIDprefix (i.e. LISTENERS_MYDB1) and the local listeners should be named LISTENER_SID (i.e. LISTENER_MYDB1 and LISTENER_MYDB2).
Under Oracle9i RAC, several issues have been noted with DBCA. If you follow the instructions below, you should finish database creation without any problems. Nevertheless, it may prove helpful to create a trace file for the DBCA installation. In case of a problem, the trace file will help you determine the cause. To enable tracing, edit the file $ORACLE_HOME\bin. Edit line 40 to read like: “E:\Program Files\Oracle\jre\1.1.8\BIN\JRE” -DTRACING.ENABLED=true -DTRACING.
11. In the “Creation Options” page, choose at least the “Create Database” option. You may also choose the “Create Template” and “Save as a Script” options. Click “Finish”. 12. The “DBCA Summary” page is displayed. Review the information, and then click “OK”. Database creation progress is displayed. Database creation may seem to hang at around 95% to 100%. This is usually a pause while remote instances are created. 13.
6. In the “Database Identification” page, enter a “Global Database Name” and an “Oracle System Identifier (SID) Prefix”. The Global Database Name is usually of the form name.domain. It is a Best Practice to make the “name” equal to the SID. The SID should be eight characters or less. 7. In the “Management Options” page, choose the “Enterprise Manager” option. If you wish to use e-mail notifications, provide the outgoing mail server and e-mail address.
15. In the “Database Storage Window”, tablespace names, file names and sizing parameters may be edited, along with redo log information. Edit the file names, locations, and sizing as desired. Check the redo logs for the correct thread numbers (one thread set of redo logs per node). After you have finished editing, click “Next”. 16. In the “Creation Options” page, choose at least the “Create Database” option. You may also choose the “Create Tem plate” and “Save as a Script” options. Click “Finish”. 17.
Post-Installation Tasks There are some post-installation tasks that are common to Oracle9i and Oracle10g installations. These tasks are covered below. First, verify that all services start properly after booting the servers. Use the following steps: 1. Reboot each node. 2. Select Start -> Settings -> Control Panel -> Administrative Tools -> Services. 3. Verify that all required Oracle services are started. Start any required services that are stopped. 4.
Summary Many companies are currently interested in deploying Oracle Real Application Clusters on the Microsoft Windows Server platform. Microsoft Windows Server offers a powerful platform for corporate databases that may also be managed with skill sets readily available within most IT departments. Oracle Real Application Clusters offers a flexible mechanism for achieving scalability with relatively inexpensive Intel-based hardware, without the need for large, expensive SMP servers.
References “Configuring Disks in Windows 2000”; Oracle MetaLink Note 109711.1; Oracle Corporation, November, 2002; http://metalink.oracle.com/metalink/plsql/ml2_documents.showFrameDocument?p_database_id=NOT&p_id=109711.1 “CRS and10g Real Application Clusters”; Oracle MetaLink Note 259301.1; Oracle Corporation, May, 2004; http://metalink.oracle.com/metalink/plsql/ml2_documents.showFrameDocument?p_database_id=NOT&p_id=259301.1 “DBCA Tips and Pitfalls in a Windows RAC Environment”; Oracle MetaLink Note 232239.
“Oracle9i Database Platform Guide, Release 2 (9.2) for Windows”, Part No. B10163-01; Primary Author: Craig B. Foch; Contributing Author: Mark Kennedy and Helen Slattery; Contributor: David Collelo; Oracle Corporation, 2003 “Oracle9i Net Services Administrator’s Guide, Release 2 (9.2)”, Part No.
