HP StorageWorks HP Scalable NAS File Serving Software provisioning guide for Oracle HP Scalable NAS 3.
Legal and notice information © Copyright 2006, 2009 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor's standard commercial license. The information contained herein is subject to change without notice.
Contents About this guide ................................................................... 6 Intended audience ............................................................................................... HP technical support ............................................................................................ Subscription service ............................................................................................. HP websites .............................................................
NFS mount options ............................................................................................ 33 Oracle server operating system parameters ........................................................... 33 4 Oracle DBA: managing Oracle databases in a NAS environment ....................................................................... 35 Configuring an Oracle database for HP Scalable NAS .......................................... NFS client I/O ...............................................
Figures 1 HP Scalable NAS Management Console mount dialog ......................... 18 2 DB Optimized filesystem mount operation ........................................... 19 3 NFS Export Group options selection for Oracle Base, Oracle CRS, and datafiles ......................................................................................... 21 4 Associating Export Groups with NAS heads ....................................... 23 5 Management Console showing four NAS heads and three Virtual NFS Services ..
About this guide This guide provides information about deploying Oracle on an HP Scalable NAS cluster. Intended audience This guide is intended for NAS administrators, system administrators, and Oracle DBAs. HP technical support For worldwide technical support information, see the HP support website: http://www.hp.
HP websites For additional information, see the following HP websites: • http://www.hp.com • http://www.hp.com/go/scalablenas • http://www.hp.com/go/storage • http://www.hp.com/service_locator • http://www.hp.com/support/manuals Documentation feedback HP welcomes your feedback. To make comments and suggestions about product documentation, please send a message to storagedocsFeedback@hp.com. All submissions become the property of HP.
About this guide
1 Introduction For several years Network Attached Storage (NAS) has been rapidly evolving into an acceptable storage option for Oracle databases. It is now by Oracle’s admission a practical deployment option for Oracle databases.1 Filesystems needed for Oracle install files (Oracle Home) or Oracle data files can be located on a remote NFS (Network File System) server. Access to the filesystem is a simple matter of mounting it using NFS before starting the database.
according to the requirements of the database. No more interaction is needed with the system or storage administration groups. Contrast this to the amount of system administrative overhead when deploying Oracle databases on raw partitions (or even filesystems) in a SAN. When one provisions storage using simple raw datafiles, or ASM or even iSCSI, there is a significant amount of administrative overhead.
NAS server to another, the filesystems remain accessible without remounting and, most importantly, processes with active file handles accessing files in the NFS filesystems are not impacted during server transitions. The VNFS technology is important for two reasons: • Transition rather than failover. If a HP Scalable NAS server fails, the Virtual NFS Services running on the failed node will be transparently transitioned to a backup NAS server.
As more resources are needed, additional servers can be added. Then filesystems can be transparently transitioned to be presented by the new server, offloading any over-burdened NAS heads. File sharing is not limited to Oracle files. Regular file serving can be mixed with database access within the same cluster. Unique benefits for Oracle deployment The filers powered by HP Scalable NAS are a radically different NAS technology implemented using 64-bit servers connected to shared SAN storage.
S.A.M.E. (Stripe and Mirror Everything). SAME is Oracle Corporation’s recommended storage layout method.2 HP Scalable NAS also offers graphical and command-line management interfaces for central management of the entire NAS cluster. All the Oracle DBA needs to do is mount the exported storage on the Oracle server and get to work. • Standards. The NAS heads used in the cluster are commodity Linux servers—no proprietary hardware is required.
Introduction
2 NAS Administrator: deploying HP Scalable NAS for use with Oracle First, it is important for the NAS administrator to be familiar with the administration of the HP Scalable NAS cluster. For information about installing HP Scalable NAS products, see the HP Scalable NAS File Serving Software installation guide. No added software module is needed to support Oracle.
1MB on most Oracle ports; on a few ports such as Solaris, even larger transfers are supported. In general, a 4MB volume stripe width is a good choice. The LUNs in the array should be mirrored (RAID1) or at least parity protected (RAID5 or RAID6, for example). A mirrored LUN striped with the volume manager achieves the optimal S.A.M.E. (Stripe and Mirror Everything) data placement methodology recommended by Oracle Corporation.
files must be located in a filesystem mounted with the DB Optimized option. Although oradata is typically found in the path [$ORACLE_BASE|$ORACLE_HOME]/ oradata, we will create a soft link from that location (found on a Standard File Optimized filesystem) to a DB Optimized filesystem location. In the following example, the DB Optimized filesystem mount point on the Oracle server (NFS client) is /u02 and we have created the subdirectory /u02/oradata that is owned by the Oracle user.
Figure 1 HP Scalable NAS Management Console mount dialog DB Optimized mount option Figure 2 shows a database optimized filesystem that is being mounted on tmr6s16.customer.com at /u04 using the DB Optimized mount option. This filesystem is intended for storage of Oracle databases and Oracle Clusterware files.
mount should be applied to all primary and backup servers that could host the database files.3 Figure 2 DB Optimized filesystem mount operation For more information about the DB Optimized mount flag, see the section “Server mount options” in Chapter 8, “Configure PSFS filesystems,” in the HP Scalable NAS File Serving Software administration guide.
Selecting Virtual NFS Service export options HP Scalable NAS requires the same NFS export options as most other NAS offerings. The four non-default required export options are: • no_wdelay • rw • no_root_squash • insecure The use of no_root_squash is necessary for exported filesystems that will be used for Oracle database or application installs, where the installer must run as user root during the install process.
The default options shown in Figure 3 are just as important as the non-default options. In particular, you must use the sync option for Oracle files. Do not use the async option; this can result in a corrupted database.
Managing Virtual NFS Services Before NFS mounting the storage on the NFS client, the NAS administrator must configure the Virtual NFS Service that the Oracle server (NFS client) will use to access the storage. If you are using DNS, the appropriate Virtual IP addresses and host names need to be assigned beforehand. It is a good idea to first verify that the host name you plan to use resolves. Do this by attempting to ping the host name, which should show the resolved IP address.
Figure 4 Associating Export Groups with NAS heads On the Virtual NFS Service Properties window, above, note that “Always active” is checked. This means that upon server failure, the Virtual NFS Service will move to an active server even if the associated Export Group monitor is inactive or down. If the box is not checked, failover will not occur when the associated monitor is inactive or down on all of the backup servers and the Virtual NFS Service will not be made active anywhere.
Dynamic rehosting and monitoring of Virtual NFS Services Figure 5 shows the Applications tab on the HP Scalable NAS Management Console. Represented in the rows are a set of three Virtual NFS Services. On the right-hand side above the columns are the NAS heads by name (tr6s15 though tmr6s18) indicating four NAS servers in this cluster. A Virtual NFS Service (VNFS) called 192.168.60.
Figure 6 Transitioning NFS heads by drag and drop Click Yes to accept the change or click No to reject it. If you accept the change, the Management Console will show the new primary, as in Figure 7, when the transition is complete. Figure 7 Management Console showing new primary server for VNFS 192.168.60.71 After the transition, tmr6s17 is the primary server for VNFS 192.168.60.71 and the first backup is tmr6s18.
Virtual NFS Server RPC reply cache The NFS protocol allows requests to be resent. Some requests such as write, create, remove, rename, link, unlink, and so forth must be executed on the server once and only once. These operations are called non-idempotent. The NFS server must keep a cache of its current non-idempotent requests to ensure that it does not repeat a request should another identical request arrive. This cache is called the RPC reply cache.
ORACLE_HOME will not saturate a cluster NAS head. Availability will not be compromised, as failover and rehosting are certainly functional on filesystems presented through a single VNFS. Configuring NLM for Oracle Home NLM is the locking protocol used by NFS. When Real Application Clusters is not being used, Oracle uses file locks on a file in $ORACLE_HOME/dbs called lk to ensure that the database is not already opened by another instance. NLM is disabled by default on FS Option for Linux.
For example, you may determine that the Oracle owner is “oracle” and user oracle will have uid 500. Groups dba and oinstall will have gids 1000 and 1001 respectively.
To disable the SizingActions script, execute the following steps on each server in the cluster: 1. Go to the directory containing the SizingActions script: # cd /etc/opt/hpcfs 2. Run the following command: # chmod 444 SizingActions 3. Reboot the node to ensure that the SizingActions parameters are cleared from the system. The SizingActions script will now be inactive at system start.
You will need to determine which NFS server was affected by the outage to know which Virtual NFS Server to rehost. You may not have access to the /var/lib/ nfs/statd/sm directory on the NFS client to view the IP addresses of the affected servers. At this point, the lock failure itself should point you to the VNFS host that is holding the lock.
3 System Administrator: deploying database servers with HP Scalable NAS This chapter describes system configuration considerations for the NFS client (the Oracle server): • Mounting and linking filesystems for Oracle Home and Oracle data files • NFS mount options • Oracle server operating system parameters Mounting and linking filesystems for Oracle Home and Oracle data files As described earlier, you should install the Oracle product ($ORACLE _HOME) on an NFS mounted filesystem exported from a standard mo
On a local disk we create a mount point /u01/app/oracle owned by the Oracle user. Then we add a mount in /etc/fstab for /u01/app/oracle exported from VNFS 192.168.60.71 and /u02 exported from VNFS 192.168.60.79. The /etc/fstab entries would be: #ORACLE_HOME 192.168.60.71:/u03/app/oracle /u01/app/oracle nfs # ORACLE DATAFILES 192.168.60.79: /u04 /u02 nfs See NFS mount options, below, for more information about the mount options.
NOTE: Once there is an Oracle installation, the Oracle uid and gids are set. Because these identifiers are used to determine user and group permission across NFS servers exporting the same filesystem, the users and groups must be created with identical uids and gids on all Oracle servers that share filesystems. Oracle has the same user id requirement for Oracle RAC installs needing access to shared data files.
• rmem_max: 262144 • wmem_default: 262144 • wmem_max: 262144 See also the setting for aio_max_nr under NFS client I/O, page 35. See the Oracle Installation Guide for your specific Oracle port for any operating system configuration issues for Oracle on NAS.
4 Oracle DBA: managing Oracle databases in a NAS environment Configuring an Oracle database for HP Scalable NAS NFS client I/O Generally speaking, the best I/O performance with Oracle is achieved by using Direct I/O (bypassing the OS buffer cache) and asynchronous I/O where possible. For Oracle ports that support Oracle on Network Attached Storage, the init.ora parameter filesystemio_options governs whether these filesystem features will be used by the server.
for all Oracle versions prior to 10.2.0.4, it is necessary to configure Oracle on the NFS client systems to use direct IO and not asynchronous I/O. Fortunately, this bug was fixed at version 10.2.0.4. All others should consider using asynchronous I/O as well as direct I/O. The instructions in this guide assume that you are using direct I/O.
top five wait events in your STATSPACK reports, simply schedule a database reboot and increase the number of writers. NFS client network performance considerations When configuring storage bandwidth for Oracle databases, there is no simple formula that meets the needs of all workloads. However, there are some guidelines. For Online Transaction Processing (OLTP), a commonly accepted ratio of database server to I/O subsystem bandwidth is 100Mb I/O for every 1GHz CPU.
Managing Virtual NFS Services using Direct NFS Beginning with Oracle 11g, Oracle includes their own NFS client, Direct NFS, which can be used to access data files over NFS. However, it is not used to access Oracle Home, CRS voting or registry files, or other files via NFS. For that, Linux-based Oracle servers use the NFS client provided by the operating system.
a single path. To use the load balancing feature, you must use the Direct NFS configuration file named oranfstab, which can be placed in several locations, depending on its scope. This file has syntax to define multiple physical IP addresses (path) for a given NFS server (server). For example, this entry in oranfstab indicates three physical interfaces on the same network (net mask 255.255.255.0) to NFSServer1: server: NFSServer1 path: 132.34.34.12 path: 132.34.34.13 path: 132.34.34.
Administrator will need to define a path in /etc/fstab so that you can maintain access to these datafiles via other utilities such as cd, mv, and certain other Oracle utilities such as dbv that do not use the database engine to access files. Merging Direct NFS client network topology with HP Scalable NAS This section shows how HP Scalable NAS and Direct NFS compliment each other to provide increased reliability and scalability.
As depicted in Figure 9, we can solve the network single point of failure issue (for Oracle data files) by using Direct NFS and configuring multiple NICs on the client and the server. An alternative would be to use bonded NICs, but that does require additional capability in the network. In addition (not shown here for simplicity) multiple networks can be configured between the Oracle server and the NFS server to avoid downtime in the event of network failure.
Suppose, instead of Direct NFS, we introduce only HP Scalable NAS to the scenario in Figure 8. With the NFS cluster, the Oracle Server uses a Virtual NFS service to mount storage rather than the physical IP address of the actual NFS server (shown with a green line). Should a server fail or should the NAS administrator need to move the service, the service is picked up by the backup server (shown with dotted lines) transparently. The database will not go down.
Now consider the case shown in Figure 11, when both Direct NFS and a HP Scalable NAS cluster are deployed with multiple network paths to the same NAS head. This scenario removes both the Network and the NAS server single points of failure. In fact, this scenario can survive all but one network loss and all but one server loss (up to 15 servers in a 16-server cluster). The workload can be balanced on the fly by Direct NFS without the need to understand the workload.
Taking this example one step further, greater scalability can be achieved. Because each of the Virtual NFS Services can export the same filesystem, the same datafile can be hosted by different members of the HP Scalable NAS cluster at the same time (see Figure 12). All this happens, even though the Oracle server thinks it is connected to the same NAS server via four separate paths. The administrator can transition from one NAS head serving to many without the database having to shutdown.
But we still have the issue of Client server failure. To remove the Oracle server (NFS client) as the last single point of failure, the customer can deploy Oracle RAC, depicted in Figure 13. HP Scalable NAS has been thoroughly tested using Direct NFS with this scenario. This scenario also provides the highest possible resource scalability with respect to both the Oracle server and the NFS server (up to 16 nodes and 2TB of storage).
Today, Oracle tests validated configurations in which NAS devices are tested in various scenarios. There are validated configurations using HP Scalable NAS File Serving Software as well. See Appendix A, page 49, for more detail.
5 Direct NFS client considerations on Windows systems Oracle 11g on Windows supports NFS for the first time, but only using Direct NFS and only for datafiles. Although the Oracle RDBMS can recognize datafiles using Direct NFS, many commands cannot. It is therefore necessary to provide shared access to the datafiles via other means, such as possibly via a CIFS mount or a Windows-based NFS client. Oracle does not allow use of NFS or CIFS protocols for every Oracle object.
Direct NFS client considerations on Windows systems
A References The following information may be helpful when provisioning Oracle databases. • Metalink note 359515.1. Mount options for Oracle files when used with NAS devices. • Metalink note 444809.1. Information about CIFS support for Oracle database objects. • Metalink note 341788:1. Use of Jumbo Frames. Recommendation for the Real Application Cluster Interconnect and Jumbo Frames.
References
B Provisioning Oracle on HP 4400 Scalable NAS systems Provisioning for Oracle on the HP 4400 Scalable NAS system is simple and quick, as the system is shipped with three filesystems already created. Customers can use those filesystems for Oracle or can create additional filesystems to meet exact size requirements. Configure filesystems for Oracle datafiles HP Scalable NAS provides a mount option called DB Optimize for PSFS filesystems.
3. For the filesystem chosen for Oracle datafiles, remount the filesystem using /u02 as the mount point. Choose to automatically create the directory and then select the non-default “DB Optimized” mount option. 4. If an additional filesystem will be for oracle datafiles, remount that filesystem using /u03 as the mount point. Choose to automatically create the directory and then select the non-default “DB Optimized” mount option. 5.
Wait a few moments for the filesystem to be unmounted on the servers. The Management Console will report an alert as the filesystem is unmounted on each server. Next, delete the persistent mounts for the VOL003 filesystem. Select the filesystem on the Management Console, right-click, and select Edit Persistent Mounts. Then, on the Edit Persistent Mounts window, select all servers and click Delete.
Now you can remount the filesystem at the new location, /u02, which matches the naming standard Oracle uses for paths to Oracle binaries and datafiles. Select the VOL003 filesystem on the Management Console, right-click, and select Add Mount. On the Mount Filesystem window, select all of the servers in the On Servers box. Next, specify /u02 as the mount point, check Create directory to create the mount point automatically, and then check the DB Optimized database mount option.
After you click OK, the filesystem will be mounted at /u02 on each server. Next, prepare the VOL002 filesystem, which will be used for oracle binaries ($ORACLE_BASE). Using the procedures described above, unmount VOL002 and remove its persistent mounts. Then select the VOL002 filesystem on the Management Console, right-click, and select Add Mount. On the Mount Filesystem dialog, highlight of all the servers in the On Servers box. Specify /u01/app/oracle as the mount point and check Create directory.
When the Management Console shows that the filesystem is mounted on all servers, you can create the NFS exports for Oracle. Configure exports for Oracle over NFS When filesystems that an Oracle database server will use are exported over NFS, Oracle expects that certain export options will be used. These options are specified in the export record for a Export Group.
The following Export Record Details dialog shows the options selected for the filesystem mounted at /u02, which contains Oracle datafiles. The client 99.10.60.13 is the Oracle server that will use this filesystem. Complete the configuration Using Chapter 3 in this document, the system administrator should establish the oracle user and groups, mount the filesystems, set permissions, and create a symbolic link for oradata in $ORACLE_BASE|$ORACLE_HOME (for example, /u01/app/ oracle/oradata) to /u02/oradata.
filesystems hosting Oracle datafiles are configured differently from filesystems hosting $ORACLE_BASE, it is important for the NAS administrator to specify which filesystem is intended for which purpose, so that the system administrator can set up the proper mount points on the Oracle server. After those steps are completed, the Oracle DBA can install Oracle. DBAs should review Chapter 4 in this guide for additional information and considerations when using HP Scalable NAS for Oracle over NFS.
