Oracle 10g Server on Red Hat® Enterprise Linux® 5 Deployment Recommendations Version 1.
Oracle 10g Server on Red Hat® Enterprise Linux® 5 Deployment Recommendations Copyright © 2008 by Red Hat, Inc. 1801 Varsity Drive Raleigh NC 27606-2072 USA Phone: +1 919 754 3700 Phone: 888 733 4281 Fax: +1 919 754 3701 PO Box 13588 Research Triangle Park NC 27709 USA "Red Hat," Red Hat Linux, the Red Hat "Shadowman" logo, and the products listed are trademarks or registered trademarks of Red Hat, Inc. in the United States and other countries. Linux is a registered trademark of Linus Torvalds.
Revision History Version Date Summary of Changes 1.0 22SEP2008 Original Report A. Brezhnev 1.1 2OCT2008 Incorporated review input A. Brezhnev 1.2 26NOV2008 Edited and formatted for Reference Architecture release Author Solutions Architecture group www.redhat.
Table of Contents INTRODUCTION......................................................................................................................6 64-bit Architecture..................................................................................................................................6 PACKAGES REQUIRED FOR ORACLE 10GR2 (10.2) INSTALLATION.........................................................................................7 KERNEL UPGRADES...................................................
APPENDIX A. SAMPLE ORACLE10G-PLATFORM.SPEC FILE...................................................................................................44 APPENDIX B. SAMPLE 39-ORACLE-MULTIPATH.CONF.........................................................................................................48 APPENDIX C. SAMPLE ORACLE.SPEC FILE......................................................................................................................48 REFERENCES..............................................
Introduction This document provides recommendations for running Oracle 10g R2 standalone and RAC databases on Red Hat Enterprise Linux (RHEL). The recommendations presented in this document are targeted to the following hardware platforms: ● 64-bit physical system (x86_64) with SAN attached storage for Oracle ● VMware virtual machine (x86_64) with virtualized disks for Oracle We will present generic tuning recommendations for systems with 1, 4, 8, 16, 32 and 64GB of RAM.
Packages Required for Oracle 10gR2 (10.
I/O Scheduler Starting with the 2.6 kernel (Red Hat Enterprise Linux 4 and later) the I/O scheduler can be modified at boot time to control the manner in which the kernel commits reads and writes to disk. Red Hat Enterprise Linux 5 allows users to change I/O schedulers dynamically for each block device (e.g. echo sched_name > /sys/block/ /queue/scheduler).
the noop scheduler for device /dev/sdd and verify the result: # echo noop > /sys/block/sdd/queue/scheduler # cat /sys/block/sdd/queue/scheduler [noop] anticipatory deadline cfq RHEL5 Running as a VMware Guest By default, the timer interrupt (tick) rate in Red Hat Enterprise Linux 4 and 5 is 1000 Hz. In some situations, this rate is too high, with frequent interrupts potentially causing a performance impact or unexpected behavior.
3776136 KB are free for other applications. Don't get confused by the first line which shows that 28160KB are free! Note that most of the memory used is for buffers and cache since Linux always tries to use RAM to the fullest extent to speed up disk operations. Using available memory for buffers (file system metadata) and cache (pages containing contents of files or block devices) helps the system to run faster because disk information is already in memory, thereby reducing I/O.
too low (less than 100). Firstly, pdflush will work on is writing pages that have been dirty for longer than it deems acceptable. This is controlled by: dirty_expire_centisecs (default 3000): in hundredths of second, how long data can be in the page cash before it is considered expired and must be written at the next opportunity. Note that this default is very long: a full 30 seconds.
database and each combination of database instances running on the same system are unique. The following settings proved useful for many Oracle database users and improved the stability and performance of Oracle servers on Red Hat Enterprise Linux 5. They should be considered as a starting point for the follow-up performance tuning. Ensure the following settings are present in /etc/sysctl.conf: vm.swappiness=0 vm.dirty_background_ratio=3 vm.dirty_ratio=15 vm.dirty_expire_centisecs=500 vm.
RAM Swap SPace 1GB - 2GB 1.5 * total RAM 2GB - 4GB 1 * total RAM > 4GB 4GB Checking Swap Space Size and Usage The size and current usage of swap space can be obtained using any of the following commands: grep SwapTotal /proc/meminfo cat /proc/swaps free Swap usage may slowly increase as shown above but should stop increasing at some point.
If the server shows sporadic swap activity or swap activity for a short period time at certain intervals, either add more swap space or more RAM. If swap usage is already very large ( do not confuse with constant swapping), then more RAM is recommended. Large Memory Optimization (HugePages) Physical memory is partitioned into pages which are the basic unit of memory management. When a Linux process accesses a virtual address, the CPU must translate it into a physical address.
Contiguous pages are preallocated and cannot be used for anything else but for System V shared memory (e.g. SGA) Less kernel bookkeeping work for that part of virtual memory due to larger page sizes Usage of Huge Pages in Oracle 10g Huge Pages in Red Hat Enterprise Linux 5 need to be requested explicitly by the application by using the SHM_HUGETLB flag when invoking the shmget() system call. This ensures that shared memory segments are allocated out of the Huge Pages pool.
# sysctl -w vm.nr_hugepages=512 To make the change permanent, append the following line to the /etc/sysctl.conf file as follows. This file is used during the boot process. The Huge Pages pool is usually guaranteed if requested at boot time: # echo "vm.nr_hugepages=512" >> /etc/sysctl.conf If you allocate a large number of Huge Pages, the execution of the above commands can take a while.
on this system would be 1048576 KB (512*1024*2). If memlock is too small, then no single page will be allocated when the Oracle database starts. For more information on setting shell limits, see Setting Shell Limits for the Oracle User section of this document. Login as the oracle user again and verify the new memlock setting by executing ulimit -l before starting the database.
Note that if you set SHMMAX to 4294967296 bytes (4*1024*1024*1024=4GB) on a 32-bit system, then SHMMAX will essentially be set to 0 bytes since it wraps around the 32-bit 4GB value. This means that SHMMAX should not exceed 4294967295 on a 32-bit system. In fact for Red Hat Enterprise Linux 5 x86, it should be approximately 3700000000. On x86-64 platforms, SHMMAX can be much larger than 4GB since the virtual address space is not limited by 32 bits.
The default shared memory limit for SHMMNI can be changed in the proc file system without reboot: # echo 4096 > /proc/sys/kernel/shmmni Alternatively, you can use sysctl(8) to change it: # sysctl -w kernel.shmmni=4096 To make a change permanent, add the following line to the file /etc/sysctl.conf. This file is used during the boot process. # echo "kernel.shmmni=4096" >> /etc/sysctl.
Removing Shared Memory Sometimes after an instance crash you may have to remove Oracle's shared memory segment(s) manually. To see all shared memory segments that are allocated on the system, execute: $ ipcs -m ------ Shared Memory Segments -------key shmid owner perms 0x8f6e2129 98305 oracle 600 0x2f629238 65536 oracle 640 0x00000000 32768 oracle 640 bytes nattch 77694523 0 2736783360 35 2736783360 0 status dest In this example you can see that three shared memory segments have been allocated.
Summary of Recommended Huge Pages & Shared Memory Settings For 64-bit systems with 64GB of RAM: kernel.shmmax=34359738368 kernel.shmmni=4096 kernel.shmall=16777216 vm.nr_hugepages=16384 For 64-bit systems with 32GB of RAM: kernel.shmmax=17179869184 kernel.shmmni=4096 kernel.shmall=8388608 vm.nr_hugepages=8192 For 64-bit systems with 16GB of RAM: kernel.shmmax=8589934592 kernel.shmmni=4096 kernel.shmall=4194304 vm.nr_hugepages=4096 For 64-bit systems with 8GB of RAM: kernel.shmmax=4294967295 kernel.
Setting Semaphores Semaphores can be described as counters which are used to provide synchronization between processes or between threads within a process for shared resources like shared memories. System V semaphores support semaphore sets where each one is a counting semaphore. So when an application requests semaphores, the kernel releases them in sets. The number of semaphores per set can be defined through the kernel parameter SEMMSL.
Oracle recommends SEMMNS to be at least 32000 and uses this value in its validated configurations. This value corresponds to SEMMSL * SEMMNI (250*128=32000) semaphores. With SEMMNI increased to 142 the value of SEMMNS can be also increased to 35500 or more if the system is using more processes and SEMMSL value is more than 250. The minimum recommended value of SEMMNS is 32000.
accordingly. As shown in the The SEMMSL Parameter section above, the SEMMSL setting should be 10 plus the largest PROCESSES parameter of any Oracle database on the system. So if you have one database instance running on a system where PROCESSES is set to 5000, then SEMMSL should be set to 5010. As shown in the The SEMMNS Parameter section above, the maximum number of semaphores that can be allocated on a Linux system will be the lesser of: SEMMNS or (SEMMSL * SEMMNI).
# echo 65536 > /proc/sys/fs/file-max Alternatively, you can use sysctl(8) to change it: # sysctl -w fs.file-max=65536 To make the change permanent, add or change the following line in the file /etc/sysctl.conf. This file is used during the boot process. Oracle uses the following value in its validated configurations: # echo "fs.file-max=327679" >> /etc/sysctl.
To make the change permanent, add the following lines to the /etc/sysctl.conf file, which is used during the boot process: net.core.rmem_default=262144 net.core.wmem_default=262144 net.core.rmem_max=4194304 net.core.wmem_max=262144 Minimum, default and maximum values of TCP socket send and receive buffers should correspond to the settings above: net.ipv4.tcp_rmem=4096 262144 4194304 net.ipv4.
Some drivers may require driver parameters in the /etc/modprobe.conf file. For example, the driver e1000 may be used with flow control parameter specified as option e1000 FlowControl=1,1 It is recommended to configure network switches and NICs for autonegotiation and let them agree on the right combination of flow control settings. Network Interface Bonding Conceptually the NIC bonding is a method for aggregating multiple network interfaces into a single logical bonded interface.
miimon=100 downdelay=300 updelay=300 primary=eth1 Note that bond0 uses balance-alb mode and ARP-based monitoring, whereas bond1 calls for mode 1 (failover) using mii link status monitoring. These are provided as working syntax examples. You can run both with miimon, if preferred, or other differences in your optionsbut both issues mentioned above are covered.
/sbin/ethtool -K eth2 tso off; \ /sbin/modprobe -i bonding -o bond0 \ mode=active-backup miimon=100 downdelay=300 updelay=300 primary=eth0 install bond1 /sbin/modprobe bnx2; /sbin/modprobe e1000; \ /sbin/ethtool -K eth3 tso off; \ /sbin/modprobe -i bonding -o bond1 \ mode=balance-alb miimon=100 downdelay=300 updelay=300 This example also shows the bond1 interface configured for load balancing.
oracle hard nofile 131072 These are the recommended values used in Oracle validated configurations. The "soft limit" in the first line defines the number of file handles or open files that the Oracle user will have after login. You can set the "soft" and "hard" limits higher if necessary. NOTE: It is not recommend to set the "hard" limit for nofile for the oracle user equal to /proc/sys/fs/file-max.
Enabling Asynchronous I/O and Direct I/O Support Asynchronous I/O permits Oracle to continue processing after issuing I/Os requests, which leads to higher I/O performance. Red Hat Enterprise Linux also allows Oracle to issue multiple simultaneous I/O requests with a single system call. This reduces context switch overhead and allows the kernel to optimize disk activity. Oracle 10g Release 2 is shipped with asynchronous I/O support enabled by default.
OCFS2 file systems. Checking Asynchronous I/O Usage To verify whether $ORACLE_HOME/bin/oracle was linked with async I/O, you can use the Linux commands ldd and nm. In the following example, $ORACLE_HOME/bin/oracle was relinked with async I/O: $ ldd $ORACLE_HOME/bin/oracle | grep libaio libaio.so.1 => /usr/lib/libaio.so.1 (0x0093d000) $ nm $ORACLE_HOME/bin/oracle | grep io_getevent w io_getevents@@LIBAIO_0.
71775 19460 13090 12495 ... 32434 15050 13015 11956 45% 77% 99% 95% 0.05K 0.27K 0.03K 0.03K 957 1390 110 105 75 14 119 119 3828K 5560K 440K 420K buffer_head radix_tree_node avtab_node size-32 Slab caches are a special memory pool in the kernel for adding and removing objects (e.g., data structures or data buffers) of the same size.
the standalone and RAC configurations. The NFS file system is supported with certified NFS servers. The main difficulties with storage configuration are related to the dynamic device detection processes in the Linux kernel. Devices may be recognized in the different order across reboots and as a result, their kernel names are not persistent.
Configuring Storage Access with Device Mapper Multipath Red Hat Enterprise Linux supports large variety of storage devices with native Linux multipathing. As an example, we will show the configuration for EMC CLARiiON and SYMMETRIX storage arrays. The following steps detail the procedure for configuring multipath on a Red Hat Enterprise Linux 5 host. 1.
4. Optionally you can assign aliases in multipaths section. The value of WWID can be obtained with scsi_id utility. For example: # scsi_id -g -u -s /block/sda In Oracle RAC, the clusterware device names must be persistent across reboots. The aliases can help to create human friendly persistent named for the OCR (primary and mirror) disks, CSS voting disks and also for ASM disks (if using ASM without ASMLib). 5. Perform a multipath dry run and evaluate the setup by running the following command.
Because it supports larger volumes, parted is the recommended tool to create partitions on Red Hat Enterprise Linux 5. The disk management utilities for x86/x86_64 based systems generally write 63 sectors of metadata (msdos label or partition table) directly at the beginning of the LUN. The addressable space begins immediately after these initial sectors causing a misalignment with the RAID strips and, as a result, excessive cross disk accesses leading to performance degradation.
2. Obtain unique SCSI disk identifiers of the Clusterware devices: # for i in `awk '/sd/ {print $4}' /proc/partitions`; \ do echo "$i: `scsi_id -g -u -s /block/$i`"; done These identifiers are required to create persistent device names. 3A. Creating persistent raw devices for single path LUNs Red Hat Enterprise Linux 4 had a rawdevices service and the /etc/sysconfig/rawdevices file for binding raw devices to block devices. Red Hat Enterprise Linux 5 has deprecated raw devices.
Check the raw device binding: # raw -qa 3B. Creating persistent raw devices for multipath LUNs It is assumed that the device-mapper-multipath package has been configured as described above. For the example below, we made following assumptions: 1. OCR primary and mirror files and CSS voting disks are not partitioned. 2. The multipaths section of the /etc/multipath.conf file contains aliases for OCR primary and mirror disks (ocr1 and ocr2) as well as CSS voting disks (vote[1-3]). 3.
The amendments in 39-oracle-multipath.
Oracle Bug 4679769: Failed to format OCR disk using CLSFMT. Oracle Note 414163.1: 10gR2 RAC Install issues on Oracle EL5 or Red Hat Enterprise Linux 5 or SLES10 (VIPCA Failures). Before running root.sh, download and apply the patch for Bug 4679769. If the root.sh script execution caused the failure, erase the content of the OCR disks: # dd if=/dev/zero of=/dev/raw/raw1 bs=1M count=25 # dd if=/dev/zero of=/dev/raw/raw2 bs=1M count=25 Before re-running root.sh, review Metalink Note 414163.
# multipath wants the devmaps presented as meaningful device names # so name them after their devmap name SUBSYSTEM!="block", GOTO="end_mpath" KERNEL!="dm-[0-9]*", ACTION=="add", PROGRAM=="/bin/bash -c '/sbin/lsmod | \ /bin/grep ^dm_multipath'", RUN+="/sbin/multipath -v0 %M:%m" KERNEL!="dm-[0-9]*", GOTO="end_mpath" PROGRAM!="/sbin/mpath_wait %M %m", GOTO="end_mpath" ACTION=="add", RUN+="/sbin/dmsetup ls --target multipath \ --exec '/sbin/kpartx -a -p p' -j %M -m %m" PROGRAM=="/sbin/dmsetup ls --target multi
ORACLEASM_SCANORDER=“dm-*” Configure ASMLib and create ASM volumes: # service oracleasm configure Configuring the Oracle ASM library driver. ... # service oracleasm createdisk VOL0 /dev/mapper/asm0p1 # service oracleasm createdisk VOL1 /dev/mapper/asm1p1 ... You can list available ASM devices by running: # service oracleasm listdisks Re-scan ASM devices, if necessary: # service oracleasm scandisks www.redhat.
Appendix A. Sample oracle10g-platform.spec file Name: oracle10g-platform Version: 1.0 Release: 1%{?dist} Summary: Prepares system to be a platform for Oracle Database 10g server. Group: System Environment/Oracle License: GPLv2+ Source0: 39-oracle-multipath.
%ifarch x86_64 /usr/lib64/libdb.so.2 /usr/lib64/libdb.so.2.0.
memlock=32 fi cat >> /etc/sysctl.conf << __EOF__ ## Added by oracle10g-platform fs.aio-max-nr = 3145728 kernel.shmmax = ${shmmax} kernel.shmmni = ${shmmni} kernel.shmall = ${shmall} kernel.sem = ${semmsl} ${semmns} ${semopm} ${semmni} net.core.rmem_default = 262144 net.core.wmem_default = 262144 net.core.rmem_max = 4194304 net.core.wmem_max = 262144 net.ipv4.tcp_rmem = 4096 262144 4194304 net.ipv4.tcp_wmem = 4096 262144 262144 net.ipv4.ip_local_port_range = 1024 65000 vm.swappiness = 0 vm.
# # # # # # # # # # # # # # # # # nofile - max number of open files rss - max resident set size (KB) stack - max stack size (KB) cpu - max CPU time (MIN) nproc - max number of processes as - address space limit maxlogins - max number of logins for this user maxsyslogins - max number of logins on the system priority - the priority to run user process with locks - max number of file locks the user can hold sigpending - max number of pending signals msgqueue - max memory used by POSIX message queues (
Appendix B. Sample 39-oracle-multipath.
%build %install rm -rf %{buildroot} %{__mkdir_p} %{buildroot}%{oracle_base} cd %{buildroot}%{oracle_base} %{__tar} zxvf %{SOURCE0} oracle_home_version=$(echo -n %{oracle_version} | sed 's/\.//g') cat > %{buildroot}%{oracle_base}/%{oracle_version}-%{oracle_cpu}/reg_inventory <
References Tuning and Optimizing Red Hat Enterprise Linux for Oracle 9i and 10g Databases http://www.redhat.com/docs/manuals/enterprise/RHELTuningandOptimizationforOracleV11.pdf Oracle Database 10g Release 2 (10.2) Documentation http://www.oracle.com/pls/db102/homepage Understanding the Linux Kernel http://www.bookpool.com/sm/0596005652 The Linux Page Cache and pdflush: Theory of Operation and Tuning for Write-Heavy Loads http://www.westnet.com/~gsmith/content/linux-pdflush.
