User`s guide

ManualsBrandsAMD ManualsComputer equipmentLE-363 User's

XgOS

User’s Guide

Part No.: E53170-02

September 2014

Summary of content (436 pages)

PAGE 1
XgOS User’s Guide Part No.
PAGE 2
Copyright © 2013, 2014 Oracle and/or its affiliates. All rights reserved. This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means.
PAGE 3
Contents Using This Documentation 1.
PAGE 4
vNIC Command Examples vHBA Commands 11 11 vHBA Command Examples vHBA Prescan Examples I/O Card Commands 12 12 13 I/O Card Examples 13 Miscellaneous Show Commands 14 Virtual Resource Naming Restrictions 2.
PAGE 5
Examples 29 Determining HCA Ports and Checking Firmware Version in Servers Hardware Status and Environmentals Syntax 31 32 32 Example: Display Hardware Status for an Oracle Fabric Interconnect F1-15 33 Example: Show Fabric Board Information for an Oracle Fabric Interconnect F1-15 (QDR Fabric) 36 Example: Show Hardware Status for an Oracle Fabric Interconnect F1-4 38 Examples: Show Fabric Board Information for an Oracle Fabric Interconnect F1-4 (QDR Fabric) 41 Interfaces and Interface State 3.
PAGE 6
Displaying CLI Attributes Syntax 56 56 Example: Display the CLI Mode 56 XgOS CLI Example: Configure the Idle Session Time-out Displaying CLI Output Vertically (Wrapping) Syntax 57 Example: Control CLI Line Wrapping CLI Display Filters Syntax 57 57 58 58 Parameter Description 58 Example: Show Configuration in a List 59 Example: Show Configuration in XML Format Example: Sort CLI Output by Columns Terminal Rows and Columns Syntax 60 60 Example: Set CLI Terminal Screen Rows Syntax 61 Searc
PAGE 7
Syntax 70 Parameter Descriptions Optional Modifiers ▼ 70 71 Create a Server Profile Default Gateway Syntax 72 73 Parameter Descriptions Optional Modifiers 6.
PAGE 8
Example: Watch an Ethernet Card Gigabit Ethernet Port State 92 92 Example: Show an Ethernet Port 93 Example: Show an Ethernet Port’s vNICs Example: Watch an Ethernet Port Maximum Transmission Unit Syntax Set the MTU vNIC Mirroring 94 Mirror Traffic 95 95 96 Hardware Support 97 97 Mirroring to vNIC Mirroring to Port 97 98 Mirroring Considerations Syntax 98 99 Parameter Description 99 ▼ Configure vNIC Mirroring for a Port ▼ Configure vNIC Mirroring for a vNIC ▼ Configure Mirror for
PAGE 9
TCP Segmentation Offload Requirements Syntax 111 112 Parameter Description ▼ Syntax 114 115 Parameter Description 115 Configure Receive Batching Virtual LANs (VLANs) Operations 115 117 117 Configuration Overview and Guidelines Host-Managed VLANs 118 119 Chassis-Managed VLANs Properties 119 120 Ethernet Port Mode 120 120 Access 121 Tag Native 121 vNIC’s VLAN Properties Mode 122 122 Access 123 Configuration Options 123 Host-Managed VLAN Configuration 124 Chassis-Managed VLA
PAGE 10
GE and 10-Port GE VLAN Examples 126 Example 1: Port Access Mode, VLAN-Unaware Host Example 2: Port Trunk Mode, VLAN-Aware Host 127 128 Example 3: Port Trunk Mode, VLAN-Unaware Host 129 Example 4: Port Trunk Mode, VLAN-Unaware Host and HA vNICs 8.
PAGE 11
Example: remove-prescan ▼ 150 Detect LUN Changes rescan Set FC Card Attributes Syntax 150 151 151 Parameters 152 Example: Display FC Cards 152 Example: Display FC Card Utilization Watch FC Card Utilization Set FC Port Attributes Syntax 153 153 154 Parameter Description 154 Example: Set the Link Down Time-Out ▼ 152 156 Configure a Port for Direct-Attached Storage Removing vHBAs 157 158 General Procedure 158 Environments Where the General Procedure Applies Environments Where Special
PAGE 12
vHBA Statistics 167 Fibre Channel Monitoring Syntax 168 Example 168 LUN Masking Syntax 168 169 171 Parameter Description ▼ Set a LUN Mask 171 172 Optional LUN Masking: No Report LUN Interception Syntax 175 Example ▼ 9.
PAGE 13
Bandwidth Burst Size 192 192 Network QoS Services 193 QoS Operations Overview QoS Feature Matrix QoS Default Sets Syntax 195 195 197 Example 197 QoS Custom Sets Syntax 193 197 198 Automatic Calculation 199 Example: vNIC Custom Policer for a 10GbE Card ▼ Create a Policer for vNIC 200 ACLs With QoS and Application QoS 202 Example: ACL-Based Policer for 10GbE I/O Cards ▼ Create ACL-Based Policer Disabling QoS on a vNIC Syntax 199 203 203 205 205 Examples 205 Application QoS With
PAGE 14
12. Access Control Lists Setting Actions Syntax 213 214 214 Parameter Description Example 216 Setting Conditions Syntax 215 216 216 Operators 217 Support 217 Example 218 Displaying ACLs and Rules Syntax 218 Parameter Description Examples 218 219 Removing ACLs Syntax 219 220 Parameter Description Example 218 220 220 Example: Denying Egress Traffic ▼ 13.
PAGE 15
Parameter Description 229 LAG Configuration Options 230 Example LAG Configuration for Peer Device Configuring a Static LAG ▼ 231 232 Configure a Static LAG 232 Configuring a Passive Mode LAG 233 ▼ Configure a LAG for “Passive” Mode (Dynamic LAG) ▼ Configure Individual Ports for Passive Mode LACP Deleting a LAG Delete All LAGs from a 10-Port GE Module Delete a Port from a LAG 237 237 Delete an Ethernet Port from a Passive Mode LAG 239 Xsigo Directory Service 240 XDS Registration Proce
PAGE 16
Using the Internal IMS User Roles ▼ 250 250 Create a User Account 251 How Access is Controlled By User Roles ▼ Grant Privileges to a Local User Account Using Active Directory as the IMS Syntax ▼ 252 253 254 254 Configure AD Users and Roles 255 Example: Active Directory Server With Default Authentication Example: Configure Kerberos as a Secondary AD Server Example: Set IMS to an AD Server 256 257 258 Example: Display All AD Server Configuration Using Role Group Mappings for AD/LDAP Users 25
PAGE 17
Example: Changing a RADIUS User’s Privileges Example: Set IMS to a RADIUS Server 268 Example: Display all RADIUS Server Configurations Common IMS Operations 269 Example: Control Cached Authentication Data Setting the Shell Inactivity Timeout for Root User 16.
PAGE 18
Listing Virtual Interfaces Virtual Interface Status Traffic Counters 287 288 288 Input and Output Traffic Counters Server Profile Status Alarms 288 289 290 Syntax Example 290 290 Xsigo ProWatch Overview ProWatch Modes 291 291 ProWatch Transmission Schedules HTTP Proxy Support 291 292 ProWatch Command Syntax 293 ProWatch Parameter Descriptions Optional Qualifiers 293 295 Internet Connection Requirement 295 Sending ProWatch Data When Alarms Occur Configuring ProWatch ▼ ▼ 296 Configure
PAGE 19
Syntax 307 Example: Display vNIC Status 307 Tracing End-to-End IB Path Continuity ▼ 17.
PAGE 20
▼ Disable or Enable Root Login Over SSH Display Login Information Syntax 324 325 Example 325 Setting System Password Strength Syntax 325 327 Parameters Example 324 327 327 Setting the Oracle Fabric Interconnect Management IP Address ▼ Set or Change the Management Address of the Oracle Fabric Interconnect 328 Restoring Factory Defaults 331 Effects on the Oracle Fabric Interconnect Power Down and Power Up Syntax Example Syntax Example 331 332 332 332 Software Information 332 332 333
PAGE 21
Resetting the HCA 341 Displaying All Option ROM Images on the Oracle Fabric Interconnect Displaying All Firmware Images on the Oracle Fabric Interconnect Removing the Option ROM Removing the Firmware 344 Applying System Patches Command Syntax 347 347 348 Parameter Description ▼ Apply a Patch ▼ Remove a Patch 348 348 349 Scripting XgOS Commands 351 Scripts That Mimic UNIX Commands Aikido Scripting Language 351 352 Example: Create 10 vNICs Using Aikido Example: Move and Rename Files SEDIT S
PAGE 22
SRC RPM File 359 Basic rpmbuild Example The SPEC File 359 360 Environment Variables 362 Build Option 1: Stock Kernels 362 Build Option 2: Custom Kernels 363 Build Option 3: Kernel With Upgraded OFED Stack 364 Build Option 4: Combination of Customer Kernel and Upgraded OFED Stack 365 Non-RPM Builds 365 OFED Patch Files 366 RDMA Headers ▼ 367 Add RDMA Headers InfiniBand Headers 367 367 Required Information for Contacting Customer Support 20.
PAGE 23
Upgrading Windows Host Servers 381 ▼ Upgrading Windows Host Servers ▼ Upgrading Windows HCA Firmware and Option ROM Xsigo Dependency Service 381 382 384 Setting the Dependency of Other Services Upon XgDependRoot Removing Dependencies Manually 385 Removing Xsigo Drivers and Any Dependencies High Availability System Upgrade 385 385 High-Availability Upgrade Process Compatible Software Versions 385 387 ▼ Upgrade First ESX Sever and Fabric Interconnect ▼ Upgrade Second ESX Server and Fabric
PAGE 24
xxiv XgOS User’s Guide • September 2014
PAGE 25
Using This Documentation This document provides information about the XgOS CLI and on upgrading XgOS for the Oracle Fabric Interconnect. ■ “Related Documentation” on page xxv ■ “Feedback” on page xxv ■ “Support and Accessibility” on page xxvi Related Documentation Documentation Link All Oracle products http://www.oracle.com/documentation Oracle Virtual Networking http://www.oracle.com/goto/FABRIC-INTERCONNECT/docs Documentation Feedback Provide feedback about this documentation at: http://www.
PAGE 26
Support and Accessibility Oracle customers have access to electronic support through My Oracle Support. For information visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id= info or visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id=trs if you are hearing impaired.
PAGE 27
CHAPTER 1 XgOS CLI Overview The XgOS command-line interface (CLI) includes commands to configure every aspect of the Oracle Fabric Interconnects F1-15 and F1-4. This chapter introduces the XgOS CLI and the categories of objects it configures.
PAGE 28
Command Syntax Conventions Convention Description courier bold Commands and keywords that must be show vnic entered exactly as shown. It also highlights significant lines in the screen output display. courier plain Actual display output that has been copied Examples resourceUnavailable from the device. Also used for variable names shown in command syntax. “ ” Quotes reference specific fields taken from the screen display on the device. See the “state” field.
PAGE 29
■ Scripting Engine—Enables you to run scripts within the CLI for each I/O card. The engine also enables you to define new commands. Script support in the XgOS CLI includes a full text editor for writing and revising your scripts. For the details about the scripting engine, see “Scripting XgOS Commands” on page 351 This section introduces the CLI itself and explains the online help that is available.
PAGE 30
For example, to get command-completion hints and context-sensitive help about adding a vNIC: add vnic ? Possible completions: Virtual NIC name Repeat '?' for detailed help. Online Help In addition to command completion, the CLI includes online help topics at various levels. To access these topics enter help . The CLI displays the detailed help topic for that item, if such a topic is available. Certain higher-level concepts also have help topics within the CLI.
PAGE 31
Editing Commands on the Command Line While entering a command, you can move around in the command-line text and edit it. The following list shows the control key strokes that are available: ■ Left and right arrow keys: Move the cursor to either the left or to the right on the command line. ■ Up and down arrow keys: Scroll up and down in the command history. ■ Ctrl-L: Retype the line ■ Ctrl-U: Erase the line ■ Ctrl-C: Quit the command ■ Ctrl-R: Search the command history.
PAGE 32
The variable vnics contains a vector of lines containing the output from the show -list vnics command. Wildcards You can use wildcards to control the display of output. Wildcards may be used in most of the XgOS show commands to select a set of objects. The only wildcard character supported is the asterisk (*) which causes zero or more of any character to be displayed in the show command. Wildcards can also represent entire names.
PAGE 33
Pipes You can use pipes to control the display of output in the CLI. Pipes in the XgOS CLI are like standard UNIX pipes, and they use the same syntax. For example, if you enter: show vnic * | grep down The resulting output displays all vNICs in the down state. XgOS pipes work with any command, but is used with most effectiveness when coupled with the grep command. XgOS pipes are not limited to only one set of pipes; many stages in the pipeline are supported.
PAGE 34
Expressions can be combined using “and”, “or”, and “xor”. Expressions can also be enclosed in parentheses. For example, if you enter: show vnic * where if = 1/1 This command shows all vNICs terminated on interface 1/1. If you enter: show vhba priv*.
PAGE 35
■ Characters accepted in the names of vNICs and vHBAs include alphanumerics and underscore (_). Avoid using other characters. ■ Characters accepted in the other objects (for example, server profiles, LUN Masks, QoS Profiles, ACLs, and so on) include alphanumerics, dash (-), and underscore (_). Avoid using other characters. Logging in to the Oracle Fabric Interconnect You can use the command-line interface through the console or by using a secure shell (SSH).
PAGE 36
Server Profile Commands Server profiles are containers that hold vNICs/vHBAs and are assigned to physical servers. Profiles provide the flexibility to move an I/O personality from one physical server to another.
PAGE 37
vNIC Command Examples ■ Create a new vNIC for xserver1 and assign it to port 2 on the Ethernet module in slot 8: add vnic vnic0.xserver1 8/2 ■ Give vnic0 on xserver1 the IP address of 11.0.0.1 with netmask 255.255.255.0: set vnic vnic0.xserver1 -addr-type=static -ip-addr=11.0.0.1/24 ■ Display the properties of all vNICs: show vnic ■ Change the netmask on vnic0.xserver1 to 255.0.0.0: set vnic vnic0.xserver1 –netmask=255.0.0.0 ■ Set vnic0.xserver1 to DHCP: set vnic vnic0.
PAGE 38
vHBA Command Examples ■ Create a new vHBA for xserver1 and assign it to port 1 on the FC module in slot 15: add vhba vhba0.xserver1 15/1 ■ Display the targets and LUN IDs the vHBA can detect: show vhba vhba0.xserver1 targets ■ Display the properties of all vHBAs (WWNN/WWPN): show vhba ■ Request a vHBA to rescan the SAN fabric: set vhba vhba0.xserver1 rescan Note – You would do this if you changed LUN masking on an array, for example.
PAGE 39
■ Request an unbound vHBA to logout of the SAN fabric: set vhba vhba0.testserver remove-prescan ■ Display vHBA targets: show vhba vhba2.testserver targets See “Virtual Host Bus Adapters (vHBAs)” on page 133 for more information. I/O Card Commands The I/O modules and ports are the termination points of vNICs and enable vNICs to access network resources.
PAGE 40
Miscellaneous Show Commands ■ Display the XgOS version: show system version ■ Display the current system configuration: show system info ■ Display management Ethernet info: show system interfaces ■ Display all logged in users: show login ■ Display environmental information: show hardware ■ Display information for supporting an issue: get log-files -all ■ Display discovered physical servers: show physical-server Virtual Resource Naming Restrictions As with all computer systems, the names you ca
PAGE 41
■ dash (-) The names of virtual resources are restricted to the following lengths: ■ vNICs: between 2 and 10 characters ■ vHBAs: between 2 and 15 characters ■ server profiles: between 1 and 31 characters Chapter 1 XgOS CLI Overview 15
PAGE 42
16 XgOS User’s Guide • September 2014
PAGE 43
CHAPTER 2 Configuring Hardware Elements When configuring virtual I/O, you must refer to the various hardware elements of the Oracle Fabric Interconnect. This chapter presents detailed information about those elements and how to refer to those elements on the command line.
PAGE 44
The following figure illustrates the InfiniBand and I/O ports on an Oracle Fabric Interconnect F1-4. When referring to an I/O module and port on the command line, you use the form I/O_Slot/Port_Number. For example, you must specify a specific slot and port to add a vNIC: add vnic foo.
PAGE 45
14/8 nwEthernet1GbPort in slot 14 port 8 (up) used by 8 resources 14/9 nwEthernet1GbPort in slot 14 port 9 (down) unused 14/10 nwEthernet1GbPort in slot 14 port 10 (down) unused add vnic foo.bar 14/1 In command output, the module and slot consists of the following information elements: ■ connection type, for example either network (nwEthernet) or fibre channel SAN (sanFC) ■ port type, for example either Ethernet or Fibre Channel.
PAGE 46
The Oracle Fabric Interconnect contains an embedded Subnet Manager (SM) that manages the switching and pathing tables within the IB fabric. When there are multiple SMs on a subnet, one SM will be the master SM through an election algorithm. The remaining SMs become standby SMs. There is only one master SM per subnet. The master SM is a key element in initializing and configuring an IB subnet.
PAGE 47
chassis-port ServerPort19 id 2c90200204929 state N/A/up m-key 0 lid 4 sm-lid 61 link-width 4x link-speed 2_5_Gbps ----------------------------------------------------------------...
PAGE 48
Field Description m-key Management key. A construct that is contained in InfiniBand Architecture (IBA) management datagrams to authenticate the sender to the receiver. lid Local Identifier. An address assigned to a port by the IB Subnet Manager (SM), unique within the subnet, used for forwarding packets within the subnet. The SM manages the switching and routing tables with the IB fabric. The Source and Destination LIDs are present in the Local Route Header.
PAGE 49
I/O Modules Use show iocard command to display available I/O line card information in the system. There are feature differences and capability nuances between the 10-port Gigabit Ethernet and 10 GE Gigabit Ethernet I/O hardware modules. For more details, see “QoS Feature Matrix” on page 195, Chapter 12, and Chapter 7.
PAGE 50
The field “v-resources” indicates the number of Xsigo virtual resources (vNICs and vHBAs,) that are associated with this card. vNICs can be bound only to network Ethernet cards. vHBAs can be bound only to SAN FC cards. Because slot numbers exceed 4 (for example, slot 10), the output displayed is for a F1-15.
PAGE 51
2/10 1 2/10 1 2/10 1 2/10 1 2/10 1 2/10 1 2/10 1 2/10 1 2/10 1 2/10 1 2/10 1 2/10 1 30 records displayed dynamic dynamic dynamic dynamic dynamic dynamic dynamic dynamic dynamic dynamic dynamic dynamic 00:13:97:01:a1:13 00:13:97:00:02:4f 00:13:97:00:00:13 00:13:97:80:00:03 00:13:97:00:00:16 00:13:97:80:00:0f 00:13:97:01:a0:c9 00:13:97:01:a0:ce 00:13:97:00:00:81 00:13:97:00:00:86 00:14:6c:81:41:36 00:03:f4:02:45:d4 The command shows information about a 10-Port GE module in slot 2 including the VLAN number
PAGE 52
Parameter Description Parameter Description |* down Causes one or all I/O modules to be deactivated. When the shutdown occurs, all power to the affected I/O module is turned off. When you issue this command, a warning message is displayed, and you are prompted for confirmation before the module is shutdown. By default, the module is not shut down. If you want the module shutdown, you must explicitly answer yes (y) to the prompt.
PAGE 53
Example: Shut Down a Single I/O Module To shut down a single I/O module: set iocard 8 down Shutting down IO cards will adversely affect any virtual IO resource connected to them and thus cut IO to the physical servers. Are you sure you want to shutdown the IO card in slot 13 (y/n)? Example: Bring Up an I/O Module To bring up a single I/O module: set iocard 8 up Resetting I/O Modules A module reset powers down one or all modules in the chassis, then powers them back up.
PAGE 54
■ 128 vNICs per card ■ Card-level High Availability (HA) ■ Access Control List (flow) policing ■ QoS on the vNICs configured on the card ■ MTU sizes from 1500 bytes to 9194 Kbytes ■ IPv4 TCP/UDP checksum offload. If you want TCP Offload configured on the vNIC, you must set this value at vNIC creation time. You cannot edit a configured vNIC to add the Offload feature later. ■ Untagged VLANs.
PAGE 55
descr type nwEthernet10Port1GbCard vnics 2 qos default acl acl1 enables qas ----------------------------------------------------------------1 record displayed I/O Ports Use show ioport to display I/O port information on an I/O port.
PAGE 56
2/2 2/3 2/4 2/5 2/6 2/7 2/8 2/9 2/10 3/1 3/2 4/1 4/2 5/1 5/2 5/3 5/4 nwEthernet1GbPort nwEthernet1GbPort nwEthernet1GbPort nwEthernet1GbPort nwEthernet1GbPort nwEthernet1GbPort nwEthernet1GbPort nwEthernet1GbPort nwEthernet1GbPort sanFcPort sanFcPort sanFcPort sanFcPort nwEthernet10GbPort nwEthernet10GbPort nwEthernet10GbPort nwEthernet10GbPort up/up up/up up/up up/up up/down up/up up/up up/down up/up up/up up/down up/up up/up up/up up/up up/up up/up 0 0 0 0 0 0 0 0 0 4 0 0 0 4 0 0 0 show ioport 3/1 ---
PAGE 57
topo F loop-delay 5 tape-support true vhbas 4 ----------------------------------------------------------------1 record displayed Determining HCA Ports and Checking Firmware Version in Servers Servers can contain single- or dual-port HCA cards. You can determine whether a single-port or dual-port HCA is installed in a server, as well as determining what firmware version is loaded on the HCAs by searching for relevant strings.
PAGE 58
The Oracle Fabric Interconnect uses only memory-free HCAs. Because of the way Windows queries the HCA for firmware information, Windows can determine whether the HCA card is mem-based or mem-free. As a result, the dual port HCA is correctly displayed as MT25218 whenever the Device Manager is used to query the HCA. For the firmware version, the value is the current revision level. ■ To check the firmware version, use the cat command as shown: # cat /sys/class/infiniband/mthca0/fw_ver 1.2.
PAGE 59
Example: Display Hardware Status for an Oracle Fabric Interconnect F1-15 The show hardware command displays comprehensive output of all the important hardware information for a Oracle Fabric Interconnect. This example is for a F1-15 with an SDR fabric.
PAGE 60
----------------------------------------------------------------4 nwEthernet4Port10GbCardEthIb up in=34 out=48 0v9=0.89 1v2=1.20 1v8=1.82 1v8ldo=1.80 2v57=2.57 2v5=2.51 3v3=3.29 3v3sb=3.29 current=4.37 power=52.89 vps_12v0=12.09 5 nwEthernet1Port10GbCard up in=35 out=46 1v2=1.20 1v5=1.51 1v8=1.80 2v5=2.50 2v6=2.59 3v3=3.29 3v3sb=3.29 5v0=5.03 6 sanFc2Port4GbCard up in=36 out=41 1v2=1.20 1v5=1.50 1v8=1.80 2v5=2.51 2v6=2.60 3v3=3.29 3v3sb=3.29 8 nwEthernet10Port1GbCardEthIb up in=33 out=56 0v9=0.89 1=0.
PAGE 61
## Front Panel Environment status ############################################ state temperatures voltages -----------------------------------------------------------------------------up in=29 out=30 1v2=1.20 1v5=1.50 1v8=1.79 2v5=2.48 2v6=2.60 3v3=3.26 3v3sb=3.29 5v0=5.
PAGE 62
name descr state actual expected deviation -----------------------------------------------------------------------------Fan-1/1 up 4800 4800 0 Fan-1/2 up 5160 4800 360 Fan-2/1 up 4800 4800 0 Fan-2/2 up 4560 4800 -240 Fan-3/1 up 4800 4800 0 Fan-3/2 up 4440 4800 -360 Fan-4/1 up 4800 4800 0 Fan-4/2 up 4440 4800 -360 8 records displayed Example: Show Fabric Board Information for an Oracle Fabric Interconnect F1-15 (QDR Fabric) The show hardware command display information about the type and speed of Fabric Boa
PAGE 63
Also, the show fabric-port command shows the fabric type and speed for an individual Fabric Board with a connected host. The following example shows a QDR fabric port.
PAGE 64
timezone America_Los_Angeles domain-search is-sm true console-speed 115200 console-speed-rotation true phone-home false ----------------------------------------------------------------1 record displayed ...
PAGE 65
timezone America_Los_Angeles domain-search is-sm true console-speed 115200 console-speed-rotation true phone-home false root-ssh-login enabled ---------------------------------------------------------1 record displayed The show hardware command displays comprehensive output of all the important hardware information for a Oracle Fabric Interconnect. This example is for a F1-4 with a DDR fabric.
PAGE 66
1 record displayed ## Front Panel Environment status ############################################ state temperatures voltages -----------------------------------------------------------------------------up in=25 out=23 1v2=1.20 1v5=1.51 1v8=1.79 2v5=2.50 2v6=2.62 3v3=3.29 3v3sb=3.29 5_d2=5.
PAGE 67
Fan-2/2 Fan-2/3 Fan-2/4 Fan-3/1 Fan-3/2 Fan-3/3 Fan-3/4 12 records displayed up up up up up up up 17040 17640 16680 18360 16320 17760 16560 15000 15000 15000 15000 15000 15000 15000 2040 2640 1680 3360 1320 2760 1560 ## Fan controller status ##################################################### state serial-num actual-temp max-temp -----------------------------------------------------------------------------up 071014903 23 65 1 record displayed Examples: Show Fabric Board Information for an Oracle Fab
PAGE 68
1 VP-FRU-FB-QDR 331018983 up 1.19 QDR in=25 mid=27 out=26 1v2_1= 1v2_2=1.19 1v8=1.79 2v5=2.50 3v3=3.27 3v3sb=3.29 1 record displayed Also, the show fabric-port command shows the fabric type and speed for an individual Fabric Board with a connected host. The following example shows a QDR fabric port.
PAGE 69
domain xlab.xsigo.com address 192.168.1.2 netmask 255.255.255.0 nameservers 192.168.90.144, 192.168.90.142 model-num F1-4-CH-QDR serial-num 4R2D234092 ipconfig dhcp default-gateway 192.168.45.117 timezone America_Los_Angeles domain-search is-sm true console-speed 115200 console-speed-rotation true phone-home false ----------------------------------------------------------------1 record displayed ...
PAGE 70
ipconfig dhcp default-gateway 192.168.48.1 timezone America_Los_Angeles domain-search is-sm true console-speed 115200 console-speed-rotation true phone-home true root-ssh-login enabled ---------------------------------------------------------1 record displayed Interfaces and Interface State Each slot/port has its own interface (if) with state information (if-state): show vnic ----------------------------------------------------------------name myvinc.
PAGE 71
CHAPTER 3 XgOS File System Access and Logging The XgOS CLI provides a basic, unix-like file system. This chapter present the following topics about using that file system and about the system logs stored there: ■ “File System Structure” on page 45 ■ “Default Login” on page 46 ■ “All user data is stored in the User data partition on the hard drive:” on page 46 ■ “Logging” on page 51 File System Structure The XgOS CLI provides the following top-level directories: bin /bin contains binary files.
PAGE 72
Default Login The default login home directory is /home/admin: pwd /home/admin All user data is stored in the User data partition on the hard drive: show system ... DISK STATUS Partition Base OS XgOS System logs Database Temporary data User data Volatile data Config data Size 253.967M 1.192G 9.169G 8.249G 6.040G 2.752G 184.901M 44.292M Available 77.037M 486.004M 8.545G 7.513G 4.588G 2.115G 175.341M 41.932M Used %used 163.816M 64% |#############--------| 672.297M 55% |###########----------| 161.
PAGE 73
file file file file file file hash list [-long] [-recursive] [-short] move [-force][-query] remove [-force] [-query] [-recursive] search [][-except][-ignorecase][-linenumbers][-recursive] show [-archive][-compressed] Parameter Description Command Description file copy [-force] Copies a file from a source location to a destination.
PAGE 74
Command Description file upload Specifies the file on the Oracle Fabric Interconnect that will be uploaded to ftp.xsigo.com. You can view a list of local files by using command completion for file upload. This command assumes “ftpguest” as the username and password for the connection to ftp.xsigo.com. Use this command when directed by Customer Support, or contact Customer Support immediately after using this command to notify them of the incoming file.
PAGE 75
Command Description file remove [-force] [-query] [-recursive] Deletes a file. The following parameters are supported: • -force—Removes a file immediately and without prompting for confirmation. • -query—Removes a file but prompts for confirmation before the file is actually deleted. • -recursive—Removes an entire file tree by descending into directories. file search Searches files for regular expressions.
PAGE 76
show tech-support > mydebug file copy mydebug scp://joeuser@computefarm06.xsigo.com/homes/joeuser/mydebug.txt joeuser@computefarm06.xsigo.com's password: Copying... ####################################################################### [100%] Example: Archive and Compress To create an archive then compress it: file archive -noconfirm foo.tar file1 file2 file compress foo.tar Example: Upload and Download To upload a file named “foo.gz” to ftp.xsigo.com: file upload foo.gz foo.
PAGE 77
Logging Log files are stored in /log. ls /log lost+found coredumps btmp ulog apache2 wtmp postgresql news ntpstats ulog-acctd ksymoops xml dmesg user.log user-debug.log daemon.log lastlog kern.log ib.log postgresql.log createdb.log osm.log install.log apache2.pid dumpster.log osinstall.out osinstall.err user.log.2.gz user.log.3.gz user-debug.log.2.gz user-debug.log.3.gz user-debug.log.4.gz user-debug.log.5.gz user-debug.log.6.gz user-debug.log.7.gz user-debug.log.8.gz user-debug.log.9.gz user-debug.log.
PAGE 78
Example: Jun 6 00:00:01 iowa vnicmanager[12532]: [ERR] VNIC::VNICManager process_simm_message:: ENTRY User debugging goes to “user-debug.log” where log rotation also occurs automatically:: user-debug.log user-debug.log.1.gz user-debug.log.2.gz user-debug.log.3.gz user-debug.log.4.gz user-debug.log.5.gz user-debug.log.6.gz user-debug.log.7.gz user-debug.log.8.gz user-debug.log.9.gz user-debug.log.10.
PAGE 79
CHAPTER 4 Configuring the XgOS CLI This chapter describes commands that configure the XgOS CLI itself or display that configuration.
PAGE 80
Syntax set set set set set set set set set set set set set set cli cli cli cli cli cli cli cli cli cli cli cli cli cli autocommit {off|on} [-noconfirm] block-entry {off|on} color {off|on} cols rows confirm {off|on} echo {off|on} idle-timeout mode {expert|user|xml} paging {off|on} progress-bar {off|on} prompt {custom |normal} space-completion {off on} wrap {off|on} Parameter Description Parameter Description autocommit {off|on} [-noconfirm] The default is on.
PAGE 81
Parameter Description echo {off|on} Displays all CLI communication. The on option will echo all commands to the terminal screen. The default is off. idle-timeout After this many idle minutes, your CLI session will timeout. Configure a value of “0” to configure an infinite CLI timeout (no timeout). mode {expert|user|xml} Controls the CLI mode. The default is user. See show cli mode. paging {off|on} Sets the CLI paging mode. When on, the display output stops when the screen is full.
PAGE 82
Displaying CLI Attributes Use the show cli command to display different attributes of the CLI itself.
PAGE 83
XgOS CLI Example: Configure the Idle Session Time-out set cli idle-timeout show cli idle-timeout The idle timeout is disabled Displaying CLI Output Vertically (Wrapping) When the terminal display output is too wide and unreadable across the screen, the system can capture the output and display it in vertical mode.
PAGE 84
CLI Display Filters Display output can be sent through different CLI display filters. By default, the CLI display is controlled automatically by the management terminal settings. However, you can set specific display characteristics by using the XgOS CLI display filters. Color is also used as an inherent way to help you filter the display output and other pertinent information in the XgOS CLI.
PAGE 85
Example: Show Configuration in a List show -list vnic foobar.pubs -------------------------------------------name foobar.pubs state up mac-addr 00:13:97:01:80:06 ipaddr descr if 6/1 if-state ha-state local-id 0 type vlans none qos ie Example: Show Configuration in XML Format show -xml vnic foobar.pubs
PAGE 86
To specify multiple columns: show -sortby=name,if vnics This command will use “name” as the primary sort and “if” as the secondary. To perform a reverse sort: show -sortby=!name,if vnics Note – This command is one place in the CLI where command completion is not available. Terminal Rows and Columns XgOS enables you to set and display the number of rows and columns for the terminal screen. By default, the number of rows and columns is determined by the terminal size.
PAGE 87
Example: Set CLI Terminal Screen Rows show cli rows 30 set cli rows 60 show cli rows 60 CLI History Use the show cli history command to display the history of issued commands. The history log can be searched using the up/down arrow keys and Ctrl-R command sequence. Syntax show cli history show cli history where is the number of saved history commands to display. The buffer limit size is 512 commands per user. The log is persistent across CLI login sessions.
PAGE 88
▼ Search the CLI History 1. Press Ctrl-R to initiate a history search: (): Ctrl-C will interrupt the search. Repeated Ctrl-R will display the previous command. 2. Enter the command text string to search on: (gogo): add server-profile gogo 3.
PAGE 89
The following example shows a simple script of displaying vNICs. During the CLI recording phase, the commands issued are as follows: show vnic * show vnic pubstest.sunset show vnic pubstest.sunset -detail The first part of the example is recording the CLI session and stopping the recording: cli record-script showvnic Recording CLI commands into the script showvnic. Use 'cli stop-recording' to end the recording. # show vnic * ----------------------------------------------------------------name iscsitest.
PAGE 90
type dhcp vlans none qos ----------------------------------------------------------1 record displayed # show vnic pubstest.sunset -detail ----------------------------------------------------------------name pubstest.sunset state up/indeterminate mac-addr 00:00:00:00:00:00 admin-ipaddr 0.0.0.0/32 oper-ipaddr 0.0.0.
PAGE 91
vlans none qos -----------------------------------------------------------------name pubstest.sunset state up/indeterminate mac-addr 00:00:00:00:00 ipaddr 0.0.0.0/32 if if-state ha-state local-id 0 type dhcp vlans none qos -----------------------------------------------------------------------------2 records displayed ----------------------------------------------------------------name pubstest.sunset state up/up mac-addr 00:00:00:00:00:00 ipaddr 0.0.0.
PAGE 92
mode access queue-map-type disabled qos -local-id 0 mirror ----------------------------------------------------------------1 record displayed Saving and Restoring Configurations Before you perform a firmware upgrade in XgOS, Oracle recommends you export your system configuration to a file. If your running-config gets lost during an upgrade, at least you can import a saved config. If you import a configuration, the system migrates the old config to the new.
PAGE 93
Example: Save and Restore System Configuration system export myconfig.xml system import myconfig.xml DISK STATUS Partition Base OS XgOS System logs Database Temporary data User data Volatile data Config data Size 253.967M 1.192G 9.169G 8.249G 6.040G 2.752G 184.901M 44.292M Available 77.037M 330.699M 8.531G 7.646G 5.701G 2.522G 175.334M 41.952M Used %used 163.816M 64% |#############--------| 827.602M 67% |##############-------| 175.754M 1% |---------------------| 188.738M 2% |---------------------| 32.
PAGE 94
68 XgOS User’s Guide • September 2014
PAGE 95
CHAPTER 5 Server Profiles and Gateways This chapter describes how to configure the Oracle server profiles and the default gateway. ■ “Server Profiles” on page 69 ■ “Default Gateway” on page 72 Server Profiles A server profile is a logical representation of a physical host server’s I/O configuration which can be assigned to a physical server. When you assign a server profile, the host server assumes all of the server profile’s I/O characteristics.
PAGE 96
Syntax server-profile server-profile server-profile {*|} connection -descr="" server-profile {*|} connect server-profile {*|} disconnect server-profile {*|} down server-profile iscsi-boot [|none] [mount {direct |LABEL=|lvm }] set server-profile {*|} reset set server-profile {*|} san-boot [|none]
PAGE 97
Parameter Description reset Resets the busy state. san-boot Configures the SAN boot options for this server profile. For details about the set server-profile san-boot command, refer to the XgOS Remote Booting Guide. up Sets the server profile administratively up. Optional Modifiers Modifier Description -default=gateway= The name of a default gateway that you have already configured. For details about configuring a default gateway, see “Default Gateway” on page 72.
PAGE 98
2. Select the desired server and create the server profile: add server-profile mytest alexander@iowa:ServerPort8 3. Verify the profile was created correctly: show server-profile mytest name state descr connection def-gw vnics vhbas --------------------------------------------------------------------------mytest up/unassigned My first server profile 1 1 1 record displayed No I/O resources (vNICs or vHBAs) have been assigned to the new server profile.
PAGE 99
Syntax add gateway [-noconfirm] set gateway [-descr="text"][-dns=][-domain-name=] [-ipaddr=] set server-profile -default-gateway=[][none] show gateway [] {server-profiles] remove gateway [] [-noconfirm] Parameter Descriptions Parameter Description Name the gateway something unique to this Oracle Fabric Interconnect.
PAGE 100
▼ Configure a Default Gateway The following figure shows the topology of a default gateway. Take the following steps to configure a default gateway: 1. From the hostserver, confirm the following entities are not reachable: default gateway address, DNS server address, and domain name. cat /etc/resolv.conf route ping 10.1.11.112 2. Issue the route command to confirm the server cannot reach the outside network because you have not yet configured a default gateway. Likewise ping 10.1.11.
PAGE 101
name descr addr dns-addr domain-name ----------------------------------------------------------------test 10.1.10.112 1.1.1.1 testorg Note – The gateway’s IP address must be on the same subnet as the vNIC’s address. 5.
PAGE 102
This example changes the DNS to 2.2.2.2. After the change is made, the defaultgateway profile must be reassociated back to the server profile: set gateway test -dns=2.2.2.2 set server-profile s23 -default-gateway=test show gateway test name descr addr dns-addr domain-name ----------------------------------------------------------------test 10.1.10.112 2.2.2.
PAGE 103
CHAPTER 6 Virtual Network Interface Cards (vNICs) The Oracle virtual Network Interface Card (vNIC) virtualizes NIC connectivity. A vNIC is a virtual NIC that appears to the OS as a physical NIC and enables a server to have a Ethernet network attachment without having a physical NIC present. Instead of the client server using an NIC, an InfiniBand (IB) HCA is used and then virtualizes the NIC allowing for Ethernet connectivity.
PAGE 104
Basic vNIC Configuration A vNIC involves the following bringup procedure: ■ Adding a server profile ■ Creating a named vNIC ■ Associating the vNIC to a server profile and physical I/O card ■ Setting IP address information ■ Verifying the configuration and state Syntax add server-profile @:ServerPort add vnic . {/|none} set vnic .
PAGE 105
Note – The Oracle Fabric Interconnect automatically assigns MAC addresses to vNICs from a pool of internal-sequential addresses. Example: Configure a Server Profile With a vNIC add server-profile myserver alexander@iowa:ServerPort8 add vnic myvinc.myserver 4/2 set vnic myvinc.myserver -addr-type=static -ip-addr=10.1.1.1/32 show vnic myvinc.myserver ---------------------------------------------------------------name myvinc.myserver state up mac-addr 00:13:97:01:80:08 ipaddr 10.1.1.
PAGE 106
rcv-bytes 0 trans-pkt 0 trans-bytes 0 invalid-ip-checksum ? invalid-l4-checksum ? mtu-err ? ipchecksum-pkt ? tcp-checksum-pkt ? udp-checksum-pkt ? tcpseg-pkt ? green-pkt ? yellow-pkt ? red-pkt ? --------------------------------------------------------------1 record displayed vNIC Counters and Statistics There are several ways to gather vNIC counters and statistics. On the host server: ifconfig Displays statistics as collected by the OS through the network layer.
PAGE 107
High Availability vNIC Pairs High availability (HA) vNIC pairs can be configured for a single Oracle Fabric Interconnect chassis, or for two separate Oracle Fabric Interconnects. The system does not support the dynamic reconfiguration of vNIC failover characteristics. Once you create an HA enabled vNIC, the system does not allow you to change its failover characteristics. You must delete the vNIC then create a new one from scratch.
PAGE 108
8/5 nwEthernet1GbPort in slot 8 port 5 8/6 nwEthernet1GbPort in slot 8 port 6 8/7 nwEthernet1GbPort in slot 8 port 7 8/8 nwEthernet1GbPort in slot 8 port 8 8/9 nwEthernet1GbPort in slot 8 port 9 8/10 nwEthernet1GbPort in slot 8 port 10 2. Bind the vNIC to a physical Ethernet card. 3. Select the slot/port that you want to link to the vNIC (in this example, “6/1”): add vnic haNIC1.vserver1 6/1 ? Possible completions: ha Specify High Availability characteristics 4.
PAGE 109
Note – Do not select the same slot/port that was assigned to the primary vNIC. add vnic haNIC1.vserver1 6/1 ha 6/3 This command set created a high-availability vNIC pair on a single chassis. The primary vNIC is named haNIC1. The secondary vNIC was created automatically and named haNIC1S. (Note the “S” appended to the end of the name.) The full name of the primary vNIC was automatically assigned as the high-availability group’s name.
PAGE 110
8/5 8/6 8/7 8/8 8/9 8/10 nwEthernet1GbPort nwEthernet1GbPort nwEthernet1GbPort nwEthernet1GbPort nwEthernet1GbPort nwEthernet1GbPort in in in in in in slot slot slot slot slot slot 8 8 8 8 8 8 port port port port port port 5 6 7 8 9 10 c. Bind the vNIC to a physical Ethernet card. d. Select the slot/port that you want to link to the vNIC (in this example, “6/1”): add vnic haNIC1.
PAGE 111
f. Configure the vNIC as the primary vNIC of the HA pair. g. Select “-primary”, then press Enter. add vnic haNIC1.vserver1 6/1 ha -primary This command set created a vNIC (haNIC1), assigned it to a server profile (vserver1), bound it to a physical slot/port (6/1), and specified the vNIC as the primary vNIC in a high-availability vNIC pair. h. Retrieve the MAC address of the primary vNIC. show vnic haNIC1.vserver -----------------------------------------name haNIC1.
PAGE 112
8/2 8/3 8/4 8/5 8/6 8/7 8/8 8/9 8/10 nwEthernet1GbPort nwEthernet1GbPort nwEthernet1GbPort nwEthernet1GbPort nwEthernet1GbPort nwEthernet1GbPort nwEthernet1GbPort nwEthernet1GbPort nwEthernet1GbPort in in in in in in in in in slot slot slot slot slot slot slot slot slot 8 8 8 8 8 8 8 8 8 port port port port port port port port port 2 3 4 5 6 7 8 9 10 b. Bind the second vNIC to a physical Ethernet card on the second chassis. c.
PAGE 113
[Optional qualifiers] -mac Secondary HA group MAC address -primary This is a primary HA VNIC -secondary This is a secondary HA VNIC (need to specify group MAC address) 3. Configure the second vNIC as the secondary vNIC of the high-availability pair. [Optional qualifiers] -mac Secondary HA group MAC address -primary This is a primary HA VNIC -secondary This is a secondary HA VNIC (need to specify group MAC address) Select “-secondary” then press Enter: add vnic haNIC1.
PAGE 114
4. Insert the primary vNIC’s MAC address. a. Select “-mac”. b. Type ‘’, enter the MAC address retrieved in Step 6, then press Enter. add vnic haNIC1.vserver1 8/2 ha -secondary -mac=00:13:97:01:80:01 This command set created a high-availability vNIC pair across two Oracle Fabric Interconnects. The HA group’s name was automatically set to haNIC1.vserver1. Both the primary and secondary vNICs are named haNIC1.
PAGE 115
When 1/1 goes down, traffic fails over to path 2/1. When 1/1 comes back online, the vNIC reverts back to using 1/1 automatically. Any failure along the path (Ethernet or InfiniBand) of the vNIC will force traffic flow to the other side. Note that show vnic -detail displays “flags” is set to “A” once -auto-switchover is enabled: add vnic test_1.01bardeen 1/1 -auto-switchover=true ha 2/1 show vnic test_1.01bardeen -detail ----------------------------------------------------------------name test_1.
PAGE 116
qos -local-id 0 ----------------------------------------------------------------1 record displayed Admin State Control Use set vnic up|down to control the administrative state of a configured vNIC. Syntax set vnic . up set vnic . down Parameter Description Parameter Description up Activates a vNIC (default) down Deactivates a vNIC Example: Display and Control vNICs show vnic myvnic.
PAGE 117
qos -----------------------------------------------------------------1 record displayed set vnic myvnic.myserver down Deactivating a VNIC will cause the network interface on the server to become inoperable. Are you sure you want to deactivate VNIC boofar.pubs2 (y/n)?y show vnic myvnic.myserver ----------------------------------------------------------------name myvnic.
PAGE 118
Example: Show Ethernet Card Utilization In the following example, the show ethernet-card utilization command is issued against a 10-Port GE module in slot 14. show ethernet-card 14 utilization name total-util server-util network-util bcast-util ----------------------------------------------------------------14 0 0 0 0 1 record displayed The command shows columns for different types of utilization for the card: ■ Total utilization, which is the sum of all the other columns.
PAGE 119
show ethernet-port / vlan-association show ethernet-port / vnics watch ethernet-port / watch ethernet-ports Example: Show an Ethernet Port In the following example, the show ethernet-port command is issued against port 1 of the card in slot 14.
PAGE 120
ha-state local-id 0 type hostManaged vlans none qos -flags ---t------------------------------------------------------------------2 records displayed Example: Watch an Ethernet Port In the following example, the watch command is issued against Ethernet port 1 on the module in slot 14.
PAGE 121
Parameter Description Parameter Description set ethernet-port / Identifies the I/O module and port to be configured. -mtu Sets the maximum transmission unit. The default MTU value is 1500. Accepted values are between 1500 and 9194. You cannot set the MTU for a port if there are any vNICs are configured on that port. -flow-control Switches MAC flow control on or off for the port. You cannot change this setting if there are any vNICs configured on the port.
PAGE 122
vnics 10 vlans none ----------------------------------------------------------------- vNIC Mirroring The Oracle Fabric Interconnect supports vNIC Mirroring, which is a feature that enables duplicating data packets from one vNIC (the input) to a different output, either another vNIC (mirror to vNIC) or a physical Ethernet port (mirror to port).
PAGE 123
traffic flow on “vn1.pubstest” is copied and sent to output vNIC “vn6.pubstest”, which is connected to another server. When traffic is carried on vn1.pubstest,” the server “Raynor” gets a copy of all packets. Hardware Support vNIC Mirroring is supported on all versions of GE module. vNIC Mirroring is supported for intra-module traffic, so traffic received on an I/O Module is copied to a destination on the same module only.
PAGE 124
Mirroring to Port When you mirror to an output port, the traffic is copied to another physical port that is on the same I/O Module. Typically, the output port is then connected to a physical device, like a sniffer. However, this is not a requirement. The mirrored traffic is copied to the physical port regardless of whether the port has a physical connection or not.
PAGE 125
■ When mirroring to a port connected to a network switch, you should use the optional -mirror-mac= argument to target a specific hardware address as the destination for the mirrored packets. ■ When mirroring to a vNIC, Oracle suggests that you use the -mac-address option to specify a target device for the mirrored traffic. ■ Mirroring adds overhead due to duplicating packets on the mirror. You can expect a negative affect on I/O module performance when mirroring is enabled.
PAGE 126
Parameter Description -mirror-direction= Provides control over which direction of traffic will be mirrored. By default, mirroring is disabled (none). Specifying none on a configured vNIC disables vNIC Mirroring but leaves the feature configured. set ethernet-port Identifies the I/O module and port on which all vNICs will be mirrored. -mirror-all-vnics= Specifies the output destination for the mirrored vNICs.
PAGE 127
3. Display detailed vNIC information to verify that the mirror was created. Look for the mirror field at the bottom of the display. show vnic vn1.pubstest -detail ----------------------------------------------------------------name vn1.pubstest state up/up mac-addr 00:13:97:01:80:19 admin-ipaddr 0.0.0.0/32 oper-ipaddr 0.0.0.
PAGE 128
1. Set the input vNIC to mirror to an output vNIC: set vnic vn1.pubstest -mirror=vn6.pubstest 2. Set the direction of traffic that will be mirrored: set vnic vn1.pubstest -mirror-direction=both 3. Display detailed vNIC information to verify that the mirror was created. Look for the mirror field at the bottom of the display. show vnic vn1.pubstest -detail ----------------------------------------------------------------name vn1.pubstest1 state up/up mac-addr 00:13:97:01:80:19 admin-ipaddr 0.0.0.
PAGE 129
▼ Configure Mirror for All vNICs To mirror all vNICs on a specific port, use the set ethernet-port commands. In the following example, assume all vNICs terminated on slot 4, port 1 will be mirrored to slot 4, port 5. Also assume that only transmitted packets will be mirrored. 1. On the Ethernet Port, set all vNICs to be mirrored to a different port on the same I/O Module. For example: set ethernet-port 4/1 -mirror-all-vnics=4/5 This will set the mirror for all vNICs attached to port 4/1.
PAGE 130
4. If you need to undo the mirror-all-vNICs option, you use the -mirror-allvnics qualifier and specify none (or leave it blank). For example, to remove all mirror vNICs from slot 4, port 5: set ethernet-port 4/5 -mirror-all-vnics=none This will remove the mirror setting for all vNICs attached to port 4/5. Are you sure (y/n)?y ▼ Disable and Enable vNIC Mirrors You can disable and re-enable vNIC mirrors through the -mirror-direction qualifier, which defaults to none. To disable vNIC mirrors: 1.
PAGE 131
oper-ipaddr 0.0.0.0/32 descr if 4/1 if-state up mcast-group type dhcp mtu 1500 group group-pref ha-state flags --vlans none access-vlan 1 mode access queue-map-type disabled qos ilocal-id 0 mirror 4/7(rxOnly) ----------------------------------------------------------------1 record displayed If the mirror field is empty, the vNIC is not configured as a mirror. For example: show vnic vn4.pubstest -detail ----------------------------------------------------------------name vn4.
PAGE 132
mirror ----------------------------------------------------------------1 record displayed If the mirror field displays the output port or vNIC but also shows (none), the vNIC is currently disabled. show vnic vn1.pubstest -detail ----------------------------------------------------------------name vn1.pubstest state up/up mac-addr 00:13:97:01:80:19 admin-ipaddr 0.0.0.0/32 oper-ipaddr 0.0.0.
PAGE 133
A common use case for delayed leaves is an ESX environment where a single vNIC may be acting as the uplink for many guests. Typically, in this scenario, multiple guests are hosting the same application which joins a given group. With the delayed leave feature, a guest terminating its membership in a group does not interrupt traffic for other guests receiving multicast traffic on the same group. You can control delayed leaves on a per-I/O module basis through the set ethernet-card command.
PAGE 134
enables qas----------------------------------------------------------------1 record displayed Setting Allowed VLANs Per vNIC With the Allowed VLANs feature, you can specify a list of VLANs that are allowed to pass over any trunk vNICs. (Access mode vNICs cannot receive the Allowed VLANs setting.) With this feature, the Network Cloud has a set of ranges of VLAN IDs that are allowed, and only traffic that is in the specified VLAN range is allowed to pass over trunk vNICs connected to the Network Cloud.
PAGE 135
Parameter Description Parameter Description Specifies the name of the vNIC to which you are adding a range of allowed VLANs. Specifies the range of allowed VLANs, which can be either a single VLAN, a comma-separated list of VLANs, or a hyphenated range of VLANs. -detail Specifies that additional, detailed output is displayed. To display the allowed VLAN range, you must use this option.
PAGE 136
community-name allowed-vlans 1-4095 ----------------------------------------------------------------1 record displayed 2. Remove the existing VLAN configuration (if any) from the vNIC. For example: set vnic vnic1.pluto remove allowed-vlans * 3. Add the specific range of allowed VLANs to the vNIC. You can add the single VLAN, a comma-separated list of VLANs, or a hyphenated range of VLANs. For example: set vnic vnic1.pluto add allowed-vlans 109-149 4.
PAGE 137
TCP Segmentation Offload Virtual NICs support TSO, which allows large chunks of TCP data coming from a host server to be divided into a smaller TCP segments. When the host server sends one or more large chunks of data, they travel on the vNIC. On the 10 GE or 10-Port GE module where the vNIC is terminated, the large chunks of data are divided into smaller ones with a size determined by the maximum segmentation size (MSS). Then, these smaller MSS-size packets are sent onto the network.
PAGE 138
not support TSO. If you have any question about the ability of a Gigabit Ethernet card to support TSO, issue the show iocard command. If the I/O card has the EthIB suffix, as shown, then that module supports TSO.
PAGE 139
enabled for TSO to function. Because TSO is configured at vNIC creation time, you cannot disable this feature through runtime on the Oracle Fabric Interconnect. If you want to disable TSO, you must delete the vNIC and recreate it without the -tso flag. 1. On a Oracle Fabric Interconnect, if a Server Profile does not already exist for the vNIC, create one now. 2. Add the vNIC with TSO enabled. For example: add vnic pubstest.
PAGE 140
4. Check the flags field for ct where: t indicates that TSO is configured on the vNIC. c indicates that checksumming is enabled on the TSO vNIC. Receive Batching Virtual NICs support Receive Batching (also called receive packet coalescing). This feature allows numerous small packets coming from the network to be bundled together into fewer and larger frames as they travel over the vNIC.
PAGE 141
Syntax add vnic -batching=[{default|true|false}] show vnic |* [-detail] Parameter Description The Receive Batching feature takes the following options: Parameter Description |* Specifies the name of a vNIC on which Receive Batching will be configured, or all vNICs if you are using the wildcard. Specifies the slot and port termination for the vNIC that will support Receive Batching.
PAGE 142
3. Check the detailed vNIC configuration to verify that Receive Batching is configured. For example: show vnic pubstest.foo -detail ----------------------------------------------------------------name pubstest.
PAGE 143
CHAPTER 7 Virtual LANs (VLANs) A Virtual LAN (VLAN) is a private, independent, logical network that is created within a physical network. A VLAN behaves like an ordinary LAN, but connected devices do not have to be physically connected to the same network segment. This chapter explains how to configure the Oracle Fabric Interconnect for hostmanaged and chassis-managed VLANs.
PAGE 144
In this example, only the default VLAN (VLAN 1) is present. As a result, no isolation exists, and all hosts (s1, s2, s3, and s4) can see each other on the network. This scenario might not be preferable in situations where you want some hosts to be isolated from others. You can use VLANs to accomplish the isolation. Consider the example in the following figure. In this figure, VLANs are configured.
PAGE 145
These properties collectively define rules by which the Oracle Fabric Interconnect operates and handles VLAN tagged and untagged frames. When configuring a VLAN, you must set VLAN properties on the Ethernet port before any vNICs are assigned to it. If you are changing the VLAN settings of an Ethernet port that already has configured vNICs bound to it, you must remove all the vNICs associated with the terminating port before changing the settings. Otherwise, set commands fail.
PAGE 146
Properties VLANs are configured through various VLAN properties on Ethernet ports and vNICs. Ethernet Port The port properties documented in this section are supported on all versions of GE I/O modules. Mode This property specifies how VLAN tagged and untagged frames must be handled across the port. It also indicates whether the port can be configured to carry multiple VLAN traffic or just allow untagged frames in its access VLAN domain only.
PAGE 147
trunk In this mode, by default, the port allows transmission and reception of both tagged and untagged frames. Any untagged frames arriving from the network are forwarded to the port’s access VLAN domain. For information about access VLAN property of the port, see “Access” on page 121. Furthermore, optionally, untagged frames can be forwarded with a default VLAN tag inserted as specified by the tag-native property. For more information about this property, see “Tag Native” on page 121.
PAGE 148
■ When tag-native=true, in the egress path, VLAN ID of the newly inserted tag on an untagged frame equals the access VLAN ID of the vNIC transmitting the frame. ■ When tag-native=true, in the ingress path, all untagged frames arriving from the network are dropped. You can change the tag-native property to “true” through the set ethernetport command, as shown: set ethernet-port -mode=trunk -tag-native=true set lag
PAGE 149
You can set the vNIC mode for the VLAN at vNIC creation time, by using the add vnic command as shown: add vnic . -mode=access|trunk The -mode option takes the following qualifiers: access By default, a vNIC operates in access mode, meaning it forwards (transmits or receives) only untagged frames in its access VLAN domain.
PAGE 150
■ chassis managed Host-Managed VLAN Configuration With this method, the user only adds a vNIC to the server on the chassis and manages its VLAN configuration from the server using the native operating system tools. To support host-managed VLANs, the vNIC must be created in trunk mode. For information about trunk mode, see “Mode” on page 122.
PAGE 151
All chassis-managed VLANs are flagged as static in the type column of the show vlans command. show vlans name state descr admin-addr oper-addr type -----------------------------------------------------------------------------100.vn1.fir up 0.0.0.0/32 40.40.40.11/24 hostManaged 100.vn1.xc15 up 40.40.40.10/24 40.40.40.10/24 static 2 records displayed The output of the show vlans command contains two address fields: admin-addr and oper-addr.
PAGE 152
Before adding a VLAN, you will find it helpful to: 1. Determine if vNIC tags must be preserved between the Oracle Fabric Interconnect and a peer Ethernet device. If tags must be preserved, the port mode must be set to trunk mode. If not, the port mode must be set to access mode. It is mandatory for the Oracle Fabric Interconnect’s port mode and the Ethernet switch’s port mode to be set to the same operational mode (either trunk or access) to transmit or receive tagged or untagged packets between hosts. 2.
PAGE 153
■ “Example 3: Port Trunk Mode, VLAN-Unaware Host” on page 129 ■ “Example 4: Port Trunk Mode, VLAN-Unaware Host and HA vNICs” on page 130 Example 1: Port Access Mode, VLAN-Unaware Host In this example, the hosts are not VLAN aware. As a result, they will not be isolated from other hosts. Consider the example in the following figure. In this example, the hosts do not require unique VLANs. When traffic is sent to or from hosts, the traffic is part of the same VLAN.
PAGE 154
2. Add vNICs to the hosts: add vnic vn1.s1 15/1 -mode=access -access-vlan=10 add vnic vn1.s2 15/1 -mode=access -access-vlan=10 3. On the hosts, configure the IP addresses. Example 2: Port Trunk Mode, VLAN-Aware Host In this example, the hosts are VLAN aware. As a result, they will be isolated from other hosts that are not part of the same VLAN. Consider the example in the following figure. In this example, the hosts are aware of VLAN tags.
PAGE 155
1. Configure the Ethernet port mode: set ethernet-port 15/1 -mode=trunk -access-vlan=1 2. Add vNICs to the hosts: add vnic vn1.s1 15/1 -mode=trunk -access-vlan=1 add vnic vn1.s2 15/1 -mode=trunk -access-vlan=1 3. On the hosts, configure the VLAN interfaces and assign an IP address to the VLAN. Example 3: Port Trunk Mode, VLAN-Unaware Host Consider the example in the following figure. In this example, traffic to and from the hosts does not contain any VLAN tags.
PAGE 156
1. Configure the Ethernet port mode: set ethernet-port 15/1 -mode=trunk -access-vlan=1 2. Add vNICs to the hosts: add vnic vn1.s1 15/1 -mode=access -access-vlan=20 add vnic vn1.s2 15/1 -mode=access -access-vlan=10 3. On the hosts, configure IP address information. Example 4: Port Trunk Mode, VLAN-Unaware Host and HA vNICs In this example, the following environment applies: ■ HA vNIC pairs are configured on the Oracle Fabric Interconnects. ■ The servers are using the Linux or Windows operating system.
PAGE 157
In this example, s1 and s2 are not VLAN-aware. The Oracle Fabric Interconnect is responsible for inserting and removing VLAN tags to keep traffic isolated. Host servers s1 and s2 have redundant paths. The configuration process for this example is as follows: 1. On Oracle Fabric Interconnect 1, configure the Ethernet port mode: set ethernet-port 14/1 -mode=trunk -access-vlan=1 2. Add vNICs to the hosts: add vnic vn1.s1 14/1 -mode=access -access-vlan=20 ha -primary add vnic vn1.
PAGE 158
132 XgOS User’s Guide • September 2014
PAGE 159
CHAPTER 8 Virtual Host Bus Adapters (vHBAs) The Oracle virtual Host Bus Adapter (vHBA) virtualizes HBA connectivity. It appears to the OS as a physical HBA and enables a server to have a Fibre Channel (FC) SAN attachment without having a physical HBA present. Instead of the host server using an HBA, an InfiniBand (IB) HCA is used which then virtualizes the HBA allowing for SAN connectivity.
PAGE 160
vHBA Topology The following figure displays a typical vHBA topology. An IB connection exists between the Oracle Fabric Interconnect and host servers supporting the OVN vHBA host software stack. Up to 24 IB ports are supported. A 2-port FC I/O card connects to a storage area network (SAN) FC switch fabric. All the host server vHBAs multiplex through the FC ports on the I/O card. A storage array is attached to the switch fabric.
PAGE 161
N_Port ID Virtualization (NPIV) enables multiple fibre channel initiators (WWNs) to log in and occupy a single physical port. Your switch device (between the Oracle Fabric Interconnect and the storage device) must support NPIV and NPIV must be turned on. Some switches might require a software upgrade to support NPIV. Without NPIV, a vHBA cannot log into the fabric. Note that some switches require configuring the max number of NPIV logins.
PAGE 162
■ (4 vHBAs) * (4 targets) * (2 LUNs) = 32, which complies with the rule of 256 total LUNs per host. This example shows that less than 256 LUNs per host are supported. ■ (1 vHBA) * (1 target) * (256 LUNs) = 256, which complies with the rule of 256 total LUNs per host, and the maximum of 256 LUNs per target Basic vHBA Configuration The following command syntax and example show basic vHBA configuration. Syntax add server-profile @:ServerPort add vhba .
PAGE 163
Optional Modifiers Modifier Description -detail An optional modifier that allows displaying detailed information (if available) for the vHBA. -local-id The identifier used by ESX Server to map its HBA to this vHBA. Enter 1 to 32. Each ID must be unique per ESX Server. -lun-mask Sets LUN mask on targets visible to this vHBA. For more about LUN masking vHBAs, see “LUN Masking” on page 169. -map Sets persistent SAN map. For more about persistent mapping, see “Persistent Binding” on page 142.
PAGE 164
2. Find an FC card (sanFc2Port4GbLrCard) on which you can terminate a vHBA: show iocard slot state descr type v-resources ----------------------------------------------------------------1 up/up sanFc2Port4GbLrCard 0 2 up/up sanFc2Port4GbLrCard 0 3 up/up sanFc2Port4GbLrCard 0 4 up/up sanFc2Port4GbLrCard 0 4 records displayed 3. Find an FC slot/port to which you will assign a vHBA.
PAGE 165
6. Repeat the steps again. Note – vHBAs must be distinct when created on distinct chassis. For example, you can not have VH1.SP1 on two different chassis that connect to one or more common servers. 7. Verify the vHBA was created and its state is “up”: show -list vhba vhba1.myserver ----------------------------------------------------------------name vhba1.
PAGE 166
vHBA Attributes Additional options are available for a vHBA through the set vhba command. These options allow for more customization of the vHBA than the basic configuration documented in the previous section. Syntax set vhba . [] -descr= -fabric-link-down-timeout={|default} -if={|none} -lun-mask={|none} -qos={|none} show vhba . [] alarms [-detail] show vhba .
PAGE 167
Parameter Description -if Sets a termination slot and port for the vHBA. This option can be used to terminate the vHBA on a different Fibre Channel port if needed. -lun-mask Binds a LUN Mask to a vHBA. The LUN Mask must already exist for it to be available for binding. This option is useful for adding a LUN Mask to a vHBA after the vHBA is already created. For more information, see “LUN Masking” on page 169. -qos Binds a SAN QoS Profile to the vHBA.
PAGE 168
When you bring a vHBA online (up) it will attempt to discover all the attached targets that are available to it. You can also have the vHBA relearn attached targets without having to bring the vHBA down, then up. For information, see “Target Prescan and Rescan” on page 146. Persistent Binding A target is a storage device on a SAN. A target can be a single disk, or it can have many devices (LUNs or volumes) within it.
PAGE 169
Parameter Description Parameter Description add san map Creates an ordered mapping of devices identified by World Wide Port Names (WWPN). The vHBA uses these SAN map device IDs in this order. All devices discovered by XgOS are subject to this binding filter. Missing devices are skipped and no substitutes are made. User-defined name for a map to configure on a new vHBA. A SAN map is the order in which the target disks come up (become active).
PAGE 170
Take the following steps to configure a persistent map (binding) for an undeployed vHBA. 1. Add a named SAN map and specify its fixed WWPN target order.
PAGE 171
3=21:00:00:20:37:90:88:90 4=21:00:00:20:37:C6:5E:B4 5=21:00:00:20:37:CC:EB:30 6=21:00:00:20:37:D5:37:18 7=21:00:00:20:37:8D:03:7D 1 record displayed 5. Bind the named server profile to a physical connection: set server-profile myserver connect ceasar@iowa:ServerPort13 6. Bind the vHBA to a physical slot/port: set vhba vhba101.myserver -if=1/1 At this point, the vHBA is bound to the persistent map named “mymap.
PAGE 172
Example: Configure Persistent Mapping While Creating a vHBA The persistent binding can be assigned while creating a vHBA, which is provided to you as a configuration convenience: add server-profile myserver ceasar@iowa:ServerPort13 add vhba vhba999.myserver 4/1 -map=mymap Example: Remove vHBA, Server Profile, and SAN Map To remove a vHBA, server profile, and SAN map in the correct order: remove -noconfirm vhba vhba101.
PAGE 173
The Oracle Fabric Interconnect relies on fibre channel’s Registered State Change Notification (RSCN) to send target-state updates from the remote switch to the Oracle Fabric Interconnect. The Oracle Fabric Interconnect’s IOP learns the update and notifies the host server of any changes. However note that RSCN is turned off by default on some fibre-channel switches. RSCN does not support reporting LUN state changes (add or remove).
PAGE 174
▼ Enable prescan To enable prescan for an unbound vHBA: 1. Create an unbound server profile, where the state is “unassigned”: add server-profile III show server-profile III ------------------------------------------------------------name III state up/unassigned ... 2. Create a vHBA under this unbound server: add vhba vhbaiii.III 4/1 At this point, show vhba . will report the state as “resourceUnavailable,” which is expected. The vHBA is not bound to a server. 3.
PAGE 175
wwpn 50:01:39:70:00:00:F1:02 map lun-mask local-id 0 -----------------------------------------------------------------------1 record displayed ▼ Bind After prescan The ideal scenario is to bind the prescan-discovery results to a host server. XgOS supports binding the server profile with the phys-con after a prescan is complete, as long as you follow the correct configuration order. Follow these steps to perform a prescan then bind the server profile: 1.
PAGE 176
5. If you are satisfied with the results, bind the server-profile: set server-profile III connect titan@ServerPort23 From now on, this vHBA has become a normal vHBA. You can run rescan against it: set vhba vhbaiii.III rescan Note – You can no longer run prescan against this normal vHBA. Example: remove-prescan You can issue prescan several times. However to detect LUN changes, the prior prescan state must be removed (remove-prescan) from the vHBA before you can re-issue prescan again: set vhba vhbaiii.
PAGE 177
4. Configure this vHBA to rediscover (rescan state) the available LUN information. If there are any LUN changes, they will be reflected after this rescan operation: set vhba vhba888.titan rescan 5. Display any new target and LUN information: show vhba vhba888.titan targets Set FC Card Attributes You can control the Fibre Channel card operational state through the set fc-card command.
PAGE 178
Parameters Parameter Description *| Specifies the physical slot to which the command will be applied. An asterisk (*) specifies all available FC cards. -descr= Applies a text description to the FC module. Quotes are required around multiple words containing spaces in between. Specifies a particular Fibre Channel port for which information will be displayed.
PAGE 179
Watch FC Card Utilization With the watch fc-card command, you can run a command that continuously updates the Fibre Channel card utilization over a period of time. The watch fc-card command (and watch fc-cards) will intermittently poll the Fibre Channel card and display the updated, real-time statistics automatically. The polling window is fixed (not configurable), and the command is useful for seeing how the card is being used as well as viewing trends in usage over time.
PAGE 180
vhbas 1 ----------------------------------------------------------------1 record displayed The most commonly used fibre-channel controls are rate, topology (topo), framesize, and execution-throttle. However, note that modified attributes do not take effect until you reset the I/O card. See the example that follows.
PAGE 181
Parameter Description -descr= Applies a text description to the FC port. Quotes are required around multiple words containing spaces in between. -detail Enables you to display detailed information about the Fibre Channel port configuration, properties, and state. Some Fibre Channel port parameters are displayed only through this option.
PAGE 182
Parameter Description -topology={f-port|l-port|n-port} Specifies the type of storage connection to be used. The following parameters are supported: • f-port specifies a point-to-point connection to the storage device through a Fibre Channel switch. This type of connection supports NPIV. This is the default configuration for a Fibre Channel I/O module. • l-port specifies a loop connection to the storage device without using a Fibre Channel switch. This type of connection does not support NPIV.
PAGE 183
type sanFcPort state up/up descr wwnn 50:01:39:71:00:00:B0:1F wwpn 50:01:39:70:00:00:B0:1F rate auto/4Gbps frame-size 2048/2048 exec-throttle 65535 int-delay 1000 fc-link-down-timeout 10 login-retry 8 login-timeout 4 fc-target-port-down-timeout 60 topo F loop-delay 5 tape-support true vhbas 4 ----------------------------------------------------------------1 record displayed ▼ Configure a Port for Direct-Attached Storage The Oracle Fabric Interconnect supports direct-attached storage.
PAGE 184
Removing vHBAs To support the graceful deletion of vHBAs, you must follow the documented procedures. Failure to do so might cause instability on the host server. General Procedure In general, the process of removing a vHBA is the same for every situation: 1. On the host server, stop I/O that uses the vHBA that you want to delete. 2. On the Oracle Fabric Interconnect, remove the vHBA. 3.
PAGE 185
Environments Where Special Procedure is Required If the vHBA is in any of the following environments, use the specialized instructions listed in “Procedures for vHBA Delete in Special Environments” on page 159: ■ vHBAs connected to any VMware virtual machine. ■ vHBAs in a Linux multipathing environment. ■ vHBAs that mount a file system for a Linux server. Caution – If your vHBA is in any of these special situations, do not use the general procedure.
PAGE 186
Syntax remove vhba . [] [-noconfirm] Remove vHBAs Connected to VMware Virtual Machines OVN supports VMware ESX, which is documented in the following procedures for removing vHBAs on virtual machines. Select one of the following: ■ If you are using VMFS, follow the instructions in “Removing vHBAs Connected to Virtual Machines Using VMFS” on page 160.
PAGE 187
Note – Changing the storage configuration when using VMware might require a rescan to discover those changes and propagate them to the ESX GUI. Remove vHBAs Connected to Servers Using Direct Disk Access Use the following procedures for removing vHBAs when the host server is accessing the storage directly. These procedures apply to regular Linux servers and to VMware virtual Linux and Windows servers that do not use VMFS.
PAGE 188
Note – Changing the storage configuration when using VMware might require a rescan to discover those changes and propagate them to the ESX GUI. ▼ Removing vHBAs Connected to Windows Servers Hosted in VMware This procedure applies only to Windows virtual machines that do not use VMFS to access storage. For regular Windows servers, use the general procedure as listed in “General Procedure” on page 158. Use the following procedure to gracefully remove the vHBA: 1.
PAGE 189
10. Log in to the I/O Director. For example, from a secure command prompt, enter the following: ssh admin@
Password: where
is the IP address of your Oracle Fabric Interconnect and is your administrator account password. 11. On the Oracle Fabric Interconnect, remove the vHBA.
PAGE 190
Note – Rebooting the server will cause service interruptions for any running applications, so this procedure is recommended only for host servers that are running non-mission critical applications. If your host server cannot be rebooted, see “Removing a vHBA While Maintaining Service” on page 164. To gracefully remove a vHBA from a Linux host server in a multipathing environment, perform the following steps: 1. On the host server, stop all I/O on the vHBA that you want to remove. 2.
PAGE 191
To gracefully delete a vHBA connected to a Linux host server in a multipath environment, follow this procedure: 1. On the host server, stop all I/O on the vHBA that you want to remove. 2. Shut down all applications that might be using storage resources through the vHBA. 3. On the host server, unmount all file systems attached to the Xsigo driver. umount 4. On the host server, stop the PowerPath service by issuing either of the following commands: ■ /etc/init.
PAGE 192
manually rescan without having to unload vHBA drivers from the host server, then reload them. This procedure is useful in situations when you are making minor changes to the storage network—for example, adding disks to a JBOD—instead of adding, deleting, or changing entire devices. Note – You cannot use this procedure if your server is SAN-booted because this procedure requires the xsigo-scan -r command, which removes all devices including the SAN boot device.
PAGE 193
8. On the host server, start the PowerPath service, by issuing either of the following commands: ■ /etc/init.d/PowerPath start ■ service PowerPath start vHBA Statistics Operational and performance statistics are available for individual vHBAs through the show vhba command (as shown in the following example). show vhba vhba1.crawford stats ---------------------------------------------------------------name vhba1.
PAGE 194
dqp-ib-remote-disconnect-err-count 0 ----------------------------------------------------------------1 record displayed You can also display vHBA statistics for multiple vHBAs through the use of the option (for example show vhba stats command. Fibre Channel Monitoring Use show fc-port to display Fibre Channel port information. Use set fc-port to control the Fibre Channel port settings. See “Set FC Port Attributes” on page 153.
PAGE 195
fc-target-port-down-timeout 60 topo F loop-delay 5 tape-support true vhbas 1 ---------------------------------------------------------1 record displayed show fc-port 8/1 stats ----------------------------------------------------------------name 8/1 controller-errs 0 device-errs 0 link-fails 0 loss-of-syncs 1 loss-of-signals 0 primitive-seq-protocol-errs 0 transmission-word-errs 0 crc-errs 0 ----------------------------------------------------------------1 record displayed LUN Masking Logical Unit Number (L
PAGE 196
In this figure, the Oracle Fabric Interconnect controls which LUNs can be seen by the vHBAs. To accomplish this, the Oracle Fabric Interconnect deploys different vHBA policies (vHBA-A, vHBA-B) to maintain LUN security. When a vHBA is created, a different LUN mask is assigned. RSCN does not report LUN state changes. Whenever the LUN masking changes on an existing vHBA, you must also issue a rescan on the Oracle Fabric Interconnect to send an RSCN update. See “Parameter Description” on page 171 for details.
PAGE 197
Syntax add san lun-mask [] target lun {|all|none} add vhba . / {-lun-mask=|-no-lun-masking|none} set vhba . {-lun-mask=|none} show vhba . [] -lun-mask={|none} [-detail] show vhba . [] targets [-detail] By default LUN masking is not applied to a vHBA. All LUNs are visible by default.
PAGE 198
▼ Set a LUN Mask 1. Create a LUN Mask named “oracle-mask” with target WWPN “20:70:00:C0:FF:0A:81:30” and LUN ID “11”: add san lun-mask oracle-mask target 20:70:00:C0:FF:0A:81:30 11 2. Create a server profile and bind it to a physical connection: add server-profile testlin2 testlin2@washington:ServerPort13 3. Create a vhba and bind the LUN Mask “oracle-mask” to it: add vhba oracle-vhba1.testlin2 1/1 -lun-mask=oracle-mask Now check to see the mask is correct.
PAGE 199
oracle-mask 21:78:00:C0:FF:0A:81:30(0,9), 20:70:00:C0:FF:0A:81:30(0,11) 1 record displayed 6. Display the LUNs that vHBA “oracle-vhba1” is allowed to see: show vhba oracle-vhba1.testlin2 lun-mask vhba name descr targets ---------------------------------------------------------------oracle-vhba1.testlin2 oracle-mask 21:78:00:C0:FF:0A:81:30(0,9), 20:70:00:C0:FF:0A:81:30(0,11) 1 record displayed However, before the rescan, the change will not take effect: show vhba oracle-vhba1.
PAGE 200
Optional LUN Masking: No Report LUN Interception When a host (Linux or Windows) issues a SCSI report LUNs, the chassis filters the response based on what is in the Oracle Fabric Interconnect database. If LUN masking changes in an array and a host issues a report LUNs, the new LUN will not be available to the host until a set vhba rescan command is run on the Oracle Fabric Interconnect. In some cases, this approach goes against customer expectations and breaks the existing model.
PAGE 201
Syntax add vhba . [] / -no-lun-masking Example To determine if LUN masking is enabled for a vHBA, see the “l” value under “flags”. This filed means LUN masking is enabled: add vhba bar.myserver 1/2 show vhba bar.myserver -detail ----------------------------------------------------------------name bar.
PAGE 202
▼ Change Port Topology from Fabric to Loop XgOS supports changing a SAN topology or migrating from one topology to another—for example, changing the port topology from Fabric (f) to Loop (l). You can set the port topology type by using set fc-port topology command. Changing the port topology can be done in real-time and does not require an FC card reset. To change the topology type from f to l, or l to f without resetting the card, follow this procedure: 1.
PAGE 203
2. Delete all the vHBAs terminated on the port for which you will change the port topology. For example: remove vhba vh3.frick remove vhba vhsan.brack-sanboot 3. If you do not want to delete the vHBAs, you can do the following steps: a. Set the vHBAs on the port to “down” state b. Set the interface to “none” c. Make the port topology change d. Disconnect the fibre channel cable, then reconnect the fibre channel cable e. Set the interface on the vHBAs back to this port. 4.
PAGE 204
178 XgOS User’s Guide • September 2014
PAGE 205
CHAPTER 9 VMware ESX Servers This chapter describes configuring virtual I/O for VMware ESX in the following sections: ■ “Introducing Virtual I/O for VMware ESX” on page 179 ■ “VMware ESX Support in XgOS” on page 180 ■ “ESX Utilities” on page 181 ■ “ESX Configuration” on page 182 ■ “Caveats” on page 188 ■ “Automatic Rescans in ESX” on page 189 Introducing Virtual I/O for VMware ESX From the Oracle Fabric Interconnect’s viewpoint, a VMware ESX server appears and works similar to a standard serve
PAGE 206
install the host drivers to see 32 vNICs not added or attached to anything. These are placeholders for when the interfaces are associated to Virtual Machine Networks. ■ Predefined vHBAs—In the configuration section of VMware Infrastructure Client, a list of 12 virtual storage adaptors are pre installed as soon as you load the Oracle host drivers. A WWN appears next to the adaptors that are configured for the Oracle Fabric Interconnect.
PAGE 207
Syntax Create a server profile: add server-profile @: then add a vNIC or vHBA with a local-id value: add vnic . / -local-id= add vhba . / A local-id maps a vNIC into 32 predefined vNIC names (vnic1 through vnic32) on the ESX server. A local-id for a vHBA is rarely used. See “Introducing Virtual I/O for VMware ESX” on page 179.
PAGE 208
ESX Configuration The ESX server in the following figure has four virtual machines (Service Console, bob, fred, joe). Each VM has Ethernet interfaces (eth0 ... 20500), a vSwitch, and belongs to a Virtual Machine Network. VNICs will appear as “vnic1”, “vnic2”,”vnic3”, and so on, up to a “vnic32”. You can have any number of vSwitches (vSwitchN), and any given vSwitch can associate with any number of vNICs.
PAGE 209
1. Install the InfiniBand RPM on the ESX server: rpm -ivh VMware-esx-commsrc-infiniband-release-3.8.0-1.09.60.rev401.i386.rpm Linux ships with its own IB drivers, but the ESX server does not. This IB RPM file must be installed before the Xsigo ESX Commsrc file (next step). 2. Install the Xsigo VMware host drivers on the ESX server: rpm -ivh VMware-esx-commsrc-xsigo-release-3.8.0-v99x3.8.0.i386.rpm reboot 3.
PAGE 210
Oracle created a Fabric Manager VMware Extension that operates with VMware VirtualCenter. The extension runs the Oracle Fabric Manager web interface. It enables you to display and manage your virtual I/O as a plug-in service to a VMware Infrastructure Client connection to VMware VirtualCenter. 5. To use the VMware extension, install the Xsigo ISO or zip file to VMware VirtualCenter: Fabric Manager-plugin4vc-3.8.0.iso Fabric Manager-plugin4vc-3.8.0.
PAGE 211
6. If the Virtual I/O tab is not displayed, click Plug-ins on the toolbar to verify the state of the Oracle Fabric Manager. This figure shows an example of the Plug In Manager with Oracle Fabric Manager enabled.
PAGE 212
If the Oracle Fabric Manager has not been integrated into Virtual Infrastructure Client, no Fabric Manager plug-in is displayed in the Plug-in Manager. If the Oracle Fabric Manager has been integrated, but a problem exists, an Fabric Manager entry is displayed in the Plug-in Manager, but the Status field will show “Disabled.” After you log into Oracle Fabric Manager, the Dashboard is displayed as shown this figure.
PAGE 213
▼ Monitor vNICs 1. From the Oracle Fabric Interconnect, monitor the health of the vNICs: show vnic . -detail All configuration can be done via the VMware Virtual Infrastructure Client. However on the ESX Server, there are many useful CLI commands available to you. 2. To find the device mapping between the pre-installed virtual resources and the ones that are attached into the Oracle Fabric Interconnect: esxcfg-xgmap vh0 -> vmhba32 vh1 -> vmhba34 vn10 -> vnic10 vn11 -> vnic11 vn12 -> vnic12 ..
PAGE 214
Caveats The following sections cover specific issues of which you must be aware when configuring virtual I/O resources for ESX servers. Set Local ID for Resources on Down or Unattached Server Profiles You must explicitly set the local-id on vNICs and vHBAs that are added to an unattached or administratively down server profile. local-id will be set automatically on resources that are added to an active server profile only. Example: add add add set server-profile server1 vnic vnic1.
PAGE 215
Automatic Rescans in ESX In ESX, discovery of new vHBA devices and changes to the FC fabric often require a rescan. Rescanning propagates fibre-channel information to the various components including storage driver, operating system, and management software. The OVN driver package includes a thin daemon called xsigo-hotplug which attempts to periodically propagate this information automatically.
PAGE 216
Also, note that using the command line tool esxcfg-rescan does not propagate information to the VI/VC-Client datastore, but instead only updates the VMKernel and ConsoleOS. You must press the “Rescan” button in the “Storage Adapters” configuration tab before they will be visible in the GUI-client. Note – One situation when you might need to request a rescan is when you remove or reconfigure a vHBA.
PAGE 217
CHAPTER 10 Network QoS for vNICs Oracle’s network Quality of Service (QoS) provides administrators the ability to treat packets differently, based on the type of traffic. This chapter explains how to configure a QoS policy and apply it to virtual resources.
PAGE 218
Note – The SAN QoS feature set uses vHBAs (not vNICs) and is different from network QoS. See “SAN QoS for vHBAs” on page 209. Network QoS assigns the amount of bandwidth and burst size to a given vNIC. The burst size is the amount of buffering retained for when traffic arrives in bursts during congestion. Bandwidth Guaranteed bandwidth on vNICs is supported through the CIR and PIR values: ■ CIR—Committed Information Rate. The amount of bandwidth guaranteed to the vNIC. The CIR is best effort.
PAGE 219
CBS and PBS values can also be automatically calculated to linear values based on the CIR and PIR values that you specify. This feature is supported for custom default sets only. For more information about automatic calculation, see “Automatic Calculation” on page 199. Network QoS Services XgOS provides Network QoS Policing services. Policing enforces rate limits on traffic to a designated rate. There are two ways to configure network QoS: Default Sets—Use the default set profiles (recommended).
PAGE 220
Each vNIC has eight traffic queues. Network traffic can enter on any of the queues, but the QoS Profiles are applied to vNIC traffic on all queues. If QoS parameters have been assigned to the vNIC, the associated vNIC Profile is applied to the traffic. As a result, the information rate and burst size parameters are applied to the traffic. Because network QoS Policer profiles are configured, any traffic that is out of the range created by the CIR, PIR, CBS, and PBS parameters is dropped.
PAGE 221
QoS Feature Matrix The following table describes the network QoS features supported Oracle Fabric Interconnect’s Gigabit Ethernet I/O modules. 10-Port Gigabit Ethernet Module Feature Ingress and egress policing 10GbE Module (1 port) Yes Yes mapping* Yes Yes IP TOS mapping Yes Yes DSCP mapping Yes Yes Assigning sets to a card Yes Yes 802.1p * See the “mark” option in “Setting Actions” on page 214. Note – 802.
PAGE 222
Note – If you have multiple 10 GE or 10-Port GE cards and want to deploy the same QoS policy to all the cards irrespective of vNIC movement, then use the same tested QoS set for all the cards. Each time a vNIC moves across I/O cards, it will be treated with the same QoS behavior. Applying different QoS sets to different cards does not guarantee QoS for vNIC movement. 4. Issue the following commands to display the default profile names and settings for the policer.
PAGE 223
Syntax set vnic {ingress-qos|egress-qos} -policer=default/ [enable|disable] show vnic show qos network policer [*|] A profile itself has no direction (ingress or egress). You must explicitly apply two profiles (one for each direction) to each object. No QoS is available for a traffic direction that is not specified. The system allows you to disable QoS on a specific vNIC. The default is enable.
PAGE 224
1. Create a custom QoS set. 2. Specify a profile within the set. Repeat this step as needed to define all profiles in the custom QoS set. As an option, you can also specify one or more profiles in the QoS set after it has been assigned to the I/O card. 3. Assign the custom QoS set to an I/O card. 4. Associate the profile to a vNIC and specify a traffic direction (ingress or egress).
PAGE 225
Automatic Calculation Automatic calculation ensures that the optimal linear-function settings are configured. XgOS supports the automatic calculation of CBS and PBS. When you specify the CIR and PIR as the first and second QoS parameters (but do not specify CBS and PBS), XgOS automatically calculates the equivalent CBS and PBS values.
PAGE 226
In this example, Server1 attaches to an Oracle Fabric Interconnect over a vNIC. The Oracle Fabric Interconnect is fitted with one 10 GE I/O card in slot 4 that connects to a vNIC attached host. The Oracle Fabric Interconnect sends traffic to Server1 over a vNIC named “test_1.whitney”. The QoS policer restricts the amount of ingress traffic (from network to server) arriving on Server1 to 100 Mbps. The egress traffic (from server to network) is also policed to 100 Mbps.
PAGE 227
4. On the same vNIC, enable policing in the egress direction (server to network): set vnic test_1.whitney egress-qos -policer=foo/2 A profile itself has no direction. You must explicitly apply two profiles (one for each direction) to each object. QoS is available for a traffic direction only if the direction is specified. 5. Verify the policer policy was assigned to the vNIC. The “policer” field indicates which policer policy was assigned to the vNIC.
PAGE 228
ACLs With QoS and Application QoS ACL rule configurations can be used with QoS. Specify an action for each matched condition. A condition identifies the application flow to be chosen. An action specifies what to do with that flow. The following figure shows an example of an ACL with a QoS set. From an ingress viewpoint traffic flows from the network, into a port, into a vNIC, into 1 of 8 queues, and onto a server. Each of the packets are evaluated against the defined ACL rules.
PAGE 229
Example: ACL-Based Policer for 10GbE I/O Cards An ACL-based policer sets up an ACL that matches a particular flow, then polices that flow using QoS. For example, you can police communication between two IP endpoints down to a specific rate. Or, you can police based on traffic type port number (i.e., HTTP 80). ACL-based policers are supported on the 10 GE and 10-Port GE modules only. The following figure shows an example of limiting egress traffic.
PAGE 230
3. Create an ACL and assign it a name: add acl web100m Warning: ACLs are not autocommitted. 'commit' when the ACL is complete You will need to enter No auto commits exist for ACLs. You must issue commit (see Step 5) after the ACL is defined completely. 4. Define the ACL condition and action. The ACL names and rule numbers must match.
PAGE 231
descr type nwEthernet1Port10GbCard vnics 1 acl web100m enables qa -------------------------------------1 record displayed Disabling QoS on a vNIC XgOS allows you to disable QoS for either ingress or egress traffic on a per vNIC basis. You can disable network QoS in different ways: ■ Naming the Policer. If you disable QoS by naming a specific Policer, QoS is no longer active but the Policer remains attached to the vNIC.
PAGE 232
By using this example, the default Policer is disabled, but remains bound to the vNIC. QoS is disabled for ingress traffic only, so egress traffic is not affected. To disable the custom policer named “qostest/200” for egress traffic on a vNIC named “foo”: set vnic foo.bar egress-qos -policer=qostest/200 disable By using this example, the custom Policer is disabled, but remains bound to the vNIC. QoS is disabled for egress traffic only, so ingress traffic is not affected.
PAGE 233
802.1p user priority IP Precedence/TOS Queue Number Best Effort, 2 2 2 Spare, 1 1 1 Background, 0 Normal, 0 0 See the ACL mark option in “Setting Actions” on page 214. Note – 802.1p and IP Precedence mapping is supported, but all queues currently have the same priority. The Oracle Fabric Interconnect uses a weighted fair queue (WFQ) algorithm to determine packet priorities.
PAGE 234
208 DSCP Name Value (Binary) Queue Number AF12 001100 1 AF11 001010 1 DF (Other) 000000 0 XgOS User’s Guide • September 2014
PAGE 235
CHAPTER 11 SAN QoS for vHBAs Oracle’s vHBAs support QoS where the bandwidth is rate limited with shaping (not dropped). There are no queues or policers associated with FC traffic—only shapers. This chapter presents the SAN QoS features and how to configure them in the following sections: ■ “SAN QoS Features” on page 209 ■ “Commands” on page 210 Note – See “Virtual Host Bus Adapters (vHBAs)” on page 133 for information about non QoS vHBA features.
PAGE 236
■ Using ACLs with SAN QoS ■ Automatic calculation on SAN QoS for CBS and PBS ■ Ingress vs egress direction control Commands QoS shaping services can be applied to FC cards by using add qos san and set qos san.
PAGE 237
▼ Create vHBA With Shaping Take the following steps to create a SAN QoS shaping policy and apply it to a vHBA: 1. Create a named QoS shaping policy. The policy name is “test” in this example: add qos san test 2. Configure the shaping-policy values.
PAGE 238
212 XgOS User’s Guide • September 2014
PAGE 239
CHAPTER 12 Access Control Lists Access control lists (ACLs) classify packets. The classification result can be applied to quality-of-service application flows (mark, police) or to network-access control (deny, allow). There are many use cases for ACLs. Consider the following examples: ■ Prioritizing outbound traffic by marking fields in the IP header, thereby enabling upstream routers to handle this marked (set) traffic in a specific way.
PAGE 240
Setting Actions You specify an action to be taken whenever a packet matches the specified condition. For each action except police and enqueue, you also specify a traffic direction: ingress, egress or both.
PAGE 241
Parameter Description Parameter Description rule The ID number for this rule. -rank= A specific rank for the evaluation order of this rule. If you do not specify a rank, the Fabric Interconnect assigns a default rank based on the size (in bits) of the information in the condition definition. Thus, if all rules in a set are defined according to different types of information, the default ranks are unambiguous and you do not need to assign a specific rank to your rules.
PAGE 242
Example set acl foo rule 3 action learn ingress Setting Conditions An ACL condition is a match-test rule to perform on a packet. A condition defines rules for fields the system checks during packet processing. Operators are available to match strings in those fields that follow a specific pattern. Rule conditions, and rules themselves, can be modified and reassigned on the fly.
PAGE 243
Operators Operators match strings following a specific pattern. Use an operator in the following table to define how a field should be checked, where can be any of the following. Operator Description < Less than. Value of the field is less than the specified value <> Not equal to. Value of this field is not equal to (i.e., anything other than) the specified value. = Equal to (including masks if appropriate).
PAGE 244
Example set acl test rule 1 condition dest ipaddr = 10.1.1.1 mask 255.255.255.255 show -list acl test --------------------------------------------------------------------------name test rule 1 rank 0 descr conditions dest ipaddr = 10.1.1.1 mask 255.255.255.255 action --------------------------------------------------------------------------1 record displayed Displaying ACLs and Rules Use the show acl command to display configured ACLs or their component rules.
PAGE 245
Examples Display all ACLs on the system. show acl * name rule rank descr conditions action -----------------------------------------------------------------------------renoset 1 0 allow, learn both renoset 2 0 src ipaddr exactly 192.168.1.1/32 test 1 0 dest ipaddr exactly 10.1.1.1 web100m 1 0 dest port allow, exactly 80 police=test/100mhttp both 4 records displayed Display an ACL by name (“renoset”) and its rules.
PAGE 246
Note – If you need to change an ACL rule, you do not need to remove the rule. You can change the rule in real-time while the ACL is still attached to an I/O module by issuing the set acl command to rewrite the rule Syntax remove acl remove acl * remove acl rule Parameter Description Parameter Description Removes a single ACL. * Removes all ACLs.
PAGE 247
▼ Deny Egress Traffic Take the following steps to deny egress traffic: 1. Create a named policy set (empty by default). No implicit assumptions or rules are made in this empty set. The set in this example is named “block16_5”: add acl block16_5 Warning: ACLs are not autocommitted. 'commit' when the ACL is complete You will need to enter Note – As indicated by the display message, the commit command must be issued after you define the condition and action. See Step 3. 2.
PAGE 248
3. Issue a commit after the ACL is defined: commit Are you sure you want to commit these changes (y/n)?y This command collects all the multiple configuration steps of your policy and stores them into the chassis’ database. 4. Specify the I/O card and apply the named ACL: set ethernet-card 3 acl -set=block16_5 The same set can be attached to multiple cards (one at a time). Once attached, the policy is downloaded and programmed into the card.
PAGE 249
7. Display ACL statistics. In this example, the “acl-deny-pkt-counter” is equal to “6”, which indicates packets are being dropped (as expected): show iocard 3 acl-stats name block16_5 acl-rule-set 1 acl-rule 1 acl-deny-pkt-counter 6 acl-mark-tos-counter 0 acl-mark-dot1p-pkt-counter 0 acl-enqued-pkt-counter 0 acl-learned-flows-counter 0 ----------------------------------------------------------------1 record displayed 8.
PAGE 250
224 XgOS User’s Guide • September 2014
PAGE 251
CHAPTER 13 Link Aggregation Groups (LAGs) A link aggregation group (LAG) is a grouping of physical Ethernet ports, which enables you to combine multiple individual physical Ethernet ports into one logical port group. As a result, the ports combined into a LAG can operate in parallel with the benefit of increased bandwidth and high availability. This chapter explains how to configure a LAG and provide those benefits to your virtual resources.
PAGE 252
Note – In many ways, configuration and management of LAGs is similar to configuration and management of Ethernet ports. Link Aggregation Control Protocol (LACP) The OVN implementation of LAG supports Link Aggregation Control Protocol (LACP) to manage link aggregation. LACP allows a network device to negotiate an automatic bundling of links by sending LACP packets to the peer device, which must also support LACP. the Oracle Fabric Interconnect supports static and dynamic LAGs.
PAGE 253
When a LAG contains multiple ports, traffic is mapped to the appropriate port based on a hashing algorithm that considers the following parameters of the packets originating from the vNICs in a LAG: ■ Source and destination MAC address and (if applicable) ■ IPv4 source and destination addresses and (if applicable) ■ TCP and UDP source and destination ports (if applicable) Ports that are assigned to a LAG remain part of the LAG regardless of their state.
PAGE 254
■ LAG port changes are not revertive. As a result, if a port in the LAG goes down, traffic flows are remapped based on the hashing algorithm (see “Port Assignment in the LAG” on page 226). When the port comes back up, traffic does not revert back to the port that was originally supporting the traffic. Only new traffic flows can be mapped onto this port.
PAGE 255
Syntax add lag port -lacp set lag {<*|*.*|} -access-vlan=|default set lag {<*|*.*|} -descr= set lag {<*|*.*|} -flow-control=default|false|true set lag {<*|*.*|} -igmp-snooping=default|false|true set lag {<*|*.*|} -mode=access|trunk set lag {<*|*.*|} -mtu=|default set lag {<*|*.*|} -rate=100m|1g|autoNegotiate|default set lag {<*|*.*|} -tag-native=true | false show lag {*|*.
PAGE 256
Parameter Description -igmp-snooping= Specifies whether IGMP Snooping is enabled for the LAG. -mode Specify access mode (access) or trunk mode (trunk) of operation: • In access mode, the LAG carries only transmitted or received packets in the Access VLAN domain. • In trunk mode, all VLAN traffic is transmitted or received by the LAG. -mtu= Sets the maximum size of packet transmitted without fragmentation. Specify a number between 1500 and 9194. By default (default) 1500-byte packets are used.
PAGE 257
The following series of commands shows you how to set LAG options individually, but you can specify them all in one command. For example: set lag 5.4 -access-vlan=2400 -flow-control=true -mtu=4906 -rate=100m This command accomplishes the same task as the individual commands in the following examples: To set the access VLAN ID, you would issue the set lag -access-vlan command. For example, to set the native VLAN ID to 2400 for LAG 5.4: set lag 5.
PAGE 258
Commands on peer: config term interface range GigabitEthernet1/0/23-24 switchport mode trunk switchport trunk encapsulation dot1q switchport trunk allowed vlan 10,99 channel-group 1 mode on There is no dependency for when to configure the peer or the Oracle Fabric Interconnect, so these commands can be issued on the peer before or after LAG is configured on the Oracle Fabric Interconnect.
PAGE 259
Note – Total number of ports that you can add to a LAG depends on the number of LAGs configured, with the maximum number of ports ranging from 10 configured in a single LAG to 2 ports configured in each of five LAGs. 4. When all ports have been configured in the LAG, verify that the LAG was correctly added to the Oracle Fabric Interconnect, by issuing the show lag command. show lag name state descr rate mtu mode access-vlan flags ports ----------------------------------------------------------------5.
PAGE 260
▼ Configure a LAG for “Passive” Mode (Dynamic LAG) When configuring a passive-mode LAG, you configure a LAG on the Oracle Fabric Interconnect and use the -lacp qualifier to allow passive mode. Passive-mode LAGs are formed dynamically through LACP, when a peer device is set to active mode and requests that the passive port is added to the LAG. 1. Configure the LAG on the peer. The LAG must be configured for “active” mode. 2.
PAGE 261
4. When all ports have been configured in the LAG, verify that the LAG was correctly added to the Oracle Fabric Interconnect, by issuing the show lag command. show lag name state descr rate mtu mode access-vlan flags ports vnics ----------------------------------------------------------------5.4 down auto 1500 access 1 -s-l 9,8,7 0 1 record displayed The contents of the flags field differs based on the options configured on the LAG.
PAGE 262
Deleting a LAG You can delete a LAG at any time by issuing the remove lag command. Whenever you delete a LAG, remember to delete the LAG on the Oracle Fabric Interconnect’s peer Ethernet device (if needed).
PAGE 263
Delete a Single LAG from a 10-Port GE Module To delete all LAGs from a specified 10-Port GE Module, issue the remove lag command plus the LAG name (which is a slot and LAG ID separated by a dot). For example, to remove LAG 5.4: remove lag 5.4 Removing LAGs may disrupt virtual network traffic. Remove LAG 5.4 (y/n)?y Delete a Port from a LAG You can delete individual ports from a LAG at any time, regardless of the port’s state.
PAGE 264
238 XgOS User’s Guide • September 2014
PAGE 265
CHAPTER 14 Clusters This chapter explains cluster configuration in the following sections: ■ “Virtual I/O Fabric” on page 239 ■ “OpenSM Decoupling” on page 244 Virtual I/O Fabric Virtual I/O Fabric enables you to expand the size of your virtual I/O capabilities by interconnecting multiple Oracle Fabric Interconnects together.
PAGE 266
The Oracle Fabric Interconnect also supports a decoupled Subnet Manager (SM), which is part of a cluster environment. OFED 1.1 and 1.2 are supported on external IB attached servers that run SM functions. See “OpenSM Decoupling” on page 244 for more information. Xsigo Directory Service The Xsigo Directory Service Daemon (XDSD) maintains a database of all the reachable chassis and host servers in the cluster. XDSD runs as an instance on each Oracle Fabric Interconnect and is enabled by default.
PAGE 267
XDS Registration Process On initial boot up, the Oracle Fabric Interconnect starts an XDS registration process to determine which chassis becomes the master XDS and which chassis becomes the standby XDS. The Oracle Fabric Interconnect that registers first with SM becomes the master. The registration algorithm is first-come-first-serve.
PAGE 268
A chassis first becomes a standby XDS, then a master. Only a standby XDS can become a master. This approach enables the system to always have backup information, which avoids conditions where SM or a master XDS dies. In these cases, all state information would be lost. Add Server Profiles Regardless of the number of chassis in your network, there is only one designated master and one designated standby.
PAGE 269
In the figure, different virtual resources (vNICs and vHBAs) have been configured on server profiles on two different Oracle Fabric Interconnnect chassis. The flow of operation is as follows: 1. The XDS registers with SM. 2. The Oracle host drivers query SM for XDS location information. 3. The Oracle host drivers query XDS for the cluster (chassis) list. (This list information is used by the host server to install virtual resources accordingly.
PAGE 270
Note – Certain SMs are qualified to work with the Oracle Fabric Interconnect. Contact Oracle customer support for more information. Use the set system is-subnet-manager command to control the OpenSM process running on the chassis. By default, the OpenSM process starts automatically. For more information about OpenSM, see “InfiniBand Ports” on page 19.
PAGE 271
nameservers model-num serial-num ipconfig default-gateway timezone domain-search is-sm 192.168.8.3, 192.168.8.2 F1-15-CH-SDR 050610240 dhcp 192.168.8.
PAGE 272
246 XgOS User’s Guide • September 2014
PAGE 273
CHAPTER 15 User Authorization and Access Control Oracle’s Identity Management System (IMS) service authenticates users and grants them suitable privileges according to assigned user roles when users access the Oracle Fabric Interconnect. The IMS service can be one of: ■ XgOS local system, which is always present ■ Microsoft Active Directory (AD) ■ Remote Authentication Dial In User Service (RADIUS) Once you apply the configuration, the IMS service is completely transparent to the operator.
PAGE 274
Configuring IMS The following figure shows how the Oracle Fabric Interconnect handles a login request when using the internal IMS and when using an external system. If you want to use the internal IMS, all you need to do is add any necessary user accounts as described in “Using the Internal IMS” on page 250. If you would like to use an existing identity management system, the process follows: 1. Set up your external system as described in the appropriate section of this chapter.
PAGE 275
Syntax set ims -cache-timeout=[ default] set ims -maps-to-root= set ims -search-order=[default|externalFirst|internalFirst] set ims -server-type=[default|ldap_ad |local_only|radius] set ims -token-timeout=[][default] set ims {ad-server|radius-server|radius-user} show login system flush ims Parameter Description Parameter Description -cache-timeout The cache time-out value is the number of minutes that XgOS maintains a local copy of authentication and access inform
PAGE 276
Parameter Description -token-timeout Use this option to control the authentication token time-out. Accepted values are from 1 to 1440 minutes. The default value is 5 minutes. Refer to “Using Active Directory as the IMS” on page 254 or ““Using Role Group Mappings for AD/LDAP Users” on page 259 for examples of setting the IMS to a particular type of server. Using the Internal IMS XgOS’s internal IMS, called “local,” comes installed with a predefined administrator (admin) account.
PAGE 277
Role Name Access operators Allows read-only access including all show commands server Allows all operations related to a server’s physical connection, compute-resource configuration, and management storage Allows vHBA configuration and management, Fibre Channel I/O module and ports, LUN masks, persistent mappings, and SAN QoS If you do not specify a role for a user, the user will have the operators role (readonly privileges). ▼ Create a User Account 1. Add a user: add user frank 2.
PAGE 278
5. Test the new user account: quit Connection to 192.168.8.133 closed. $ ssh frank@192.168.8.133 Password: Welcome to XgOS Copyright (c) 2007-2012 Xsigo Systems, Inc. All rights reserved. Enter "help" for information on available commands. pwd /home/frank How Access is Controlled By User Roles User privileges determine administrative abilities. For example, the frank account has operator privileges which grant read-only access to the chassis and its configuration.
PAGE 279
▼ Grant Privileges to a Local User Account 1. Add the user account. For example, the following adds an account named “newuser1”: add user newuser1 ? Possible completions: [Optional qualifiers] -password set password -role role for user 2. Attach the “user” object to a role or give it a password. 3. Display the available roles.
PAGE 280
Using Active Directory as the IMS Microsoft Active Directory (AD) uses either Kerberos or simple (default) user authentication. You can configure up to two AD servers: one primary and one secondary. These two servers act as an active server and a hot standby in case of failures. When using AD as the IMS, use the following process: 1. Set up the necessary user accounts as described in “Configure AD Users and Roles” on page 255. 2.
PAGE 281
▼ Configure AD Users and Roles For AD to work as the IMS for the Oracle Fabric Interconnect, you must create user accounts on the AD server, and on the Oracle Fabric Interconnect, specify the AD server by its IP address, and map the AD group to a role on the Oracle Fabric Interconnect. To create the users on AD server, follow this procedure: 1. Set up the necessary groups for your AD user accounts.
PAGE 282
4. If you need to give everyone under in a particular group administrative access, assign that group to administrators group. Example: Active Directory Server With Default Authentication This example shows how to configure an AD server with simple (default) authentication. The example takes advantage of the default settings for the port (389), the -domain-represented-by option (group), the server mode (primary) and the authentication type (simple). add ims ad-server AD sfcorpdns1.xsigo.com user@xsigo.
PAGE 283
Example: Configure Kerberos as a Secondary AD Server This example configures Kerberos as a secondary AD. This example takes advantage of the default values for port (389), the -domain-represented-by option (group), and the -kdc-port-num option (88). When configuring Kerberos, be aware of the following syntactical considerations: ■ The kerberos-default-realm and kerberos-default-domain must be entered in all capital letters ■ the user-dn can be a simple name format, for example joe_user@xsigo.
PAGE 284
kerberos-default-realm XSIGO.COM kerberos-default-domain xsigo.com kdc-host-name host-name2.xsigo.com kdc-port-num 88 ------------------------------------------------------------------ If the configuration is not correct, the state will be “up/down”. The error field will show the corresponding warning so the administrator will know how to use set ims ad-server to resolve the problem.
PAGE 285
Example: Display All AD Server Configuration show ims ad-server * -detail ----------------------------------------------------------------name AD1 descr host-name ad1.xsigo.com port 389 state up/up error user-dn user@xsigo.
PAGE 286
Note – It is important to understand that regular expressions are used to match the user’s groups against the role group mappings (for example, admin* is not the same as admin.*). Detailed explanation of regular expressions is outside the scope of this document. More information about regular expressions can be found on line.
PAGE 287
Interaction Between Existing Groups and Role Group Mappings In previous versions of the XgOS IMS model, groups were required to be configured with an “xg-” prefix if users of those groups would be accessing the Oracle Fabric Interconnect. The previous model (groups with the “xg-” prefix) are still supported, so you do not need to delete and recreate those existing groups.
PAGE 288
Syntax add ims role-group-mapping [{admininstrators|network|no-access|server|storage|operator}] set ims role-group-mapping [-group=] [-role=] [-descr] show ims role-group-mapping [] remove ims role-group-mapping Parameter Description Role group mapping commands take the following options: Parameter Description Specifies the name of the role group mapping that you are adding.
PAGE 289
When you are configuring a role group mapping, you can do so either of the following ways: ■ Add the role mapping (add ims role-group-mapping) as one step, then use the set ims role-group-mapping command to set the group and role as a second step. This method is used for the examples in the following sections. ■ Add the role mapping (add ims role-group-mapping) and group(s) as one step, then use the set ims role-group-mapping command to set the role as a second step.
PAGE 290
Example: Add Role Group Mapping With a Regular Expression By using a regular expression, you can simplify the creation of a role group mapping if many groups are similar and have the same role. In the following example, a role group mapping will be created for the AD groups tech-marketing, tech-pubs, and tech-supports. A regular expression will be used in the role group mapping to allow matching against all of these groups.
PAGE 291
When deleting the role group mapping the AD groups remain configured on the AD/LDAP server. The remove ims role-group-mapping command deletes the mapping only, and with the mapping gone, no regular expression matching occurs. You can make edits to existing role group mapping through the set ims rolegroup-mapping command instead of deleting and recreating the mapping. Using RADIUS as the IMS RADIUS uses either CHAP or PAP (default) authentication. You can configure up to five servers.
PAGE 292
Example: RADIUS IMS Server With Default Authentication When configuring IMS to use a RADIUS server, you can use the following minimal command.
PAGE 293
Enter a secret: New password: New password again: show ims radius-server RAD2 -detail ----------------------------------------------------------------name RAD2 descr host-name cesar port 200 state up/indeterminate error user-name joe auth-type CHAP timeout 60 retries 6 ----------------------------------------------------------------1 record displayed Configure RADIUS Users and Roles When using RADIUS for IMS, you configure the IMS and then add users through the XgOS command-line interface.
PAGE 294
Example: Set IMS to a RADIUS Server The following shows the command to set IMS to a RADIUS server.
PAGE 295
Common IMS Operations The examples in this section are the same for all IMS services, regardless of their type. Example: Display the IMS Search Order A search-order of “internalFirst” means that XgOS searches the local user database before searching your external IMS.
PAGE 296
The chassis has a local cache to store role information for 240 minutes by default. The next time you log in within that time frame, IMS does not need to query the external IMS server again: show ims -detail ----------------------------------------------------------------cache-timeout 240 ... Configure set ims -cache-timeout=0 to disable the cache. The external IMS will be queried every time someone logs in.
PAGE 297
3. Using vi or any other standard Linux-compliant file editor, display the contents of .bashrc. For example: vi .bashrc # ~/.bashrc: executed by bash(1) for non-login shells.
PAGE 298
272 XgOS User’s Guide • September 2014
PAGE 299
CHAPTER 16 Monitoring XgOS This chapter provides reference and procedural information about monitoring XgOS and its managed objects. It contains the following sections: ■ “SNMP” on page 273 ■ “Monitoring With Xsigo’s SNMP MIBs” on page 276 ■ “Alarms” on page 290 ■ “Xsigo ProWatch Overview” on page 291 ■ “Displaying XgOS System Configuration” on page 301 ■ “Example: Display the Log Level” on page 306 ■ “Tracing End-to-End IB Path Continuity” on page 308 SNMP XgOS supports SNMPv1, v2 and v3.
PAGE 300
Syntax add snmp trap-dest [:] [-community=] [-version=] remove snmp trap-dest [:] [-noconfirm] set snmp -descr= set snmp -read-community= set snmp -sys-contact= set snmp -sys-location= set snmp -sys-name= show snmp Note – If you are entering a description with internal blank spaces (for example Xsigo Systems) you must enclose the string in double quotes. The default read-community string is “public”.
PAGE 301
Replace with the IP address of the system where you are going to receive SNMP traps. Note – Trap IDs are not sequential because they use the virtual resource id (VID). Removed virtual resources will leave gaps in the VID sequence. MIB Support The Xsigo MIB files are available on the Oracle Fabric Interconnect through the standard admin user login. Xsigo MIBs are available in /opt/xsigo/xsigos/mibs. To use the Xsigo MIBs, load all MIBs in this directory. The following MIBs are supported: ■ IF.
PAGE 302
The following table lists statistics available in the ifTable. IF Attribute vNIC ifInDiscards y vHBA ethPort fcPort y ifOutDiscards ifInErrors ibPort y y ifOutErrors y y y y XSIGO-IODIRECTOR-ENTITY-MIB The following tables return valid values for SNMP queries: Object Name MIB Table Object ID (OID) Traps Chassis xsigoIoDirectorChassis 1.3.6.1.4.1.24440.3.1.1.1 Up, Down IO Card xsigoCardTable 1.3.6.1.4.1.24440.3.1.1.2 Inserted, Removed, Up, Down Fan xsigoFanTable 1.3.6.1.4.1.
PAGE 303
Note – If you need to integrate/compile the Xsigo MIBs, you can get them from the Oracle Xsigo Support Portal (http://www.xsigo.com/support).
PAGE 304
With symbolic object name: $ snmpwalk -c public -v2c -mALL 192.168.10.100 xsigoCardType XSIGO-IODIRECTOR-ENTITY-MIB::xsigoCardType.10 = INTEGER: nwEthernet10Port1GbCard(63) XSIGO-IODIRECTOR-ENTITY-MIB::xsigoCardType.12 = INTEGER: sanFc2Port4GbLrCard(71) XSIGO-IODIRECTOR-ENTITY-MIB::xsigoCardType.14 = INTEGER: nwEthernet4Port1GbCard(61) With numeric OID: $ snmpwalk -c public -v2c -mALL 192.168.10.100 -On xsigoCardType .1.3.6.1.4.1.24440.3.1.1.2.1.2.10 = INTEGER: nwEthernet10Port1GbCard(63) .1.3.6.1.4.1.
PAGE 305
XSIGO-IODIRECTOR-ENTITY-MIB::xsigoPortLocation.9 = STRING: 10/9 XSIGO-IODIRECTOR-ENTITY-MIB::xsigoPortLocation.10 = STRING: 10/10 XSIGO-IODIRECTOR-ENTITY-MIB::xsigoPortLocation.11 = STRING: 12/1 XSIGO-IODIRECTOR-ENTITY-MIB::xsigoPortLocation.12 = STRING: 12/2 XSIGO-IODIRECTOR-ENTITY-MIB::xsigoPortLocation.13 = STRING: 14/1 XSIGO-IODIRECTOR-ENTITY-MIB::xsigoPortLocation.14 = STRING: 14/2 XSIGO-IODIRECTOR-ENTITY-MIB::xsigoPortLocation.15 = STRING: 14/3 XSIGO-IODIRECTOR-ENTITY-MIB::xsigoPortLocation.
PAGE 306
With numeric OID: $ snmpwalk -c public -v2c -mALL 192.168.10.100 -On xsigoPortOperState .1.3.6.1.4.1.24440.3.1.1.6.1.7.1 = INTEGER: up(2) .1.3.6.1.4.1.24440.3.1.1.6.1.7.2 = INTEGER: down(3) .1.3.6.1.4.1.24440.3.1.1.6.1.7.3 = INTEGER: up(2) .1.3.6.1.4.1.24440.3.1.1.6.1.7.4 = INTEGER: up(2) .1.3.6.1.4.1.24440.3.1.1.6.1.7.5 = INTEGER: down(3) .1.3.6.1.4.1.24440.3.1.1.6.1.7.6 = INTEGER: down(3) .1.3.6.1.4.1.24440.3.1.1.6.1.7.7 = INTEGER: down(3) .1.3.6.1.4.1.24440.3.1.1.6.1.7.8 = INTEGER: down(3) .1.3.6.1.4.1.
PAGE 307
.1.3.6.1.4.1.24440.3.1.1.3.1.2.6 = STRING: Fan-3/2 .1.3.6.1.4.1.24440.3.1.1.3.1.2.7 = STRING: Fan-4/1 .1.3.6.1.4.1.24440.3.1.1.3.1.2.8 = STRING: Fan-4/2 With symbolic object name: $ snmpwalk -c public -v2c -mALL 192.168.10.100 xsigoFanOperState XSIGO-IODIRECTOR-ENTITY-MIB::xsigoFanOperState.1 = INTEGER: up(2) XSIGO-IODIRECTOR-ENTITY-MIB::xsigoFanOperState.2 = INTEGER: up(2) XSIGO-IODIRECTOR-ENTITY-MIB::xsigoFanOperState.3 = INTEGER: up(2) XSIGO-IODIRECTOR-ENTITY-MIB::xsigoFanOperState.
PAGE 308
XSIGO-IODIRECTOR-ENTITY-MIB::xsigoTemperatureProbeIndex.7 = INTEGER: 7 XSIGO-IODIRECTOR-ENTITY-MIB::xsigoTemperatureProbeIndex.8 = INTEGER: 8 XSIGO-IODIRECTOR-ENTITY-MIB::xsigoTemperatureProbeIndex.9 = INTEGER: 9 XSIGO-IODIRECTOR-ENTITY-MIB::xsigoTemperatureProbeIndex.10 = INTEGER: 10 XSIGO-IODIRECTOR-ENTITY-MIB::xsigoTemperatureProbeIndex.11 = INTEGER: 11 XSIGO-IODIRECTOR-ENTITY-MIB::xsigoTemperatureProbeIndex.12 = INTEGER: 12 XSIGO-IODIRECTOR-ENTITY-MIB::xsigoTemperatureProbeIndex.
PAGE 309
XSIGO-IODIRECTOR-ENTITY-MIB::xsigoTemperatureProbeLocation.7 = STRING: fabricCard XSIGO-IODIRECTOR-ENTITY-MIB::xsigoTemperatureProbeLocation.8 = STRING: fabricCard XSIGO-IODIRECTOR-ENTITY-MIB::xsigoTemperatureProbeLocation.9 = STRING: fabricCard XSIGO-IODIRECTOR-ENTITY-MIB::xsigoTemperatureProbeLocation.10 = STRING: IO Card Slot - 14 XSIGO-IODIRECTOR-ENTITY-MIB::xsigoTemperatureProbeLocation.11 = STRING: IO Card Slot - 14 XSIGO-IODIRECTOR-ENTITY-MIB::xsigoTemperatureProbeLocation.
PAGE 310
With numeric OID: $ snmpwalk -c public -v2c -mALL 192.168.10.100 -On xsigoTemperatureProbeTable .1.3.6.1.4.1.24440.3.1.1.4.1.1.1 = INTEGER: 1 .1.3.6.1.4.1.24440.3.1.1.4.1.1.2 = INTEGER: 2 .1.3.6.1.4.1.24440.3.1.1.4.1.1.3 = INTEGER: 3 .1.3.6.1.4.1.24440.3.1.1.4.1.1.4 = INTEGER: 4 .1.3.6.1.4.1.24440.3.1.1.4.1.1.5 = INTEGER: 5 .1.3.6.1.4.1.24440.3.1.1.4.1.1.6 = INTEGER: 6 .1.3.6.1.4.1.24440.3.1.1.4.1.1.7 = INTEGER: 7 .1.3.6.1.4.1.24440.3.1.1.4.1.1.8 = INTEGER: 8 .1.3.6.1.4.1.24440.3.1.1.4.1.1.9 = INTEGER: 9 .
PAGE 311
.1.3.6.1.4.1.24440.3.1.1.4.1.4.2 = STRING: 14 .1.3.6.1.4.1.24440.3.1.1.4.1.4.3 = STRING: 49 .1.3.6.1.4.1.24440.3.1.1.4.1.4.4 = STRING: 24 .1.3.6.1.4.1.24440.3.1.1.4.1.4.5 = STRING: 24 .1.3.6.1.4.1.24440.3.1.1.4.1.4.6 = STRING: 28 .1.3.6.1.4.1.24440.3.1.1.4.1.4.7 = STRING: 29 .1.3.6.1.4.1.24440.3.1.1.4.1.4.8 = STRING: 30 .1.3.6.1.4.1.24440.3.1.1.4.1.4.9 = STRING: 24 .1.3.6.1.4.1.24440.3.1.1.4.1.4.10 = STRING: 28 .1.3.6.1.4.1.24440.3.1.1.4.1.4.11 = STRING: 37 .1.3.6.1.4.1.24440.3.1.1.4.1.4.12 = STRING: 27 .1.
PAGE 312
With symbolic object name: $ snmpwalk -c public -v2c -mALL 192.168.10.100 xsigoPowerSupplyOperState XSIGO-IODIRECTOR-ENTITY-MIB::xsigoPowerSupplyOperState.1 = INTEGER: failed(4) XSIGO-IODIRECTOR-ENTITY-MIB::xsigoPowerSupplyOperState.2 = INTEGER: up(2) With numeric OID: $ snmpwalk -c public -v2c -mALL 192.168.10.100 -On xsigoPowerSupplyOperState .1.3.6.1.4.1.24440.3.1.1.5.1.7.1 = INTEGER: failed(4) .1.3.6.1.4.1.24440.3.1.1.5.1.7.
PAGE 313
System Controller (SCP) Status You can monitor utilization of the CPU on the System Control Processor (SCP) and utilization of memory on the SCP: ■ SCP CPU usage, with the symbolic object name: $ snmpwalk -c public -v2c -mALL 192.168.10.100 xsigoChassisCpuUsage XSIGO-IODIRECTOR-ENTITY-MIB::xsigoChassisCpuUsage.0 = STRING: 2.50223 ■ SCP CPU usage, with the numeric OID: $ snmpwalk -c public -v2c -mALL 192.168.10.100 -On xsigoChassisCpuUsage .1.3.6.1.4.1.24440.3.1.1.1.15.0 = STRING: 2.
PAGE 314
With numeric OID: $ snmpwalk -c public -v2c 192.168.10.100 -On ifDescr .1.3.6.1.2.1.2.2.1.2.3 = STRING: vnic1.webserver-1 .1.3.6.1.2.1.2.2.1.2.4 = STRING: vnic2.webserver-1 .1.3.6.1.2.1.2.2.1.2.5 = STRING: vhba1.webserver-1 Virtual Interface Status All virtual interfaces should be up. With symbolic object name: $ snmpwalk -c public -v2c -mALL 192.168.10.100 ifOperStatus IF-MIB::ifOperStatus.3 = INTEGER: up(1) IF-MIB::ifOperStatus.4 = INTEGER: up(1) IF-MIB::ifOperStatus.
PAGE 315
With symbolic object name: $ snmpwalk -c public -v2c -mALL 192.168.10.100 ifHCInOctets IF-MIB::ifHCInOctets.3 = Counter64: 11783 IF-MIB::ifHCInOctets.4 = Counter64: 926 IF-MIB::ifHCInOctets.5 = Counter64: 0 With numeric OID: $ snmpwalk -c public -v2c .1.3.6.1.2.1.31.1.1.1.6.3 .1.3.6.1.2.1.31.1.1.1.6.4 .1.3.6.1.2.1.31.1.1.1.6.5 -mALL 192.168.10.100 -On ifHCInOctets = Counter64: 12263 = Counter64: 926 = Counter64: 0 With symbolic object name: $ snmpwalk -c public -v2c IF-MIB::ifHCOutOctets.
PAGE 316
With numeric OID: $ snmpwalk -c public -v2c -mALL 192.168.10.100 -On xsigoServerProfileTable .1.3.6.1.4.1.24440.3.1.2.3.1.1.34 = INTEGER: 34 .1.3.6.1.4.1.24440.3.1.2.3.1.2.34 = STRING: webserver-1 .1.3.6.1.4.1.24440.3.1.2.3.1.3.34 = STRING: .1.3.6.1.4.1.24440.3.1.2.3.1.4.34 = INTEGER: 1 .1.3.6.1.4.1.24440.3.1.2.3.1.5.34 = INTEGER: up(1) .1.3.6.1.4.1.24440.3.1.2.3.1.6.34 = INTEGER: up(2) .1.3.6.1.4.1.24440.3.1.2.3.1.7.
PAGE 317
Xsigo ProWatch Overview The Xsigo ProWatch feature supports periodically transmitting the contents of log files and the output of the show tech-support command, to Oracle Technical Support. This enables Oracle Technical Support to proactively look for and diagnose potential problems without requiring you to collect data, package it, and transmit it Oracle. The information collected is only from show tech-support and system logs. No sensitive customer data is gathered and transmitted to Oracle.
PAGE 318
■ when major or critical alarms occur When you configure periodic transmission of data, the Oracle Fabric Interconnect starts a timer when any of the following events occurs: ■ the Xsigo ProWatch facility is enabled ■ XgOS starts because of a chassis reboot ■ the daily, weekly, or monthly interval is changed. The frequency is calculated from that time at which the timer starts. The result is the next scheduled transmission of data. For example, if the Xsigo ProWatch feature is enabled at 6:00 p.m.
PAGE 319
ProWatch Command Syntax system phone-home [-noconfirm] set system phone-home disable set system phone-home enable set system phone-home noproxy set system phone-home proxy {|default} {[] []} set system phone-home snooze m|h|d|off set system phone-home -contact-email-address=|none set system phone-home -contact-phone-numbers=|none set system phone-home -copy-url=|none set system phone-home -customer-name= set s
PAGE 320
Parameter Description m|h|d|off Sets the timer for the phone snooze period, or disables an in-progress snooze timer. When the snooze timer expires, or is manually disabled with the off option, then the Phone Home schedule resumes based on its configuration.
PAGE 321
Optional Qualifiers Parameter Description -contact-email-address Specifies an email address for a person who Oracle technical support can contact if it appears that the Oracle Fabric Interconnect needs attention. -contact-phone-numbers Specifies one or more comma-separated telephone numbers for Oracle technical support to contact if it appears that the Oracle Fabric Interconnect needs attention.
PAGE 322
port to connect to the Internet. Make sure that the Oracle Fabric Interconnect can connect to: http://phone-home.xsigo.com:6522/ Xsigo ProWatch uses an HTTP POST transfer and data is encrypted while it is on the public Internet. Sending ProWatch Data When Alarms Occur Major alarms include things like an I/O card not coming up or a server failing to connect. All major alarms can negatively affect the data center’s I/O.
PAGE 323
enabled freq next notify strip alarm name email phone copy p-host p-user -----------------------------------------------------------------------------true weekly no yes yes The following section documents how to configure Xsigo ProWatch with your specific parameter values. Before performing the configuration process, you will find it helpful to do the following: ■ Determine if you will want to keep a copy of the information sent.
PAGE 324
4. Specify the email address of a person within your company who Oracle can contact. For example: set system phone-home -contact-email-address=dude1@xyz.com 5. Specify whether Oracle should contact you if potential problems exist. For example: set system phone-home -notify=true 6. (Optional) Configure whether a copy of the Xsigo ProWatch information is stored locally at your company.
PAGE 325
11. When prompted for confirmation, answer “yes” (y). For example: set system phone-home enable This will enable periodic transmission of diagnostic data to Xsigo. The Oracle Fabric Interconnect must have access to the internet for an HTTP connection in order to send the data. Are you sure you want to enable the phone-home facility (y/n)?y 12. Display the Xsigo ProWatch configuration to verify that it is configured correctly.
PAGE 326
Snoozing ProWatch The snooze feature allows any configured phone home to be temporarily delayed for an amount of you determine. The snooze feature is helpful in many ways (for example, if you have an planned maintenance window) because it allows you to temporarily suspend phone home notifications without having to remove the phone home configuration before maintenance, then completely reconfigure it afterward.
PAGE 327
2. (Optional) If you need to deactivate the current snooze interval, issue the set system phone-home off command. set system phone-home snooze off Displaying XgOS System Configuration Use the following commands to display various system attributes.
PAGE 328
Parameter Description Parameter Description watch {ethernet-port|ethernet-ports| fc-port|fc-ports|ioport|ioports|vhba| vhbas|vlans|vnics|vnics} A dynamic window that displays the real-time performance counters of single or multiple instances of: • Ethernet ports. Use the ethernet-port option for the 10 GE module.
PAGE 329
Parameter Description log [debug|syslog] Displays the logs. The show system log and show system log debug commands are interactive, and do not return to a command prompt after they have been issued. You must enter input to manipulate the command output (for example, pause or resume output), or exit the log file and return to the command prompt. If you do not provide any input, the system show log or system show log debug output is displayed indefinitely.
PAGE 330
Parameter Description user Displays internal information about the current user. version [-all] Displays version information for the system, including the minimum driver version required for the purpose of keeping the XgOS and driver software in sync. Or, using -all, shows the version information for all images in the system.
PAGE 331
Example: Display Operating System Details Use show system to display information about either the Oracle Fabric Interconnect F1-15 or Oracle Fabric Interconnect F1-4. The command’s output is the same regardless of the model of Oracle Fabric Interconnect on which it is issued. show system Booted on: Tue May 6 17:15:44 GMT 2008 uptime: 1308 hours, 55 minutes, 44 seconds RECENT UPGRADES AND DOWNGRADES Tue Nov 4 17:15:11 GMT 2011: Upgraded to xsigo-3.6.0.xpf Tue Nov 4 15:09:18 GMT 2011: Upgraded to xsigo-3.5.0.
PAGE 332
Example: Display the Log Level Use the show system loglevel to show the event levels configured for different processes running in the Oracle Fabric Interconnect.
PAGE 333
DiagService ProcessMonitor 42 records displayed iop iop 14 14 WARNING INFO 4 6 Displaying System Statistics The system collects real-time statistics, which are displayed whenever you issue a show stats command. Real-time statistics can be cleared at any time.
PAGE 334
red-pkt 0 ----------------------------------------------------------------------------1 record displayed Tracing End-to-End IB Path Continuity A diagnostic command that supports tracing the IB path from host server, through the IB Fabric, to the I/O module that terminates the IB path on the Oracle Fabric Interconnect.
PAGE 335
■ You can reset statistics to zero to allow them to accumulate by using the optional -clear argument. ■ Statistics are reset if the Fabric Board is reset, for example if the Oracle Fabric Interconnect is power cycled. ▼ Determine the IB Path Continuity As an option, if you have root access to the Oracle Fabric Interconnect, you can log in as root and issue the ibclear errors first, then the ibcheckerrors command to reset all the IB error counters.
PAGE 336
frick hca-19bbfffff847ec hca-2c90200253eb0 jumpstart leven Physical server running VMware/ESX-4.1.0:xg-3.1.0-r5722.ESX4.1/260247 Physical server Physical server Physical server Physical server running Linux/2.6.18-92.el5:xg-2.7.1/x86_64 4. Clear the counters on the server (brick) to get a fresh batch of statistics. show diagnostics ib-path brick -clear 5. Issue the show diagnostics ib-path command for the server (brick) to display the IB path and counters.
PAGE 337
■ the magenta text shows another link in the overall IB path. These entries show how the IB path flows through the various points in the Oracle Fabric Interconnect Fabric Board. Entries are ordered in pairs to show the ingress and egress points in the Fabric Board’s leaf switches/chips. ■ the cyan text shows the I/O module that is the other end of the IB path, which in this example is a Fibre Channel card in slot 3 which is terminating the vHBA on the IB path.
PAGE 338
312 XgOS User’s Guide • September 2014
PAGE 339
CHAPTER 17 System Management This chapter provides reference and procedural information about managing XgOS.
PAGE 340
■ File Transfer Protocol (FTP) ■ Local file TFTP system upgrades are not supported. Before upgrading your operating system, back up your current configuration. For instructions, see “Saving and Restoring Configurations” on page 66. The system upgrade and system import commands feature a timer that flushes out an invalid config, or a config that fails to load. If a configuration cannot be imported, the Oracle Fabric Interconnect starts a five-minute timer window.
PAGE 341
When issuing this command, the file name will autocomplete if you omit the URL scheme or use the file:// option. Parameter Description http:// Upgrade using HTTP. https:// Upgrade using HTTPS. scp:// Upgrade using SCP. file:// For upgrading from a file stored locally on the Oracle Fabric Interconnect. For example from disk, USB (a mounted /usb device), or a /home directory.
PAGE 342
2. Issue the system upgrade command and supply the full path to the new system image. Here is an example of each of the supported upgrade types. system system system system system upgrade upgrade upgrade upgrade upgrade http://cairo.xsigo.com/upgrades/xsigo-V3.8.0.xpf https://cairo.xsigo.com/upgrades/xsigo-V3.8.0.xpf scp://root@cairo.xsigo.com/upgrades/xsigo-V3.8.0.xpf file:///upgrades/xsigo-V3.8.0.xpf ftp://root@cairo.xsigo.com/upgrades/xsigo-V3.8.0.xpf The CLI copies the XPF image to disk.
PAGE 343
Copyright (c) 2007-2012 Xsigo Systems, Inc. All rights reserved. Enter "help" for information on available commands. 3. If you get the following error during the upgrade: Installation failed (Unable to unpack package file xsigo-.xpf where is the system image, then issue the system clear garbage command to remove any partial or failed installs. 4.
PAGE 344
System Configuration Issue the show config command to display the running configuration in table format. There is also an XML version of the configuration file in /config/config.xml. The config.xml file is large and not easy to parse on the Oracle Fabric Interconnect. Use file copy to copy config.xml to some remote location and read the file with an XML reader. Syntax show config printconfig /config/config.
PAGE 345
Example: Print the Configuration printconfig /config/config.xml PAGE 346
Parameter Description Parameter Description broadcast Sends a message to all CLI users who are logged in. cancel {restart|shutdown} Cancels a pending operation. clear {config|garbage|logs} The garbage option removes garbage, such as failed image installs, from the disk. cold-restart [-delay=] [-force][-noconfirm][-now] Restarts the system with the exception of the System Control Processor (SCP), and removes power from the I/O cards.
PAGE 347
Message received from admin at Mon Nov 21 21:51:02 GMT 2011 Broadcast message We should go get some lunch Examples: Initiate Immediate Cold Start ■ To perform an immediate cold restart of the system: system cold-restart Are you sure you want to restart the system (y/n)? y *********************************** Xsigo system is being shut down now *********************************** Connection to iowa closed.
PAGE 348
Network Time Protocol (NTP) Server Syntax set system ntp-server
[-prefer] show system ntp-server remove system ntp-server
[-confirm] Parameters Parameters Description
Specifies the IP address or domain name of the NTP server. For example, 192.168.99.100 or ntp.xyz.com -prefer An optional qualifier that specifies whether the NTP server is the preferred NTP in networks with multiple NTP servers.
PAGE 349
■ Data bits: 8 ■ Stop bits: 1 ■ Parity: none ■ Flow control: none The default username is “admin”. The default password is “admin”. XgOS places you directly into a CLI session with full administrative privileges: login: admin Password: ******** Welcome to XgOS Copyright (c) 2007-2012 Xsigo Systems, Inc. All rights reserved. Enter "help" for information on available commands.
PAGE 350
▼ Change root Password The default root password is root. To change your root password: 1. Log in as the administrator: $ ssh admin@iowa Password: 2. Set the new root password: set Old New New system root-password password: password: password again: Note – As with any system, take care to keep track of your root password. If you forget it, Oracle will not be able to help you by recovering it.
PAGE 351
Use set cli idle-timeout 0 to configure an infinite CLI time-out (no time-out). For information about creating using accounts, see “Create a User Account” on page 251. Syntax show login [] show user Example show login ----------------------------------------------------------------session 1 time 2011-08-20 21:28:20 name admin descr roles administrator interface cli type local logged-in-from 172.16.48.
PAGE 352
■ the minimum number of numbers required for Oracle Fabric Interconnect passwords. ■ the minimum number of special characters required for Oracle Fabric Interconnect passwords. ■ the minimum number of uppercase letters required for Oracle Fabric Interconnect passwords. This feature is supported for local passwords, which are the passwords that allow users to log in to the Oracle Fabric Interconnect.
PAGE 353
Syntax set system password-strength Parameters Parameter Description -min-length Is a number from 0 to 20 that sets the minimum number of characters in the password -min-lower-case Is a number from 0 to 20 that sets the minimum number of lower case letters in the password -min-number Is a number from 0 to 20 that sets the minimum number of numbers in the password -min-special Is a number from 0 to 20 that sets the minimum number of special characters in the password -min-upper-case I
PAGE 354
Setting the Oracle Fabric Interconnect Management IP Address To use the Oracle Fabric Interconnect’s management interface remotely, you must configure an IP address and prefix length for the interface. You initially configure these parameters when you first run the installation wizard at the console port. If you have changes to your management network, you might need to change the IP address you use for the Oracle Fabric Interconnect.
PAGE 355
4. If show login indicates that others are logged on, issue a system broadcast to warn them of the change. For example: system broadcast Changing management IP to 192.168.10.97 in 2 minutes! Note – Changing the system management address can affect management connectivity to the Oracle Fabric Interconnect for other administrators and for management systems. If you are remotely connected, this command will break that connection. Take this into account before making the change. 5.
PAGE 356
The qualifiers for the set system management-interface command are available as individual commands also. For example, the set system address is the equivalent of the set system management-interface -address command. However, by using the set system management-interface command, you have the following distinct advantages: You can see all the relevant parameters in one location, which facilitates completing the management interface’s configuration.
PAGE 357
Restoring Factory Defaults XgOS supports resetting any Oracle Fabric Interconnect to its factory defaults, which is the Oracle Fabric Interconnect’s original state when it was shipped from Oracle. When factory defaults are restored, all configuration information (including the Oracle Fabric Interconnect node name and system management IP address) is cleared. Note – Restoring factory defaults is dangerous. As a result, you should issue it only when you are sure it is safe to lose all configuration.
PAGE 358
Power Down and Power Up Because this command removes all configuration, you will typically only use this command shortly before the Oracle Fabric Interconnect is powered off. You can power off the Oracle Fabric Interconnect by removing both power cables. After you restore factory defaults, the Oracle Fabric Interconnect can be manually restarted by physically restoring facility power (reinserting power cables).
PAGE 359
Example show software ## System status ############################################################# Booted on: Wed Mar 19 21:05:28 GMT 2008 uptime: 48 days, 21 hours, 5 minutes, 55 seconds RECENT UPGRADES AND DOWNGRADES Tue Nov 4 17:15:11 GMT 2008: Upgraded to xsigo-3.6.0.xpf Tue Nov 4 15:09:18 GMT 2008: Upgraded to xsigo-3.5.0.xpf Fri Jul 2 14:48:19 GMT 2008: Upgraded to xsigo-3.0.0.xpf Current Base OS Version Information ReleaseNumber: 176 CompatOS: 71 ReleaseDate: 2008/04/22 18:58:25 KernelVersion: 2.6.
PAGE 360
start_xvnd.sh iop 10 5.89453 00:01:25 0 apache2_prerun.sh scp 0 00:00:00 0 xtctrl scp 0 00:00:00 0 vnctrl scp 0 00:00:00 0 resurrect_db scp 0 00:00:00 0 reap_db scp 0 00:00:00 0 resurrect_sysctl scp 0 00:00:00 0 vnctrl scp 0.582031 00:00:00 0 xsmib_service scp 0.796875 00:00:00 0 xgdiscoverd scp 0.839844 00:00:00 0 xtctrl scp 0.921875 00:00:00 0 xdsd scp 1.14453 00:00:01 0 opensm scp 1.84375 00:00:05 0 postmaster scp 2.85156 00:00:00 0 snmpagent scp 16.8359 00:00:03 0 apache2 scp 22.
PAGE 361
get log-files -noarchives get log-files -nocores get log-files -silent Parameter Description Parameter Description show tech-support > Send content of show tech-support to a file. You must use the redirection switch ( >) to redirect the output to the file. get log-files -all Gather all log, archive, and core files and put them into xsigo-logs.tar.
PAGE 362
Example: Gather All Files With get log-files -all Use the get log-files command to gather all available files, logs, and cores, as well as the output of show tech-support, and put it into a gzipped tar file, which can then be sent off of the Oracle Fabric Interconnect, for example to Oracle Customer Support. get-log-files -all copying /log/cli.log... copying /log/createdb.log... copying /log/daemon.log... copying /log/dumpster.log... copying /log/ib.log... copying /log/install.log... copying /log/kern.log..
PAGE 363
copying /log/user-debug.log.7.gz... copying /log/user.log.8.gz... copying /log/user.log.9.gz... copying /log/wtmp.1.gz... copying /log/coredumps/dmsg_iocard-8_ts67_0... copying /log/coredumps/dmsg_iocard-8_ts68_0... copying /log/coredumps/dmsg_iocard-8_ts74_0... copying /log/coredumps/dmsg_iocard-8_ts86_0... copying /log/coredumps/mimm.1727.core... Warning: cannot open file: /log/coredumps/mimm.1727.core copying /log/coredumps/mimm.5532.core... Warning: cannot open file: /log/coredumps/mimm.5532.
PAGE 364
Considerations When you perform the upgrade, be aware of the following: 338 ■ You can selectively upgrade. You upgrade either the Option ROM, or the firmware, or you can do both at the same time. ■ You must upgrade all HCAs in a particular server that is connected to a Oracle Fabric Interconnect. At present, you cannot selectively upgrade some HCAs in the server, but not others. ■ You can upgrade HCAs in one or all servers.
PAGE 365
■ It is important to understand that the in-band firmware upgrade procedure is not complete until the HCA is rebooted. Until then, the new Option ROM and firmware are resident on the HCA, but not yet active because they have not yet been loaded into memory. ■ The in-band upgrade procedure does not automatically perform the reset, so you must reboot the HCA (or server) to complete the upgrade procedure.
PAGE 366
Install Option ROM and Firmware and Pushing it to Hosts When the Option ROM and firmware images are installed on the Oracle Fabric Interconnect, they are unpacked to the /install directory. Any previous versions of Option ROM or firmware are not automatically deleted. In fact, the Oracle Fabric Interconnect retains a maximum of 64 images. If you need to delete some images, you can do so through a software command.
PAGE 367
Parameter Description The Option ROM and firmware upgrade command have the following options: Parameter Description Specifies the name of the server that contains the HCA that will be upgraded, or allows all servers to be upgraded. Specifies the InfiniBand local Identifier for the HCA that you want to upgrade. Specifies the name of the Option ROM or firmware file that you will be using for the upgrade.
PAGE 368
For example, if you have link state for the HCA but the following error is displayed, you would want to use the reset command to be able to restart the upgrade session on the HCA set physical-server alma upgrade-hca 14 firmware Retrieve version information...
PAGE 369
XgBoot-mt26428-DEBUG.bin 2.8.7 8ac804c34 12 records displayed 77312 2011-08-02 13:10 fd0ab3efe0065b3dbb79a2d Displaying All Firmware Images on the Oracle Fabric Interconnect At any time, you can display a list of all the firmware images installed on the Oracle Fabric Interconnect. The Oracle Fabric Interconnect can retain a maximum of 64 images, but has no restriction on the mix of Option ROM or firmware images that comprise the total.
PAGE 370
Removing the Firmware If needed you can delete an installed Option ROM image from the Oracle Fabric Interconnect by issuing the system remove hca-figure command. system remove hca-firmware Upgrading the Option ROM and Firmware Images You can upgrade the firmware on an HCA by following this procedure, which assumes the server “kingston” will be upgraded. ▼ Upgrade the Option ROM and Firmware Images 1. Issue the show physical-server display HCA information about the server’s HCA.
PAGE 371
3. When you find the correct firmware version, download it to the Oracle Fabric Interconnect. For example: system install hca-image scp://mellanox/downloads/firmware/mt_0a5012xxxx/file1234 When this step completes, the firmware file is installed to the Oracle Fabric Interconnect’s /install directory where a total of 64 Option ROM and firmware images can be kept. 4. Log in to the Oracle Xsigo Support Portal and download the Option ROM tar file.
PAGE 372
11. Upgrade the firmware by issuing the set physical-server command. For example: set physical-server kingston upgrade-hca 7 firmware 3.0.0-fw.bin Retrieve version information... Upgrading HCA 0002c903000a9f7a firmware from 2.9.1000 to 3.0.0 will require a manual server reboot in order to take effect. Do you wish to continue (y/n)?y Upgrading HCA for 0002c903000a9f7a, current version is 2.9.1000... Note – The CLI will temporarily pause while the new firmware is being pushed to the HCA.
PAGE 373
Applying System Patches A system patch is an XPF (Xsigo Package File) that provides a focused fix for a specific file or files in the Oracle Fabric Interconnect file system. A patch tends to be smaller and more manageable, and allows for applying hot-fixes to the Oracle Fabric Interconnect.
PAGE 374
Command Syntax system patch show system patches [-detail] show system version [-all] show system [status] Parameter Description The system patch commands take the following command options: Parameter Description Specifies the name of the patch file that you want to apply. Patches must be applied individually. For example, to use a patch name patch-19408 that is located in the patches directory on a server named cairo you could issue: system patch scp://root@cairo.
PAGE 375
The following procedure assume that you are using the system patch command. To apply a patch, follow this procedure: 1. Get the appropriate patch from Oracle. 2. Issue the system patch command. For example: system patch scp://root@cairo.xsigo.com/patch-19408.xpf The system patch command copies the named patch file, and installs the patch to the appropriate part of the Oracle Fabric Interconnect file system. 3. Issue the system show patches command to verify that the patch was successfully installed.
PAGE 376
b. Run the system unpatch command against the patch. For example: system unpatch patch-19408 3. To upgrade minus the patch: a. When the downgrade is complete, upgrade again to the original version of XgOS (the version that had the patch). For example: system upgrade scp://root@cairo.xsigo.com/upgrades/xsigo-V3.8.0.xpf b. Issue the show system version command to verify that the correct version of XgOS software is in use after the upgrade.
PAGE 377
CHAPTER 18 Scripting XgOS Commands The XgOS CLI scripting engine provides the Aikido scripting language, completed scripts for simplified user commands, and a full text editor for creating your own scripts.
PAGE 378
Aikido Scripting Language All onboard scripts were created using the Aikido Language System. Aikido is an interpreted, dynamically typed language that can be used for general purpose programming but is best suited for prototyping and scripting. It has been derived from the ideas present in a large number of languages including Pascal, Ada, C, C++, Java, JavaScript, and Verilog. See help scripts for more information about the use of OVN scripts. See the following sites for more information on Aikido.
PAGE 379
Example: Move and Rename Files cat /bin/mv #> Rename files /* * (C) 2004,2005 XSIGO SYSTEMS Inc. All rights reserved. This material may not * be reproduced, displayed, modified or distributed without the express prior * written permission of the copyright holder. * * Author: David Allison * Email: dallison@xsigo.com * * $Id$ * $Date$ * $Revision$ * $Author$ * * Description : */ if (args.size() < 2) { throw "usage: mv file... dest" } var allfiles = [] for (var i = 0 ; i < args.
PAGE 380
if (movetodir) { var destname = dest + "/" + Filename.filename (file) System.rename (file, destname) } else { System.rename (file, dest) } } SEDIT Script Editor The Script Editor (SEDIT) is a simple but powerful onboard text editor that runs from within the CLI.
PAGE 381
SEDIT runs as a script named sedit: file edit /bin/sedit See help sedit for documentation: help sedit ▼ Create Your Own Commands Use the Xsigo Script Editor to create your own commands (scripts) and aliases. 1. Use file edit to create and open a file: file edit who The Xsigo Script Editor starts. 2. Define the behavior. 1 show user 3. Save the file and exit the editor: ctrl-w ctrl-d 4. Set the file access permissions and make the file executable: chmod +x who 5.
PAGE 382
356 XgOS User’s Guide • September 2014
PAGE 383
CHAPTER 19 Source RPM: Building OVN Host Drivers Oracle provides source RPM Package Managers (RPMs) for advanced users and developers to help support a wide array of Linux distributions. There are numerous requirements that must be satisfied in order to both compile and produce a compatible driver. The utmost of care should be taken when preparing a driver from the available source, and careful documentation should be kept in order to assist Oracle Customer Support in understanding your environment.
PAGE 384
Overview Oracle distributes two types of host driver RPMs—binary and source. Binary RPMs are compiled for a specific kernel and system architecture. Source RPMs contain the source code for building the binary package. Oracle host drivers are kernel modules. Since it is impossible for Oracle to directly support every version of Linux distribution (kernel and architecture), Oracle provides its hostdrivers as source RPMs.
PAGE 385
headers, kernel symbol-files, kernel config (.config), additional patches, updates, and fixes. In some cases, the OVN host drivers require updates or fixes in your base kernel, dependent drivers, or related tools/compilers. One example of both updated features and fixes is the ib_mthca.ko from pre-OFED-1.2.
PAGE 386
Then install the binary RPM: # # # # rpm –Uvh xsigo-hostdrivers-kmod-2.6.18-53.el5_3.8.0-1.x86_64.rpm chkconfig xsigo on reboot service xsigo status The SPEC File Often, a user will find it necessary to customize some aspect of the driver build process. Many of these behaviors are set through default environment variables, SPEC files at the top of the rpm-SPEC file, or through system scripts. To make these customizations, you should first install the RPM source: # rpm -i xsigo-hostdrivers-kmod-linux_3.8.
PAGE 387
Automatically Checked Default Value Acceptable Values Yes 0 0 or 1 fixup_module_symvers Enable this option if you are building No against an OFED installation which is installed outside the kernel source tree. This option is needed for kernels prior to 2.6.18 which supported finding the Module.symvers file in the top level of kernel source directory first. By default, there is no check done for this so this option must be specified by the user before building the binary RPM.
PAGE 388
Environment Variables When building the drivers, you might need to override some default locations and values. These values are set through environment variables. See the following table. Variable Description kversion This environment variable can be set to specify the kernel version you would like to build the Oracle host drivers for. The default value for this is the kernel you are currently running with (e.g. uname -r).
PAGE 389
■ Dependencies: kernel-devel RPM In this scenario, all of your kernel source and devel-headers/objects should be located inside the path /lib/modules/`uname -r`/build. This symbolic link is the default location for the xsigo-hostdriver src-rpm to look for the kernel source directory. Command sequence procedure: # rpm -ivh xsigo-hostdrivers-kmod-linux_<#version>-1.src.rpm # rpmbuild -bb /usr/src/redhat/SPECS/xsigo-hostdrivers.
PAGE 390
Build Option 3: Kernel With Upgraded OFED Stack Source RPMs are available by request from Oracle Customer Support. If you will be building a source RPM, contact Oracle Customer Support, who will provide instructions for obtaining the image and can provide assistance with using the source. You will need to provide Oracle Customer Support with some basic information For more information, see “Required Information for Contacting Customer Support” on page 368.
PAGE 391
Build Option 4: Combination of Customer Kernel and Upgraded OFED Stack Source RPMs are available by request from Oracle Customer Support. If you will be building a source RPM, contact Oracle Customer Support, who will provide instructions for obtaining the image and can provide assistance with using the source. You will need to provide Oracle Customer Support with some basic information For more information, see “Required Information for Contacting Customer Support” on page 368.
PAGE 392
While Oracle intends their drivers to be installed on a system which leverages the RPM (Redhat Package Manager), it is still possible for advanced users to extract the source code and build each driver manually. When you do this, you should also take care to include the appropriate xsigod userland configuration application and startup scripts. Here is a command sequence to build the 1.5 drivers manually from the src-RPM file: # # # # # # # # # # # rpm2cpio xsigo-hostdrivers-kmod-linux_3.8.0-1.src.
PAGE 393
Example: patch < Note the first < is part of the command and the <> denotes the file name. RDMA Headers To support installing the Oracle Source RPM on RHEL4u4, you might be required to add source RPM RDMA headers. ▼ Add RDMA Headers If you need to add the RDMA headers, you can do so by performing the following steps: 1. Install the matching kernel source RPM. 2. Issue the following command: rpmbuild -bp --target /usr/src/redhat/SPECS/ 3.
PAGE 394
If running against a OFED-1.2.5.X IB stack, the following kernel log message (dmesg) is benign: ib_cm: req timeout_ms 16896 > 8192, decreasing ib_cm: req remote_cm_response_timeout 22 > 21, decreasing ib_cm: req local_cm_response_timeout 22 > 21, decreasing It can be eliminated by setting max_timeout ib_cm module parameter to 23.
PAGE 395
CHAPTER 20 Upgrading XgOS This section provides information of upgrading the XgOS software on the Oracle Fabric Interconnect.
PAGE 396
■ file, for upgrading from a file that has already been downloaded to the Oracle Fabric Interconnect. This option is useful for upgrading the Fabric Interconnect from a local file system, for example from disk, USB (a mounted /usb device), or a /home directory. ■ ftp. Upgrade through FTP prompts you for a password. You can also perform upgrades in confirmation or non-confirmation mode by using the -noconfirm argument in the system upgrade command.
PAGE 397
Note – When importing a backed-up configuration, it is very important that the hardware configuration on the Oracle Fabric Interconnect match the hardware configuration when the XML file was exported. If the two do not match, the import will abort. Exporting your configuration is not required before upgrading. If you use the system export command, the file is exported to XML format (the default format) to the /home directory of whichever user account you used to log in.
PAGE 398
Basic Upgrade Process If you are upgrading from XgOS 3.5.0 to XgOS 3.6.0 or later, you do not need to keep the Oracle Fabric Interconnectr software and host driver software in exact synchronicity. In fact, XgOS software and host driver software can be upgraded independent of each other. However, you should keep the XgOS software and the host driver software reasonably in sync. For example, running 3.6.0 XgOS with 3.5.
PAGE 399
▼ Upgrade OS on the Oracle Fabric Interconnect 1. Log in to the Oracle Fabric Interconnect as admin. 2. (Optional) If any other administrators are likely to be connected to your Oracle Fabric Interconnect, issue a system broadcast to warn them to disconnect: system broadcast Upgrading the Fabric Interconnect in 2 minutes! 3. Save your current configuration: system export xgos-3.9.0.xml Where xgos-3.9.0.xml is the existing configuration file. 4.
PAGE 400
■ Using FTP, type: system upgrade ftp://root@cairo.xsigo.com/upgrades/xgos-3.9.2.xpf If you upgrade using FTP, you will be prompted for a password. Note – If you get the this error message during the upgrade, Installation failed (Unable to unpack package file xsigo-3.9.2.xpf,issue the system clear garbage command to remove any partial or failed installs. When you issue the system upgrade command, the Fabric Interconnect begins the upgrade.
PAGE 401
5. Wait for the I/O cards in the Fabric Interconnect to finish initializing before proceeding. To determine whether the I/O cards are initialized, use the show iocard command. The Fabric Interconnect displays the state of all cards. When the state of the cards is up/up, you are ready to proceed. 6. When the I/O cards all show up/up, check the state of the vNICs and vHBAs: show vnic * show vhba * The state of the vNICs and vHBAs should be up/up. 7.
PAGE 402
While the first Fabric Interconnect was unavailable, host servers that were using it failed over to use the other Fabric Interconnect for their I/O. Assuming that you do not have automatic switchover configured, they will remain with that Fabric Interconnect until they reboot after receiving their host driver upgrade. If this is your first Fabric Interconnect, the next task is to upgrade the host drivers, HCA firmware, and the option ROM for each host server.
PAGE 403
■ ConnectX and ConnectX-2: 2.7.0 and later It requires option ROM 2.8.7 to support SAN boot. If your firmware and XgBoot versions are as shown above, you can skip Step 2. 2. On your Linux host server, upgrade the HCA firmware and the option ROM if necessary. a. If you haven’t already done so, log in as root to the host server. b. Upgrade the Xsigo HCA firmware package on the server. rpm -Uvh Note – Replace xsigo-hca-firmware_2.8.7.i386.
PAGE 404
Board ID CA type Firmware version Hardware version Option ROM version : : : : : 'MT_0150000001' 'MT25208' '5.3.0' 'a0' 'XgBoot Version 2.8.7' 1) 5.2.0 (XgBoot Version 2.8.7) 2) 5.1.400 (XgBoot Version 2.8.7) 0) Return to previous menu Select firmware to use> e. Select the most recent firmware (the one displayed first). You will need to reboot for the firmware upgrade to take effect. However, you can wait to reboot until you have upgraded the host drivers. XgOS version 3.9.0 requires support for OFED 1.
PAGE 405
7. Reboot the server. This completes the Linux host server upgrade. If you have more servers to upgrade, complete that task now. If you have upgraded all servers, proceed to “Upgrade OS on the Oracle Fabric Interconnect” on page 20-373 to upgrade your second Fabric Director’s operating system.
PAGE 406
Caution – Make sure to select the compatible pair of drivers. If you attempt to install an incompatible Xsigo driver after installing the Infiniband driver, you will receive an error message ▼ Upgrade VMware Host Servers 1. Copy the host driver package to your system. The Xsigo host driver package contains both the InfiniBand stack and the Xsigo host drivers. For example, the following command illustrates copying the package using scp: scp user@host:/mydrive 2.
PAGE 407
Upgrading Windows Host Servers When using vNICs in an HA configuration, system .DLL files are in use. This prevents the upgrade process from accessing the .DLL files needed for upgrade. To upgrade the Windows host drivers from 3.5.0 to 3.6.0 using HA vNICs, it is best if you first uninstall the previous version of host drivers, restart your Windows host, and then proceed with a clean installation of the Windows host drivers.
PAGE 408
Note – The space following the equals sign (=) is mandatory. This command causes the service to delay its start until LUNs are available across the vHBA. For the details of the XgDependRoot service, see “Xsigo Dependency Service” on page 384. ▼ Upgrading Windows HCA Firmware and Option ROM XgOS supports the following firmware levels: ■ InfiniHost Single Port HCA: 1.2.0 ■ InfiniHost Dual Port HCA: 5.3.0 ■ ConnectX and ConnectX-2: 2.7.0 and later When the Xg_FWUpdate.
PAGE 409
GUID Descr : node port1 port2 sys image GUIDs : 0002c9020021f1f0 0002c9020021f1f1 0002c9020021f1f2 0002c9020021 f1f3 BOARD ID : mt_0370110001 VSD : PSID : mt_0370110001 FW Version : HCA mlx FW Ver : 5.1.400 1) Flash HCA Firmware 2) Change selected card 0) Quit Select option> 4. When prompted, enter 1 to enter the Flash HCA Firmware Menu.
PAGE 410
The following example shows updating the HCA with firmware version 5.2.0. Select Firmware to Burn> 1 Upgrading HCA firmware 5.1.400 to 5.2.0 This Will Flash HCA with Firmware file .\Image\fw-25218-5_2_0mhea28-xtc_a1-a2.bin Please do not interrupt the burn process or reboot the machine... Wait till burn completes ... ................................................................. ----------------------------------------------------------------The firmware on one or more of the HCAs has been upgraded.
PAGE 411
For example, for lanmanserver (the server service): sc config lanmanserver depend= xgdependroot Note – The space following the equals sign (=) is required. To set the dependency of another service, substitute its name for lanmanserver in the example above. For example, use iisadmin if the web site or FTP site is residing on a vHBA LUN. After you have installed the drivers and set the dependency, reboot the server.
PAGE 412
VM VM VM VM ESX Server 1 with Xsigo Host Drivers ESX Server 2 with Xsigo Host Drivers Fabric Interconnect 1 with XgOS 3.5.0 Fabric Interconnect 2 with XgOS 3.5.0 Ethernet You must upgrade one ESX Server and one at a time. After they reboot running the new software version, you will use VMotion to move all guest machines to the upgraded server and upgrade the second ESX Server and Fabric Interconnect. In the procedures that follow, ESX Server 1 and Fabric Interconnect 1 are upgraded first.
PAGE 413
Compatible Software Versions You will typically need different host drivers depending on your version of ESX Server software and the HCAs in your servers. When upgrading a VMware host server, be aware of the particular versions of ESX server software and Xsigo host drivers required. Host drivers for each supported operating system are listed in the Release Notes. ▼ Upgrade First ESX Sever and Fabric Interconnect Use these steps to upgrade the ESX Server 1 host drivers and XgOS on Fabric Interconnect 1: 1.
PAGE 414
4. Save your current I/O configuration: system export xgos-3.9.0.xml Where xgos-3.9.0.xml is the existing configuration file. 5. Disable the server profiles to both ESX Server hosts. set server-profile esx1 down set server-profile esx2 down Where esx1 and esx2 are the server profiles. In the next several steps, you will upgrade the host drivers on ESX Server 1. Note – When upgrading a VMware server, be aware of the particular versions of ESX server software, HCA firmware, and Xsigo host drivers required.
PAGE 415
9. Install the new Xsigo drivers package: esxupdate update --bundle 10. Confirm that the updated packages have been installed: esxupdate query ---Bulletin ID---------- Installed----- --------------Summary----ESX410-Update01 2011-07-14T12:53:00 VMware ESX 4.1 Complete Update XSIGO-4.1.0.260247.3.9.2-1 2011-07-19T09:31:18 Xsigo Systems Virtual Driver + IB Bundle Note – Do not reboot the host server at this time.
PAGE 416
When you issue the system upgrade command, the Fabric Interconnect begins the upgrade. While the upgrade occurs, status messages are displayed, as shown in the following example: Copying...############################################################# [100%] You have begun to upgrade the system software. Please be aware that this will cause an I/O service interruption and the system may be rebooted. The following software will be installed 1. XgOS Operating System software including SCP Base OS 2.
PAGE 417
6 up/up 10 up/up 12 up/up 14 up/up 15 up/up 6 records displayed nwEthernet10Port1GbCard sanFc2Port8GbCard nwEthernet1Port10GbCard nwEthernet1Port10GbCard nwEthernet10Port1GbCard 0 2 2 0 0 16. When the I/O cards all show up/up, check the state of the vNICs and vHBAs: show vnic * show vhba * The state of the vNICs and vHBAs should be up/up. The Fabric Interconnect uses a hold time which pauses the vNICs and vHBAs for a window of time while the I/O cards come back online.
PAGE 418
b. For vHBAs, type: esxcfg-mpath -l fc.5001397100002505:5001397000002505fc.2001000b08002539:2400000b08043110-eui.00 0b080074002539 Runtime Name: p3:C0:T0:L2 Device: eui.000b080074002539 Device Display Name:Pillar Fibre Channel Disk (eui.000b080074002539) Adapter: p3 Channel: 0 Target: 0 LUN: 2 Adapter Identifier: fc.5001397100002505:5001397000002505 Target Identifier: fc.2001000b08002539:2400000b08043110 Plugin: NMP State: active : : When the State field shows active, the vHBA is up and connected. 19.
PAGE 419
When you have finished, your system configuration resembles that shown in the figure. VM VM VM VM ESX Server 1 with Xsigo Host Drivers ESX Server 2 with Xsigo Host Drivers Fabric Interconnect 1 with XgOS 3.6.0 Fabric Interconnect 2 with XgOS 3.5.0 Ethernet 3. Log in to Fabric Interconnect 2 as admin. 4. Save your current I/O configuration: system export xgos-3.9.0.xml Where xgos-3.9.0.xml is the existing configuration file. 5.
PAGE 420
6. Use VMotion to restore your initial configuration. Your system configuration should now resemble that shown in the figure. VM VM VM ESX Server 1 with Xsigo Host Drivers ESX Server 2 with Xsigo Host Drivers Fabric Interconnect 1 with XgOS 3.6.0 Fabric Interconnect 2 with XgOS 3.6.
PAGE 421
Glossary A Active Directory Admin State Active Directory (AD) is an implementation of LDAP directory services by Microsoft for use primarily in Windows environments. Its main purpose is to provide central authentication and authorization services for Windows based computers. Active Directory also allows administrators to assign policies, deploy software, and apply critical updates to an organization. Administrative state. The intention of the operator by setting a given resource up or down.
PAGE 422
upper-level protocols (FC-4). Each of these levels defines a different and separate part of how the FC equipment communicates. The different FC-4 protocols (FCP, IP, Virtual Interface, and others) are tied directly to different kinds of applications (storage, networking, and clustering) for different uses. For more background information, see www.fibrechannel.org.
PAGE 423
I/O Module I/O Port A physical card that is installed in one of 15 slots in the chassis’ card bay. There are two types of I/O module: Ethernet and Host Bus Adapter. The Ethernet and Host Bus Adapter modules provide access to Ethernet and Fibre Channel networks, respectively. A single port on an Ethernet module, a Host Bus Adapter module, or one of the 24 InfiniBand server ports. J JBOD Just A Bunch of Disks. Very large storage arrays, capable of storing terabytes and terabytes of data.
PAGE 424
M Managed Object An object-oriented representation of a resource managed in a device. This can be a physical or logical resource. N NAS NPIV Network Attached Storage. NAS uses common client networks, such as Ethernet, to connect client computers to a host file server. Unlike SANs, the client does not directly communicate with the storage. Data exchange occurs at the file level, unlike a SAN where data is operated at the block level over FC.
PAGE 425
Q Quality of Service The Quality of Service (QoS) object allows the data traffic of individual applications or interfaces to be managed. The performance of a particular application can be guaranteed by raising the priority of its dataflow, relative to the other applications. R RADIUS RAID RDMA Remote Authentication Dial In User Service (RADIUS) is an Authentication, Authorization, and Accounting (AAA) protocol for controlling access to network resources.
PAGE 426
SCSI Server Profile Small Computer Systems Interface. In the early 1980s, SCSI was the standard direct-attach storage interface to SCSI-enabled disks. As computer systems increased in speed and data storage needs increased, the parallel bus architecture of SCSI began hitting performance and distance limits. In response to this need, FC was introduced to provide gigabit-speed serial networking capabilities for storage.
PAGE 427
VM Virtual Machine. A VM is a software entity that runs its own operating systems and applications, as if it were a physical computer. A VM behaves exactly like a physical computer and contains its own virtual (software based) CPU, RAM, hard disk, and NIC. An operating system installed on a VM is called a guest operating system. vNIC Virtual Network Interface Card - An Ethernet interface, provided without a physical NIC.
PAGE 428
402 XgOS User’s Guide • September 2014
PAGE 429
Index Symbols * (wildcard), 6 Numerics 10GE VLAN examples, 126 10-Port GE VLAN examples, 126 1-port 10GbE I/O modules, 27 A AAA, 247 access mode, 120 access VLAN, 121, 123 accounts, 250 ACLs, 213 with QoS, 202 action, 214 Active Directory, 254 AD, 254 AD/LDAP groups, mapping, 259 add acl, 221 add gateway, 73 add ims, 254 add lag, 229 add qos network, 198 add qos san, 210 add san map, 142 add server-profile, 70, 136 add snmp, 274 add snmp trap-dest, 274 add user, 251 add vhba, 136 add vlan, 126 add vnic,
PAGE 430
collecting debug information, 49 command completion, 3 command history, 5 command-line interface, 1, 369 command completion, 3 configuring the CLI, 53 controlling echo, 55 controlling line wrapping, 57 controlling terminal screen rows, 61 display filters, 58 displaying history, 61 displaying mode, 56 editing commands, 5 history, 5 line wrap, 57 online help, 4 redirecting output, 5 top-level commands, 9 commit, 221 committed burst size, 192 committed information rate, 192 compressing files, 50 config.
PAGE 431
ESX Server configuration, 182 persistent mapping, 143, 146 prescan vHBA, 148 removing prescan information, 150 rescanning, 150 resetting I/O modules, 27 saving and restoring Fabric Interconnect configuration, 67 searching in text files, 50 setting CLI terminal screen row, 61 setting MTU, 95 setting vNICs down, 90 shutting down I/O modules, 27 sorting CLI output by column, 59 vHBA basic configuration, 137 VLAN 10GE and 10-port GE, 126 F Fabric Interconnect backing up configuration, 67 Fabric Manager plugin
PAGE 432
displaying search order, 269 using internal, 250 using local, 250 predefined user roles, 250 using RADIUS, 265 in-band HCA upgrade, 337 InfiniBand, 19, 134 displaying ports, 20 ports, 19 ingress-qos, 198 initiator, 134 interfaces, 44 internal IMS, 250 IOCPort16, 19 IP address, 328 ip-addr, 78 isolation, network, 118 isolation, traffic, 118 J link aggregation, dynamic LAG, 226 link aggregation, static LAG, 226 load balancing with ESX Server, 188 local ID, 179 local IMS, 250 local-id, 137, 181, 188 setting
PAGE 433
see link aggregation groups NIC teaming see link aggregation groups NIC teaming with ESX Server, 188 no-confirm, 137 no-lun-masking, 137, 174 NPIV, 135, 398 NTP, 319 ntp-server, 322 O OFED, 358 patch files, 366 online help, 4 OpenSM decoupling, 244 Option ROM, upgrading, 337 P packet coalescing, 114 PAP, 265 parity, 323 passive-mode LACP, 226 password, setting strength, 325 patches, applying, 347 PBS, 192 peak burst size, 192 peak information rate, 192 persistent binding, 142 phone-home, 291 physical serv
PAGE 434
root fs logout timer, setting, 270 root login, 323 root password, 324 rpmbuild, 359 RSCN, 147 rule modifier, 214, 216 rule parameter, 214 S SAN, 134 SAN maps adding, 143 removing, 146 SAN QoS, 209 scp, 47 scripts, 351 editor, 354 searching in files, 50 server profiles, 69 command introduction, 10 configure and add vNIC (example), 79 creating (example), 71 removing, 146 resetting the busy state, 71 set acl, 204, 221 rank, 214, 216 rule, 214 set cli cols, 60 set cli idle-timeout, 325 set cli rows, 60 set cli
PAGE 435
show, command introduction, 14 slots, numbering, 17 SM, 20 SNMP, 273 add snmp trap-dest, 274 SNMP MIBs, monitoring, 276 source RPM, 357 SPEC file, 360 SSH login, 323 static, 78, 126 static LAG, 226 static LAG, configuring, 232 statistics real-time, 307 vHBAs, 167 vNICs, 80 stop bit, 323 strength of password, 325 subnet manager, 20, 22 subnets, 72 system broadcast, 320 system cancel, 319 system clear, 319 system clear config, 317 system cold-restart, 317, 319 system downgrade, 319 system export, 66, 314 syst
PAGE 436
virtual I/O fabric, 239 virtual I/O resources naming restrictions, 14 virtual LANs, 117 virtual Network Interface Card, 77 VLAN configuration, 119 10GE VLAN examples, 126 10-Port GE VLAN examples, 126 chassis managed, 124 host managed, 124 VLAN port mode, tag native.
Who We Are
About Us
Company
Careers
Terms & Privacy
Resources
List of Manufacturers
Support
For the Press
Media assets
FAQ
Get in touch
Blog
Email
DMCA
ManualShelf © 2013-2025