TM Sun Fire X4600/X4600 M2 Server Architecture A Technical White Paper May 2007 SunWIN Token # 470898
Sun Microsystems, Inc. © 2006 Sun Microsystems, Inc., 4150 Network Circle, Santa Clara, CA 95054 USA All rights reserved. This product or document is protected by copyright and distributed under licenses restricting its use, copying, distribution, and decompilation. No part of this product or document may be reproduced in any form by any means without prior written authorization of Sun and its licensors, if any.
Sun Microsyste Table of Contents Introduction.................................................................................................................. ..........................5 Sun Fire x4000 Server Family Overview............................................................................................................5 Sun Fire X4600 Overview...................................................................................................................................
Sun Microsystems, Inc. Key Productivity Features............................................................................................................................46 Interoperability.............................................................................................................................................47 Availability...................................................................................................................................................
Sun Microsyste Introduction As the first of its kind, the Sun Fire X4600 server is the fastest, most scalable, and energy-efficient four-way to 16-way x64 (x86 64-bit) server. It scales quickly from two sockets to eight sockets simply by adding processor boards, all in a compact 4RU form factor. The new modular processor board for the Sun Fire X4600 M2 server allows up to ¼ terabyte of memory in a fully configured system.
Sun Microsystems, Inc. Figure Intro-1: Sun Fire X4000 Server Family Customers seeking a business edge can look to Sun to provide smart options for building a simple, secure, and standardized IT infrastructure. The Sun Fire x64 family of servers enables standardization of the datacenter by providing the unity of vertical scaling with the economies of horizontal scaling, while running any major OS and 32-bit/64-bit application.
Sun Microsyste Figure Intro-2: The Sun Fire X4600 Server The highly scalable 4- to 16-way Sun Fire X4600 server allows customers to dramatically improve business agility and growth without any data center disruption and no change to the operating system, software, rack, network infrastructure, or storage connections. The Sun Fire X4600 server can be used to build 5-teraflop (or larger) systems. Table Intro-1 shows how the new Sun Fire X4600 server compares to the Sun Fire X4100 and X4200 servers.
Sun Microsystems, Inc. Features Sun Fire X4100/X4200 Sun Fire X4600 M2 HDD Number Four HDDs Optical Drive CD/DVD-ROM NIC Controller Dual Intel FW82546GB NICs Quad Gigabit Ethernet ports Graphics Controller ATI Rage XL Expansion Slots Six PCI-E Low Profile and Two PCI-X MD2 Front I/O Two USB 2.0 ports (Type A) Rear I/O Four Gigabit Ethernet ports (RJ45) One 100BaseT Management port (RJ45) One RS-232 Serial (RJ45) Two USB 2.
Sun Microsyste Sun Fire X4600 Target Markets and Applications The Sun Fire X4600 server provides mission-critical enterprise data center solutions across all industries.
Sun Microsystems, Inc. Chapter 1 Architecture Overview Sun Fire X4600 Server The Sun Fire X4600 server measures 4 rack-units (6.9 inches/176 mm) high, 17.5 inches (445 mm) wide, and 24.75 inches (629 mm) deep. It is a 4- to 16-way rack-optimized, x64-based system with a symmetric multiprocessor. Cooling is accomplished with internal fans that direct airflow from the front to the back of the chassis. I/O ports are located on the rear panel, with additional USB on the front panel.
Sun Microsyste Figure 1-2: Rear View of the Sun Fire X4600 Server 11 of 63
Sun Microsystems, Inc. Figure 1-3: Internal View of the Sun Fire X4600 Server The Sun Fire X4600/X4600 M2 server also provides the following system architecture features: • Sun Fire X4600: Embedded dual-channel DDR1 memory controllers on each CPU to provide maximum memory capacity and bandwidth scaling and deliver up to 64 GB of capacity and 51.2 GB/sec. of aggregated bandwidth with eight populated CPUs and 2 GB DIMMs.
Sun Microsyste • Embedded management and legacy I/O to offer maximum operational flexibility without compromising PCI slots for optional features Figure 1-4: Sun Fire X4600 Server Architecture Block Diagram with Eight CPUs 13 of 63
Sun Microsystems, Inc. Chapter 2 CPU Architecture The Sun Fire X4600 server supports up to eight Single-Core or Dual-Core AMD Opteron Series 800 processors. Each CPU module has an attached PCB containing: • 1 AMD Opteron CPU and heatsink • 4 DDR1 memory DIMM slots • 1 CPU/RAM VRM • Airflow redirection baffles for greatest cooling efficiency The Sun Fire X4600 M2 server supports up to eight AMD Opteron Series 8000 (Dual-Core and Multi-Core in the future) processors.
Sun Microsyste Figure 2-1: Sun Fire X4600 CPU Module Systems configured with four or six processors require filler modules. The filler modules maintain the HyperTransport connectivity between all CPUs and provide airflow redirection for efficient cooling throughout the system. Figure 2-2 illustrates the placement of CPU and filler modules, physical slot numbering, and the CPU enumeration from the BIOS perspective.
Sun Microsystems, Inc. Front of Server Rear of Server Figure 2-2: CPU Module Placement Note: The Sun Fire X4600 server supports four and eight CPU configurations. Two, four, six, and eight CPU configurations are supported in the Sun Fire X4600 M2. In Sun Fire X4600 systems, each processor contains an integrated memory controller that supports a 128-bitwide path to memory and three 16x16-bit HyperTransport links.
Sun Microsyste In Sun Fire X4600 M2 systems, each processor contains an integrated memory controller that supports a 128bit-wide path to memory and three 16x16-bit HyperTransport links. Each HyperTransport link runs at speeds up to 1 GHz and is clocked on both edges of the clock pulse, which allows for a maximum of 4 GB/sec. (2 gigatransfers/sec., 2 bytes wide) throughput in each direction (8 GB/sec. aggregate data rate bi-directionally). Memory support is for registered DDR2/667 240-pin SDRAM DIMMs.
Sun Microsystems, Inc. Figure 2-4: Single-Core AMD Opteron Processor Architecture The Sun Fire X4600 server supports both Single-Core and Dual-Core AMD Opteron 800 Series processors. The Sun Fire X4600 M2 server supports Dual-Core AMD Opteron 8000 Series processors. From the start, AMD64 processors were designed to add a second core—the port already existed on the crossbar/SRI. Dualcore AMD Opteron processors have unique instances of L1 cache (I-cache and D-cache) and L2 cache for each CPU core.
Sun Microsyste Figure 2-5: Dual-Core AMD Opteron Processor Architecture Note: Sun Fire X4600 servers support 95 W and 120 W 90 nm technology package infrastructure AMD Opteron Rev E and Rev F processors only. CPU Packaging AMD Opteron 800 series processor packaging specifications include: • 940-pin lidded organic micro PGA • 1.27 mm pin pitch • 31 x 31 row pin array • 40 mm x 40 mm organic substrate • Organic C4 die attachment • 16.5 mm x 11.
Sun Microsystems, Inc. Processor VRM Each processor receives core voltage input from an individual voltage regulator module (VRM). The VRMs regulate the system current and voltage according to the VID code output by the CPU package. The VRM is a DC-to-DC point-of-load convertor specified for +12 Volt input and programmable 100 Amp output. The VRM is different on the new 8-DIMM Processor Boards to accommodate the additional memory requirements.
Sun Microsyste populated in pairs in memory slots from the top of the board toward the processor heat sink at bottom of the board. Memory slots should be populated with D7 and D6 first, then D5 and D4, then D3 and D2, and finally D1 and D0 next to the processor heat sink. LED fault indicators controlled by the ILOM Service Processor provide the ability to easily identify failed DIMM modules.
Sun Microsystems, Inc. • • • • 512 MB to 2 GB low profile (1.2 inch/3.05 cm) DIMM support Up to 32 GB per CPU in with 4 GB DIMMs in 8 DIMM slots per CPU Single supply (2.50 VDC) Standard SPD (VCC-SPD=3.3V) x64 Architecture The x64 architecture is an x86-compatible architecture that enables simultaneous 32-and 64-bit computing. It enables end users to run their existing installed base of 32-bit applications and operating systems at peak performance, while providing a migration path that is 64-bit capable.
Sun Microsyste • • Provide digital content creation capabilities such as computer-aided design (CAD), computer-aided manufacturing (CAM), and computer-aided engineering (CAE), digital music production and video editing, and real-time media streaming solutions Require maximum performance for realistic and cinematic consumer experiences, including computer games, digital video, and real-time collaboration x64 processors are designed to maintain full compatibility with x86 while providing the architectural
Sun Microsystems, Inc. to 8 GB/sec. aggregate bandwidth per link, enabling a peak bandwidth of 24 GB/sec. per processor. The AMD Opteron processor with HyperTransport Technology provides a scalable direct connection between processors, I/O subsystems, and other chipsets. HyperTransport technology helps reduce the number of buses in a system, which can reduce system bottlenecks and enable today's faster microprocessors to use system memory more efficiently in high-end multiprocessor systems.
Sun Microsyste Figure 2-7: Intel vs. Opteron Architectures I/O Expansion Capability to High-Speed Industry Buses The traditional Northbridge/Southbridge architecture is not intended to support more than two core-logic elements. Adding additional high-speed functionality (such as Gigabit Ethernet, PCI-X, the InfiniBand Architecture, or any combination thereof) would have to occur in one of three ways: • The functionality would have to be attached to an existing bus interface such as the PCI bus.
Sun Microsystems, Inc.
Sun Microsyste Chapter 3 I/O Subsystem Sun Fire X4600 I/O Subsystem Overview The Sun Fire X4600 I/O subsystem is connected to the CPU complex through a HyperTransport link from processors A and H.
Sun Microsystems, Inc. • • • AMD-8132 HyperTransport PCI-X Tunnel NVIDIA nForce4 Professional 2200 NVIDIA nForce4 Professional 2050 TM AMD-8132 HyperTransport PCI-X Tunnel TM The AMD-8132 HyperTransport PCI-X 2.0 tunnel is a high-speed device that provides two independent, highperformance PCI-X bus bridges, integrated with a high-speed HyperTransport technology tunnel. Designed for operation in modes 1 and 2, the AMD-8132 component brings high-performance PCI-X 2.
Sun Microsyste NVIDIA nForce Professional The Sun Fire X4600 also uses the following NVIDIA nForce Professional chipsets: • NVIDIA nForce 2200 • NVIDIA nForce 2050 Both the NVIDIA nForce Professional 2200 and NVIDIA nForce Professional 2050 MCPs (media and communications processors) provide I/O connectivity to a CPU via a 1-GHz, 8-GB/sec. HyperTransport connection. The NVIDIA nForce Professional 2200 attaches to the AMD 8132 PCI-X tunnel, while the NVIDIA nForce Professional 2050 attaches to CPU H.
Sun Microsystems, Inc. Figure 3-4: LSI SAS1064 Architecture Block Diagram Although the LSI SAS1064 controller is capable of supporting both SAS and SATA drive types, the Sun Fire X4600 server will only support the SAS hard disk drives, which provide higher performance and better reliability. Currently, high-density, 10,000-RPM, 73-GB hard disks are supported.
Sun Microsyste Port Gigabit Ethernet controller include: • 802.3ab PHY compliance and compatibility (CAT-5 use) • 802.3ab auto-negotiation • 802.3x full-duplex flow control • 802.
Sun Microsystems, Inc. • 1024x768 TMDS Figure 3-7: ATI Rage XL Architecture Block Diagram SMSC LPC47B272 Super I/O Controller The Super I/O device offers multiple I/O functions required to provide legacy I/O devices to the operating environment.
Sun Microsyste PCI-X and PCI-E Expansion Slots The PCI slots on the Sun Fire X4600 server use a 3.3 Volt signaling bus. Therefore, they support only 3.3 Volt keyed expansion cards. The use of 5 Volt keyed expansion cards is unsupported and will result in damage to the motherboard. The PCI-X slots accomodate expansion cards which adhere to the MD2 form factor standard. MD2 PCI-X cards are a maximum length of 169.3 mm (6.7 inches) and a maximun width of 51.5 mm (2.5 inches).
Sun Microsystems, Inc. Figure 3-9: PCI Expansion Slot Locations The two PCI-X slots have a maximum operational speed of 100 MHz. It is important to note that if an expansion card that operates at speeds less than 100 MHz is inserted into these slots, then the transfer speed along the entire bus will be degraded, affecting embedded disk performance. Since the shared bus speed will be lowered to the lowest common denominator, a 33 MHz expansion card will degrade performance more than a 66 MHz expansion card.
Sun Microsyste Chapter 4 Motherboard All control and datapath functionality, with the exception of the disk connectors and CPU module boards, reside on the motherboard. There are sockets for connecting each of the 8 CPUs that are interconnected through AMD's HyperTransport technology. There are also HyperTransport links to the PCI-X bridge and the Southbridge chipsets. All external connections, with the exception of power, disks, and front panel I/O, come into the motherboard.
Sun Microsystems, Inc.
Sun Microsyste Chapter 5 Cooling and Power Typical heat dissipation and power consumption metrics for a Sun Fire X4600 server equipped with eight Dual-core AMD Opteron processors and 32 GB RAM are as follows: • UL Maximum Power: 1975 W • Typical Power Consumption: 1400 W • Typical Heat Dissipation: 5850 BTU/hr • Ambient Temperature: 5-32 C Cooling The chassis of the Sun Fire X4600 server is segregated by an air divider into two distinct chambers: the motherboard/PCI and fan chamber and the Power Supply U
Sun Microsystems, Inc. Because heat is so destructive to electronic components, lowering operating temperature will increase reliability. Sun's cooling design enables the CPUs to operate approximately 15% more efficiently. Fans Cooling fans in the Sun Fire X4600 server are positioned in two rows for redundancy just behind the front bezel. They are hot-swappable, and each fan module has an LED fault indicator.
Sun Microsyste Each row of fan trays is occupied by two fan modules for a total of four cooling fans. In a Sun Fire X4600 server, the fan modules are populated by 172 x 160 x 51mm fans. Figure 5-3: Sun Fire X4600 Server Cooling Fan Layout Note: Do not operate the system for more than 60 seconds when a fan tray 2 or 3 is removed. Power Supplies The Sun Fire X4600 server is powered by dual redundant hot-swappable power supplies.
Sun Microsystems, Inc. Figure 5-4: Sun Fire X4600 Power Supply Units Power supplies are monitored and managed by ILOM. However, PSU fans are NOT individually managed by ILOM. Instead, they are considered part of the PSU unit. Therefore, a failure of a PSU fan is reported by ILOM as a failure of the entire PSU unit.
Sun Microsyste Chapter 6 ILOM: Integrated Lights Out Management Integrated Lights Out Management (ILOM) is the Service Processor used in Sun Fire X4600 servers. ILOM enables multiple in-band and out-of-band management solutions. In-band management through the host operating system, or platform, is enabled by IPMI and SNMP OS-resident agents. Out-of-band management is enabled through the serial port or dedicated Ethernet.
Sun Microsystems, Inc. • • • • Three USB connections to the NVIDIA nForce Professional 2200 to provide remote keyboard, mouse, and storage functionality. LPC connection to the NVIDIA nForce Professional 2200 via FPGA. 10/100 MB Ethernet connected to the onboard Broadcom BCM5221 10/100 Ethernet controller, which automatically detects straight or cross-over Ethernet cables.
Sun Microsyste In-Band Server Management Interfaces Server management through the platform operating system in a Sun Fire X4600 server is possible using either IPMI with a Keyboard Controller Style (KCS) inteface and an IPMI kernel driver or by using SNMP OS-resident agents. IPMI 2.0 and SNMP v1, v2c, and v3 are the supported standards to perform autonomous platform management functions.
Sun Microsystems, Inc. The core of the model for the DMTF SMASH CLP is a hierarchical objects namespace in which an object in the namespace is a target to a command. There are two namespaces that an SP can reference: its own namespace, whose root is /SP, and the overall system namespace, whose root is /SYS. The purpose of the /SP namespace is to represent configuration and state for the SP.
Sun Microsyste The remote storage can be either physical storage devices or bootable media image files on the hard drive. The following types of remote storage are supported by the ILOM Remote Console: • CD/DVD-ROM • Floppy • CD/DVD-ROM image (.iso files) • Floppy image (.img files) When the Sun Fire X4600 server attempts to access a Virtual Floppy or Virtual CD-ROM as set up in the BIOS, ILOM redirects this access request to the ILOM Remote Console.
Sun Microsystems, Inc. implemented in hardware and firmware. The key differentiator of Intelligent Platform Management is that these functions are independent of the main CPU, BIOS, and OS. There are two major components of platform management: the Service Processor (or BMC) and System Management Software (SMS). Intelligent Platform Management capabilities are a key component in providing enterprise-class management for high-availability systems.
Sun Microsyste devices and manage configurations, statistics collection, performance, and security on a network. SNMP is a network management protocol used almost exclusively in TCP/IP networks. The Sun Fire X4600 servers provide SNMP MIBs to manage and monitor the servers using any SNMP-capable network management system, such as HP OpenView Network Node Manager (NNM), Tivoli, CA Unicenter, or IBM Director.
Sun Microsystems, Inc. manage a group in the same manner as managing a single system. • Software Provisioning – provisions Solaris, Linux, and Windows OS, as well as firmware and software updates, from a single Web console. • Hardware and OS Monitoring – monitors hardware components and OS attributes to ensure the health of a system. Operators can customize the alarm thresholds for each component to meet business needs.
Sun Microsyste Chapter 7 Software Operating Environment The Sun Fire X4600 server supports multiple 32-bit and 64-bit operating systems, including Solaris, Linux, and Windows. The pre-installed Solaris 10 01/06 image also includes the patch 119851-13 to enable the use of the raidctl command for RAID configuration. VMware ESX Server compatibility and certification is planned for inclusion in the ESX Server 3.01 release in Q3CY06.
Sun Microsystems, Inc.
Sun Microsyste the reasons for suboptimal system and application performance. Solaris Dynamic Tracing (DTrace) technology makes it possible to delve deeply into today’s complex systems to troubleshoot problems in real time and quickly eliminate bottlenecks. Additional Solaris 10 features provide enhanced system insight, enabling a system administrator to quickly identify and resolve hardware problems and streamline and automate patch management.
Sun Microsystems, Inc. Sun Studio (90-Day Trial Version) Sun Studio 10 software is Sun's latest and best developer tool suite for C, C++, and Fortran application development. It continues to provide corporate developers and ISVs with a comprehensive, integrated suite of tools for the development, debugging, tuning, and deployment of enterprise applications on Sun platforms.
Sun Microsyste Real-Time Video Creation and Broadcast Support A Java Media Framework (JMF) technology player provides access to the latest industry-standard audio and video files, including MPEG1/2, QuickTime, VIVO, AVI, AIFF, GSM, WAV, RMF, AU, and MIDI. Solaris 10 OS Bundled Desktop Environments The following desktop environments are included with the Solaris 10 Operating System.
Sun Microsystems, Inc. GNOME 2.0 for the Solaris OS is open source, free software with no upgrade or licensing fees. More information about GNOME 2.0 is available at http://www.sun.com/gnome. Solaris OS Licensing and Usage Under the Free Solaris™ Binary License Program, Sun is making the binary (runtime) version of its Solaris 10 OS available to anyone who accepts the terms of the Solaris OS Binary Code License (BCL) and the Free Solaris Binary License Program.
Sun Microsyste Chapter 8 Additional Software Included Sun Installation Assistant The Sun Installation Assistant software on CD-ROM is included in the ship-kit with every Sun Fire X4600 server. Sun Installation Assistant is a Linux installation utility that reduces the complexity of installing supported Linux distributions on new hardware. A newly released platform seldom has a certified set of drivers already in a current Linux distribution.
Sun Microsystems, Inc.
Sun Microsyste 5 6 7 8 NC RXNC NC No Connect Negative Side of Receive Data No Connect No Connect Figure A-3: 10/100 BaseT Pinouts Serial Attached SCSI (SAS) Table A-4: Serial Attached SCSI (SAS) Pinouts Pin # Pin Name Description Signal Segment S1 GND 2nd mate S2 TX+ Transmit from PHY to hard drive S3 TXTransmit from PHY to hard drive S4 GND 2nd mate S5 RXReceive from hard drive to PHY S6 RX+ Receive from hard drive to PHY S7 GND 2nd mate Backside Signal S8 GND 2nd mate S9 S10 S11 GND 2nd mate S12 S13
Sun Microsystems, Inc.
Sun Microsyste 9 10 11 12 13 14 15 KEY S_GND IDO IDI HSYNC VSYNC ID3/SCL No Pin Syn Return (Ground) IDO (Ground) IDI (No Connect) Horizontal Sync Vertical Sync No Connect Figure A-6: VGA Pinouts 59 of 63
Sun Microsystems, Inc. Appendix B Sun Fire X4600/X4600 M2 Rack Mounting Specifications Sun Fire X4600/X4600 M2 server should fit in nearly all 3rd-party racks that meet these criteria: • • • • • • Rack has a horizontal opening with unit vertical pitch conforming to ANSI/EIA 310-D-1992 and/or IEC 60927 Four-post structure (i.e.
Sun Microsyste Appendix C Sun Fire X4600/X4600 M2 Server Dimensions The Sun Fire X4600/X4600 M2 server has the following size and weight dimensions. Table B-1: Sun Fire X4600/X4600 M2 Server Dimensions Weight Height Width Depth Sun Fire X4600 and X4600 M2 106 lbs. / 48 Kg maximum with rack kit and cable mgmt arm 88 lbs. / 40 Kg maximum standalone server 6.9 in. / 176 mm 4RU 17.5 in. / 445 mm 24.75 in. / 629 mm chassis with bezel 25.25 in.
Sun Microsystems, Inc. Appendix D Environmental and Regulatory Compliance The Sun Fire X4600 server meets or exceeds the following environmental and regulatory specifications.
Sun Microsyste Sun Microsystems, Inc. 4150 Network Circle, Santa Clara, CA 95054 USA Phone 1-650-960-1300 or 1-800-5559SUN Web sun.
