Xen Expert Days Virtualization with Xen SUSE Linux Enterprise Server 10 Ralf Dannert Technology Specialist rdannert@novell.
Agenda 2 • Use cases • Terminology and Architecture • VM installation • Using Xen • Case Studies • Roadmap • Novell offerings • Helpful Links © Novell Inc.
Server Virtualization: Analyst's View 3 © Novell Inc.
© Novell Inc.
Executive Summary ● Novell product portfolio offers choice – ● Novell virtualization strategy is focused on Xen – ● 5 Customers can choose to deploy virtualization technologies provided by Novell and other VT vendors Customer demand for server consolidation and price / performance will foster rapid acceptance Novell supports customers – Virtual Machine Server Hardware from partners and Virtual Machine configurations are listed in YES certification bulletins – Novell Technical Services supports ins
Use Cases
Uses of Virtualization Virtual Resources Sharing Virtual Resources Aggregation Physical Resources Examples: VMs, LPARs, Virtual Disks, VLANs Physical Resources Examples: Virtual Disks, Virtual Storage Pools Virtual Resources Extension Transparent Change Physical Resources Examples: iSCSI, Architecture Emulators 7 © Novell Inc.
Use cases • Virtualization allows for more flexibility – Virtual Machines (VM) isolate hardware differences due to a abstracted resource layer between hardware and OS – Decouples software stacks from hardware life cycles – Dynamic provisioning reduces time to operation: preconfigured application stacks are faster to deploy – Integrated high availability increases reliability A A OS A OS A OS A A OS VM Virtualization Layer Hardware 8 © Novell Inc.
Terminology and Architecture
SLES 10 VM Server SLES 9 and Windows XP - Fully Virtualized VMs 10 © Novell Inc.
Xen Technology Background Originally a research project from University of Cambridge Open source Xen 2.0 released November 2004 Xen 3.0.0 released December 2005 Xen 3.0.2 release May 2006 (SLES 10 Target) Xen 3.0.4 SLES 10 SP1 Target 11 © Novell Inc.
Xen Community Terminology Domain: A container for a running virtual machine. Colloquially, the VM itself. Domain 0: The first domain. Privileged to manage other domains. a.k.a. “dom0”. Unprivileged domain: Any domain other than domain 0. Cannot manage other domains. a.k.a. “domU”. Driver domain: A domain that contains physical drivers. Usually this is just domain 0. Physical driver: A device driver (usually in the driver domain) that talks to the hardware.
Full & Paravirtualization Overview Full Virtualization Apps Apps Operating System Operating System Virtual Machine Virtual Machine Apps Operating System A P Virtual A I Machine P I Apps Operating System A P Virtual A I Machine P I Virtual Machine Monitor Virtual Machine Monitor HW Platform HW Platform Runtime modification of Guest OS: VMM manages the conflict, then returns to OS 13 Paravirtualization © Novell Inc.
Novell Terminology Fully Virtual: A VM mode that can run a native, unmodified operating system by emulating all hardware devices. Paravirtual: A VM mode that can run a modified operating system, which cooperates with the VMM. VT Computer: Computer supporting HVM Intel VT, AMD Standard Computer: A computer that does not support virtualization technology and therefore can run Xen VMs only in paravirtual mode.
Privilege Rings Xen runs at ring 0 (highest privilege) All domains run at rings 1 - 3. • Kernel is ring 1 • User-space is ring 3 Applications Applications ring 3 domain 0 (management) Linux Kernel Kernel Hypercalls Events Hypervisor (XEN) Physical Hardware 15 © Novell Inc.
Hardware assisted virtualization •VT Computer •run multiple OS concurrently •protected execution environments •priviledge ring expansion •simplify hypervisor •Intel VT for directed I/O(VT-d) - direct assign I/O –no emulated drivers necessary 16 © Novell Inc.
Extending Intel Virtualization Technology • support for I/O device virtualization – • direct I/O virtualization to the chipset(“VT-d”). Currently, I/O devices aren’t aware of virtualization and must go through the VMM before being assigned to a virtual machine.
Full Virtualization Mode on VT using qemu-dm using “device model” ●hypervisor intercepts mmio regions ●forwards request to qemu ●i.e.: read request to harddisk ● VM emulates the following devices ●requires the VM's operating system to install, load, and run its native device drivers ● Network card: AMD PCnet, NE2000 ● Disk drive: IDE ● Graphics card: Cirrus Logic* GD5446, VESA-compliant VGA ● Input: PS/2 mouse and keyboard ● Sound: Creative* Sound Blaster 16, Ensoniq* ES1370 ● 18 © Novell Inc.
Intel Pre- and Post-VT 19 © Novell Inc.
AMD IOMMU •in DomU OS not loaded at address 0 •Xen: direct access to memory difficult-->corruption •hypervisor intervenes in I/O, apply translation-->overhead •solutions: •rewrite graphics driver ? •HW to support IOMMU •AMD IOMMU -provides isolation and memory protection •IOMMU: device remap address accessed by HW, 20 © Novell Inc.
Ring Buffers Network, disk I/O is done via shared memory, asynchronous ring buffers. One ring per VM reduces cross-talk. Events replace hardware interrupts for notifications. Page-for-page swap between VM and Xen. request consumer request producer response consumer response producer 21 © Novell Inc.
Xen Architecture – Simple View Dom0 Dom1 Dom2 Linux Linux Netware Pd Vd Vd Hypervisor Hardware Pd = Physical Device Driver Vd = Virtual Device Driver
Xen Architecture – Simple View Dom0 is the management domain for Xen guests • controls compute resources dynamically (e.g. memory, CPU, I/O) • provides interfaces to the physical server.
Xen Architecture – Simple View Dom0 is the management domain for Xen guests • controls compute resources dynamically (e.g. memory, CPU, I/O) • provides interfaces to the physical server.
Xen Architecture – Simple View Dom0 is the management domain for Xen guests • controls compute resources dynamically (e.g. memory, CPU, I/O) • provides interfaces to the physical server.
Distribution • pDistro – Tuned thin, platform distribution specialized for specific physical hardware system drivers hypervisor (vmm) © Novell Inc.
Xen Technology – Architecture (cont) 27 © Novell Inc.
Emerging Virtual Architectures vDistros Are used to deploy virtual machines on any physical server and to move application stacks between them in 'real-time' Novell Customer Center vDistros Application stack distribution containers w/ fault containment and intrusion protection Virtual Machines (XEN/VMW) VM Management pDistros Are used to deploy physical machine specific hypervisors with those drivers and agents needed by specific hardware vendors.
VM Installation
VM Installation using YaST Novell makes VM installation simple with YaST VM Installation using YaST on SLES 10 • Support for para-virtualized guests • Support for fully virtualized guests • Support for new installation or use of existing disk • Install over network (SUSE) or from CD/DVD (SUSE & others) Support for simple lifecycle management: • Start • Stop • View • Shutdown 30 © Novell Inc.
SLES 10 VM Installation Using YaST 31 © Novell Inc.
VM Disks – Supported Storage • mapping “virtual” disk (within the VM) and a “physical” disk (within a driver domain). • Generic Block Devices (SCSI, SATA, IDE, LVM, ...) • iSCSI Target (Server), iSCSI Initiator (Client) • FibreChannel Protocol, SAN • Native Multipath IO (Kernel Devicemapper - DMMPIO) • Multidisk (MD) • Oracle Cluster Filesystem (OCFS2), Heartbeat2 • Enterprise Volume Manager (EVMS2) – 32 Novell iSCSI, OCFS2 and Heartbeat2 Plugins ® © Novell Inc.
How Does Xen Compare? •Performance: 1-20% actual overhead (older claim: 1-5%) •Scalability: 10s of Vms •Memory overhead: Negligible for para. ~12MB per VM for full. •Hardware-enforced fault isolation between VMs: Yes •Ability to dynamically grow/shrink hardware resources: Yes •Open source •Commodity hardware •Ease of deployment: Use YaST •Support: From Novell 33 © Novell Inc.
Using Xen
Booting VM Server (Xen) If you selected the Xen pattern during installation, Grub should be correct Installing kernel-xen or kernel-xenpae later should update Grub, also Select “XEN” boot option from Grub Grub loads Xen, Linux kernel, and initrd. Xen initializes, then hands control to Linux kernel 35 © Novell Inc.
Troubleshooting Boot Loader Copy/modify the normal entry in /boot/grub/menu.lst: title Xen root (hd0,0) kernel /boot/xen.gz dom0_mem=65536 module /boot/vmlinuzxen root=/dev/hda1 vga=0x31a selinux=0 splash=silent resume=/dev/hda2 elevator=as showopts module /boot/initrdxen Note: • dom0_mem is in KB. (Optional. Defaults to taking nearly all memory.) • Copy kernel args verbatim • Specify “module” not “initrd” 36 © Novell Inc.
Daemons xend: • Required • Runs in VM Server • Must be running to access management commands • Manages VM configuration xendomains: • Optional • Starts (or restores) VMs when VM Server starts • Stops (or saves or migrates) VMs when VM Server stops 37 © Novell Inc.
VM Configuration Files Use an example file as a template: cp /etc/xen/examples/xmexample1 /etc/xen/vm/MyVm Fields to change: if manual kernel: kernel: The location of the kernel, as visible from dom0. initrd: (*nix-centric) Extra module for kernel, as per Multiboot specificiation. if domUloader: bootloader: /usr/lib/xen/boot/domUloader.py bootentry: paths of kernel and initrd to extract from VM's disk. memory: Memory to give the guest, in megabytes. name: The name must be unique among running VMs.
i.e.: /etc/xen/vm/sles10pv disk = [ 'file:/var/lib/xen/images/sles10pv/hda,hda,w', 'phy:/dev/hdc,hdb,r' ] memory = 256 vcpus = 1 builder = 'linux' name = 'sles10pv' vif = [ 'mac=00:16:3e:41:09:60' ] localtime = 0 on_poweroff = 'destroy' on_reboot = 'restart' on_crash = 'restart' extra = ' TERM=xterm' bootloader = '/usr/lib/xen/boot/domUloader.py' bootentry = 'hda2:/boot/vmlinuz-xen,/boot/initrd-xen' 39 © Novell Inc.
/etc/xen/examples/xmexample.iscsi kernel = "/boot/vmlinuz-xen" ramdisk = "/boot/initrd-xen" memory = 128 name = "nbd" # Please change MAC vif = [ 'mac=00:16:e3:10:00:a0, bridge=xenbr0' ] # Replaced the ':' in the iname with '@' disk = [ 'iscsi:iqn.2006-09.de.suse@0ac47ee2-216e-452a-a341a12624cd0225,hda,w'] dhcp = "dhcp" hostname= "nbd" root = "/dev/hda1" extra = "3" 40 © Novell Inc.
VM Migration Migration • Moving a VM from one physical machine to another, without interrupting the VM's state. • IP address follows the VM, so networking is not interrupted Live Migration • Migration with minimal down-time (10's of ms) • Streams VM's pages over network while VM is still running Disk must still be visible on destination! (e.g., use iSCSI) 41 © Novell Inc.
Managing VMs with the xm Command Must be root. Use “xm” command (Xen Management) Querying: xm list xm list -l xm console id (disconnect from VM's console with Ctrl-]) Managing: xm create -c configfile xm shutdown id xm destroy id xm migrate id destination-host xm migrate -l id destination-host 42 © Novell Inc.
More xm Commands Hot-plugging: xm mem-set xm block-attach xm block-detach Debugging: xm info xm top xm dmesg For more information: man xm xm help --long 43 © Novell Inc.
Files & Directories /etc/xen/xend-config.sxp Machine-wide configuration /etc/xen/vm/ /etc/xen/auto/ /etc/xen/examples/ /etc/xen/scripts/ VM definition files VMs to auto-start Example VM definition files Scripts (see xend-config.sxp) /var/lib/xen/images/ VM disk images /var/log/ Log files, when things go wrong /usr/share/doc/packages/xen/README.SuSE Tips & late-breaking information 44 © Novell Inc.
Convert VMware image to Xen image dd if=/dev/zero of=disk.img bs=32k count=32k #Virtual Hard Disk to Network Block Device mapper vmware-loop -p SLES9-0.vmdk modprobe nbd vmware-loop SLES9-0.vmdk 2 1 #Extract VMware partition from the device to the image file SLES9.img: dd if=/dev/nbd1 of=/tmp/SLES9.img bs=32k #inject the root FS into the Qemu image. file disk.img disk.img: x86 boot sector, FREE-DOS Beta 0.9 MBR fdisk disk.img .. Units = cylinders of 16065 * 512 = 8225280 bytes ..
Case Studies
Case Study: HA Web Server • Small office solution – High available web server for non-disruptive online business > Redundant setup using 2 machines with SLES 10, LAMP stack and heartbeat2: Xen based apaches, remote storage access via iSCSI and OCFS2, web shop application > Automated load balancing and high availability, easy extension of compute and storage resources Shared Storage 47 © Novell Inc.
Case Study: HA mail, file and print • Small office solution – 48 Small scale consolidation with increased throughput and improved fault tolerance > Past: 4 dedicated servers for mail, file, print and an Internet gateway > Today: 2 state-of-the-art servers running mail, file, print and Internet gateway in dedicated and thus isolated VMs, DMZ realized with firewall VM, all VMs instrumented with Heartbeat v2 to fail-over services on standby virtual machines.
Novell Xen Roadmap
Novell Virtual Server Architecture Overview ● ● ● ● ● ● ● 50 SUSE Linux Enterprise Server 10 includes Xen 3.0.
SUSE Linux Enterprise Server 10 Xen 3.0.2 support status Overview ● Runs both SLES 10 VM Server / SLES 10 VM (32/32bit and 64/64bit) ● VM server RAM support: x86 4GB, x86 PAE 16GB, x86-64 32GB ● Multiple vCPUs, NICs and disks for VM Server ● Multiple vCPUs, vNICs and vDisks for VM ● Technical preview allows to run different unmodified OS if hardware support is present (full virtualization) Maintenance update Oct 2006 51 • Updated hypervisor to credit scheduler of Xen 3.0.
Xen 3.0.
Roadmap - Xen 2006 2007 Virtual Machine Server • SLES10 Xen VM server • SLES10 paravirtualized VM Tech Preview unmodified OS with Intel VT / AMD Virtualization SLES 9, 10 x86 ● RHEL 4 x86 ● Microsoft Windows Server ● Solaris 10 x86 ● Technical Preview PV OS • SLES9 SP3 paravirtualized VM Virtual Machine Server • SLES10 Xen VM server • SLES10 PV VM • OES2 VM server • Netware PV VM Unmodified OS using Intel VT and AMD-V SLES 9 / 10 MS Windows Server x86 ● MS Windows XP x86 ● RHEL 4 x86 ● ● Technical Prev
SUSE Linux Enterprise Server 10 SP1 Xen 3.0.4+ planned support Q2/2007 Overview ● ● 54 CIM provider for VM lifecycle mananagement, monitoring and resource management CIM client for VM lifecycle mananagement, monitoring and resource management ● Update from Xen 3.0.2 to 3.0.4+ (due to system management) ● SLES 9 PV installation support (preview / documented) ● OES2 using Netware 6.5 SP7 PV ● VM server RAM support: x86 4GB, x86 PAE 64GB, x86-64 256GB ● Basic NUMA support © Novell Inc.
Xen 3.0.4+ support status (1) Overview – planned Xen supported features 55 • x86 PAE max VM server physical memory: 64GB • x86 PAE max VM memory: 62GB • x86-64 max VM server physical memory: 128GB • x86-64 max VM memory: 126GB • 32 logical hardware threads per VM Server, 256 VMs per VM Server • 8 vCPUs per VM, 8 VMs per physical CPU core • 8 vNIC per VM © Novell Inc.
Xen 3.0.4+ support status (2) Overview – planned Xen supported features 56 • Save & Restore of Virtual Machines • Live Migration of paravirtualized VMs • use of asynchronous IO with sparse file mounted loopback in Virtual Machine Server (dom0) • paravirtualized device drivers for IO and network loads for Linux and MS Windows to boost fullvirtualized OS performance © Novell Inc.
OES2 virtualization support • OES2 shares same common code base of SLE 10 SP1 • NetWare runs as paravirtualized OS 32-bit mode on 32bit and 64bit processor architectures that support the x86/x86-64 instruction set.
Virtualization Pilot Program • Intel-VT or AMD-V hardware assisted • Full Virtualization with PV Device Drivers: • 58 – SLES 9 and RHEL 4 – MSFT WinXP/2000/2003 increase in performance © Novell Inc.
cluster aware logical volumes for hosting Xen domUs
cluster aware hosting of Xen domains • Howto use Heartbeat and EVMS (in dom0) to create cluster aware logical volumes for hosting Xen domUs – Heartbeat 2.0.4 or later – EVMS 2.5.5 or later – hb2-1.0.0.so is the EVMS plugin for Heartbeat2 • • 60 Add these lines to your /etc/ha.d/ha.cf – respawn root /sbin/evmsd – apiauth evms uid=hacluster,root © Novell Inc.
cluster aware hosting of Xen domains cont • Start Heartbeat2 and it'll start the EVMS daemon on each node – • 61 Renewal time frames are not well communicated or defined verify the cluster aware behavior of EVMS – evmsgui->Settings->Node Administered... – all cluster nodes listed – EVMS and Heartbeat interact properly © Novell Inc.
cluster aware hosting of Xen domains Summary • use EVMS to create a logical volume per each Xen domU OS image – e.g.10G image of hda – /dev/evms/MyVirtualMachineHda – copy guest OS image into that logical volume – logical volume will have same (persistent) device name across all cluster nodes – corresponding Xen control file will be location transparent too – reference /dev/evms/MyVirtualMachineHda as path for domU's hda on all nodes > 62 http://sourceforge.net/mailarchive/message.
PV drivers
PV Driver for Windows • – Novell PV Xenbus Driver for Windows – PCI Device --> Update Driver > 64 Xen Virtual Block Device – Add a New Hardware Device – Network Drivers – Xen Virtual Nic --> Update Driver © Novell Inc.
setting up PV drivers for SLES9 • create new SLES9 VM – edit /boot/grub/menu.
Novell Xen based offerings
Infos on Pricing & Policies Easy pricing and policy is in place • Today we are charging for first instance (physical or virtual) – first SLES 10 on physical machine or VMware etc – http://www.novell.com/products/server/virtualization.html • Outlook – SLES 10 / SLED 10 or vice versa -> pay first server (like current policy) – OES2 / Netware PV is planned to pay per user – SLES 10 / OES2 PV o.
Microsoft and Novell Collaborating on Virtualization and Interoperability Collaboration on software to enable SUSE Linux Enterprise Server 10 to run as a fully virtualized guest on SP1 of Virtual Server 2005 R2 SUSE Linux Enterprise Server 10 add-ins will be available in 2007 Collaboration on software that translates between virtualization technologies Windows Server “Longhorn” supporting paravirtualized SUSE Linux Enterprise Server 10 guest SUSE Linux Enterprise Server 10 supporting enlightened Wind
Where Do I find Information ?
External Resources Where do customers get more information • Novell Virtualization home page – http://www.novell.com/linux/virtualization/ – Virtualization Technical Library, Whitepapers • Novell's pricing for virtualization and available support offerings – http://www.novell.com/products/server/virtualization.html – http://support.novell.com/linux/ • Novell online documentation on Xen http://www.novell.com/documentation/vmserver/ • Supported hardware and technical limits – http://developer.novell.
Other Resources Recommended links • Xen project home http://www.xensource.com/xen/ wiki – Xen summit presentations – Xen architecture docs, developer discussions, future roadmap – Recent changes, indexes, doc, FAQs – Deployment examples, third party projects related to Xen • Latest Novell's Xen externally available technical preview http://forge.novell.com/modules/xfmod/project/?xenpreview – Latest code for test (SLES 10, SLES9, ...), not production 71 © Novell Inc.
Outlook: Data Center Automation
The Solution – Management Blueprint 73 © Novell Inc.
ZENworks Orchestration Server ® Resource Discovery Workload Management Dynamic Scheduling VM Lifecycle Mgmt Policy Management Auditing/Accounting Autonomic Availability 74 © Novell Inc.
ZENworks Orchestrator 75 © Novell Inc.
ZENworks Orchestrator Features • Agenten basiertes Entdecken • Paralleles Ausführen • Abschätzung der Workloads • Planen • Überwachung – Nutzdaten, Abrechnung – 76 © Novell Inc.
ZENworks Orchestrator Features - Continued • • 77 Erweiterbare Job Definition – Sprache: Python – Regeln: XML Format – Voreinstellungen historischer Daten – Ressourcen Optimierung Regelbasiert – Reservieren der Ressourcen – Vorrang kritischer Aufgaben – ereignisorientiert © Novell Inc.
ZENworks Virtual Machine Management Features • 78 Heterogenes Virtual Machine Management – VMware, Xen, Microsoft – Entdecken der Server für VM Kommissionierung – Off/On-line Vm's & Templates – Deployment, re-deployment/rollback – Assoziieren physischer, virtueller und Storage Compute Nodes © Novell Inc.
ZENworks Virtual Machine Management Features • Cluster-aware Virtualisierung – • 79 Redundanz, Hochverfügbarkeit, Disaster Recovery Regel-basiertes, dynamisches Workload Deployment – Job Zuweisung & Regelausführung – Verteilen und Bewegen der Workloads dynamisch © Novell Inc.
Novell Portfolio Integration Applications VS VM RM IDM Backup SUSE Enterprise Linux Virtual Machines • • • Consolidation Migration Effective hardware utilization Virtual Storage • • • • 80 Volume manager High availability software Cluster file system Business Continuance © Novell Inc.
Virtual Machine Lifecycle Discover On-box incubator Check in Version control Create On-box tool (find) Incubation job (make) Edit Config: Change (using Sandbox) Runtime: Facts (temporary) Config (static) Developer Actions Operator Actions Runtime (Dynamic) Destroy Deploy Test (provision) Production (Sandbox) Manage Lifecycle, Cloning Location, Runtime 81 © Novell Inc.
Virtual Machine Lifecycle Control • • System maintains library of VMs and images, hosts – Like physical resources VMs have ‘facts’ describing attributes – VMs can be grouped Actions can be performed on VMs: – 82 Provision, Shutdown, Suspend, Create Template, Create VM, Create Template From Physical, Create VM From Physical, Clone VM, Clone Online VM, Clone Template, Destroy, Restart, Migrate, Check Status, Template to Instance, Instance to Template, Affiliate With Host, Make Standalone, Checkpoint,
Virtual Machine Lifecycle Control (cont.) • VM lifecycle controlled by: – – – Programmatically (Job / JDL) Manually (through mgmt.
Virtual Provisioning and Life Cycle Management 84 © Novell Inc.
Architektur ZENworks Orchestration Server • Orchestration - Rules engine - Policy - Scheduling - Grid services • - Utilization & billing Storage Resource Manager • Discovery • Logical unit naming • Quality of service • ACL control Management Server • Physical inventory • Bare metal provisioning • Patch & Update • Configuration • Remote Control UMF VM Repository • Visualization • Reporting Data Model Reconciliation Federation Managed Nodes • CIMOM • CCM Agent • Orchestration Agent 85 © Novell Inc,
© Novell Inc.
© Novell Inc.
© Novell Inc.
© Novell Inc.
© Novell Inc.
Unpublished Work of Novell, Inc. All Rights Reserved. This work is an unpublished work and contains confidential, proprietary, and trade secret information of Novell, Inc. Access to this work is restricted to Novell employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of Novell, Inc.
