Stacking Dell Networking Switches: N4032, N4032F, N4064, N4064F Dell Engineering February 2014 A Dell Deployment and Configuration Guide
Revisions Date Description Author(s) February 2014 Initial Release Victor Teeter THIS PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND. © 2014 Dell Inc. All rights reserved. Reproduction of this material in any manner whatsoever without the express written permission of Dell Inc. is strictly forbidden. For more information, contact Dell.
Table of contents Revisions ............................................................................................................................................................................................. 2 1 2 Introduction ................................................................................................................................................................................ 4 1.1 Stacking and management..................................................................
1 Introduction Most Dell switches today include a stacking feature that allows multiple switches to operate as a single unit. These stacks include up to twelve Dell Networking N4032/N4032F/N4064/N4064F switches in any combination, using up to four links between stack members. Any Ethernet port type on the front panel can be used in creating these links, including SFP+, QSFP+, and 10GbaseT.
1.1 Stacking and management An important advantage of stacking is that it provides a consolidated interface for management of multiple switches that are linked together. One switch acts as the Master through which the entire stack is managed through various interfaces (Web, CLI, and SNMP). After a stack is deployed in the network, operators can easily add units to the stack as their port requirements increase, with minimal administrative overhead.
Daisy Chain topology 1.5 Ring (loop) topology Nonstop forwarding on the stack The Nonstop Forwarding (NSF) feature allows the forwarding plane of stack units to continue to forward packets while the control and management planes restart as a result of a power failure, hardware failure, or software fault on the stack Master and allows the standby switch to quickly takeover as the Master. 1.
console#show run !Current Configuration: !System Description "N4064F, 6.0.1.3, Linux 2.6.32.9” !System Software Version 6.0.1.3 !Cut-through mode is configured as disabled ! configure slot 1/0 5 ! N4064F slot 1/1 8 ! Dell 10GBase-T Card stack member 1 4 ! N4064F exit interface out-of-band ip address 172.25.194.24 255.255.0.0 172.25.194.
Note: The terms Master and Manager/Management Unit are often used interchangeably in regards to stacking. In most cases, a switch that is added to an existing stack becomes a stack member, and not the Management Unit.
3 Stack Mbr N4064 N4064 OK 6.0.1.3 Note that in this example Switch 1 is the Master (Mgmt Sw) and Switch 2 is the Standby (Oper Stby) ready to take over as Master in the event the Master fails. To find the same information from the Web UI go to System > Stack Management > Stack Summary. 1.9 How to select a Master during initial stack setup After creating a stack it is easy to go into the settings on the Master and select any of the members to take its place as Master.
Stacking Dell Networking Switches: N4032, N4032F, N4064, N4064F
2 Stacking scenarios The following sections present examples in a variety of areas concerning stacking the Dell N4000 switches and provides step-by-step guidance using the CLI and Web UI, with screen shots as a visual guide. Consult the table of contents above for a list of examples covered in this document. Each scenario in this document assumes that all Dell switches are using the same firmware revision. It is also recommended that all devices are using the latest firmware version.
2.1.1 QSFP+ quad-breakout cables QSFP+ quad-breakout cables may also be used for stacking. This allows a 40G QSFP+ port to connect to four 10G SFP+ ports on the adjacent switch. When a breakout cable is used, the connections are considered to be four links taking up four ports.
2.2 Creating a Stack Examples below provide steps on how to create a stack. Graphics shown in this section only depict some of the possibilities of how to cable together members of a stack. Note: While the cable pictures below come before the configuration steps, it is important not to cable the stack until instructed to do so.
stacking port on the next switch. This process is repeated until all of the devices are connected. To complete the ring topology for the stack, one stacking port on the last switch is connected to a stacking port on the first switch. See the Ring vs. daisy-chain topology section for more information on ring topologies. Things to consider when cabling a stack together: Only use interfaces with the same bandwidth to stack between members.
Connect power to all switches to be stacked. Performing the command show switch stack-ports will provide information on the unit number for the switch, which is used in the stacking commands. It also shows Interfaces of the switch that may be used for stacking and whether they are in Ethernet or Stacking mode.
Te1/0/3 Te1/0/23 Te1/0/24 Fo1/1/1 Fo1/1/2 Ethernet . . . Ethernet Ethernet Stack Stack Ethernet Link Down Unknown Ethernet Ethernet Ethernet Ethernet Link Link Link Link Unknown Unknown 40 40 Down Down Down Down The Configured Stack Mode is now Stack, but the Running Stack Mode is still Ethernet. The Running Mode changes to Stack upon reloading the switch, as instructed below. Save the configuration to the Startup-Configuration.
2.2.1.1 Validation After the entire stack is created, it can be validated with the show switch command from the Master switch: console#show switch SW --1 2 3 Management Status ---------Mgmt Sw Stack Mbr Stack Mbr Standby Preconfig Status Model ID --------- ----------N4064F Oper Stby N4064 N4032 Plugged-in Model ID ----------N4064F N4064 N4032 Switch Status -------OK OK OK Code Version ------6.0.1.1 6.0.1.1 6.0.1.
Select a switch to be Master of the stack and for managing all ports and member units with the stack. Login to the Web UI for this switch by entering the IP address of the switch into a supported Web browser. A username and password is required and should be setup beforehand on each switch. See the switch User Guide for more information regarding username, password, and Web access. After login, the first screen to appear is the Home screen, which shows the current stacking member number.
Click System > Stack Management > Stack Port Summary to bring up the next page used to select the ports for stacking. For this example ports Te1/0/7 and Te1/0/8 are used. Selecting the Edit check box activates the drop-down menu in the Configured Stack-mode column. Use this menu to change port modes from Ethernet to Stack. Click Apply. This will set the Configured Stack Mode to Stack, but the Running Stack Mode remains Ethernet until the switch is reloaded as instructed below.
Save the configuration to the Startup-Configuration on the System > File Management > Copy Files page. Perform the same steps for the remaining switches. Don’t forget to configure stacking ports on the last and first switch to create a more resilient ring topology if desired. Once every switch has been configured, remove power from all switches. While powered off, cable all of the switches together using the stack ports that were configured.
The following figure shows shows the Stack LED that is located on the back panel of every Dell N4000 series switch. Figure 12 Stacking LED for the Dell N4000/N4000F series switch When the switch is the Master of a stack, the LED glows solid blue. If it is a stack member only, it glows solid amber. When in stand-alone mode, the LED is off. Note: Each stack members’ role (including the Master and Standby) can be defined at any time after the initial stack is created. 2.2.
stack. In the event the new stack member unit being added has a newer firmware version, a downgrade is also allowed. To prevent the downgrade of the new stack member unit’s firmware, disable the AllowDowngrade option on the Master. This allows all new member units to synchronize with the Master only if the firmware revision of the new members is older than that of the Master. Use the System > Stack Management > Stack Firmware Synchronization page to view and set these features.
2.3 Adding new member units to a stack It is important that all stack member units run the same version of firmware. Make sure to either upgrade firmware on the new units to be added to match the firmware on the Master, or use the automatic firmware update method found in section Automatic firmware updates for new members joining the stack. The example below shows how to add a stack member to an existing stack.
console#config console(config)#stack console(config-stack)#stack-port fortygigabitethernet 1/1/1 stack console(config-stack)#stack-port fortygigabitethernet 1/1/2 stack console(config-stack)#show switch stack-ports | include Stack Stack Stack Link Link Fo1/1/1 Stack Ethernet Link Down 40 Fo1/1/2 Stack Ethernet Link Down 40 The Configured Stack Mode is now Stack, but the Running Stack Mode is still Ethernet. The Running Mode changes to Stack upon reloading the switch, as instructed below.
Note: Ring topologies are highly recommended for resiliency and fault tolerance over the daisy-chain. Daisy Chain topology Ring (loop) topology On a daisy-chain stack: 1. Configure the existing stack with new stack ports to be used to connect to the new switch. 2. Cable the new switch to one of the end units of the existing stack. 3. Power up the new switch. The switch will be added to the stack.
SW --1 2 3 4 Status Status ---------- --------Mgmt Sw Stack Mbr Oper Stby Stack Mbr Stack Mbr Model ID ----------N4064F N4064 N4032 N4032 Model ID Status Version ----------- -------- ------N4064F OK 6.0.1.3 N4064 OK 6.0.1.3 N4032 OK 6.0.1.3 N4032 OK 6.0.1.3 The example above shows a new stack member (N4032) was added. The show switch stack-ports command can also be used to see all of the ports that have been added to the stack as well as which ones are being used for stacking.
2.3.2 Web interface method The example below shows how to add a stack member to an existing stack using the Web user interface. Before cabling an additional member to a stack, perform the steps below to setup the switch to join the stack. Once configured, continue to follow the instructions for cabling to complete the task. This works best when adding one stack member at a time. If multiple new members are to be added, follow these directions and complete the install of one before going to the next.
In the Edit column, select the check box to allow editing on the ports to be used for stacking. Then use the drop-down menu in the Configured Stack-mode column to put each port in stack mode. The Configured Stack Mode is now Stack, but the Running Stack-mode does not show Stack until the switch is reloaded, as instructed below. Save the configuration to the Startup-Configuration using the System > File Management > Copy Files page. After saving the configuration, remove power from the switch.
Note: Hot insertion of units into the stack is not supported. Do not connect a new member to the stack while the new unit is powered up. Follow the steps in the following section to properly add the stacking cables and complete the process. 2.3.2.1 Ring vs.
2.3.2.2 Validation After the entire stack is created, validate the configuration by selecting System > Stack Management > Stack Summary from the main navigation menu: The example above shows a new stack member (Unit 3) was added. If a switch (SW) number is configured prior to adding it to the stack and there are no other devices using that number, then the switch continues to use the same number. Otherwise, the switch sets its number to the lowest unassigned number upon joining the stack.
2.4 Updating firmware on a stack Note: Make sure to check the Release Notes and any special instructions that may have come with new firmware updates. It is important to follow instructions found in those documents if they deviate from this white paper. Steps for upgrading the firmware on a stack of switches are similar to upgrading the firmware on a single switch.
TFTP code transfer starting 12487236 bytes transferred... File contents are valid. Copying file to flash... Attempting to send the STK file to other units in the stack… File transfer operation completed successfully. Note: This procedure may take a few minutes longer than updating a single, non-stacked switch. Perform another show version command to see the new firmware has been downloaded to all members of the stack. Notice the firmware was only copied into the backup image for each member.
3 4 6.0.0.8 6.0.0.8 6.0.1.3 6.0.1.3 6.0.0.8 6.0.0.8 6.0.1.3 6.0.1.3 Notice the next-active column now shows the same version as the backup column. Next, type the reload command to complete the update. All switches in the stack reloads. console#reload Are you sure you want to reload the stack? (y/n) y Connection to host lost. The “Connection to host lost” message appears only if performing the reload from a telnet session. It will not appear if performing the command from a console session.
Each member of the stack will be updated with the latest firmware simply by updating the Master switch. The screenshots below demonstrate how to copy the firmware to the Master using a TFTP server. The stack needs access to a TFTP server on the network and the firmware file needs to be present in the download folder of the TFTP server. From the System > File Management > File Download page, select the appropriate settings. The most common settings are shown here. Click Apply. A warning will appear.
Click OK. Note: Be patient as this procedure may take several minutes longer than updating a single, non-stacked switch. The following screen appears for several minutes while the firmware is being downloaded to the Master and distributed to all members of the stack. Once the file has been distributed to the entire stack, the following dialog box appears.
Click Close. Go back to the Active Images screen to confirm that the new firmware was downloaded to all members of the stack. Notice the firmware was only copied into the backup image for each member. The next step is to activate the image that contains the new firmware using the drop-down menus in the Next Active Image column. In this example the backup image is selected for each unit.
After selecting backup for each unit, click Apply. Go to the System > General > Reset page to reload the switch. Click Apply to reload all switches in the stack. If the following screen appears, click OK. Click OK again if this screen appears.
If either of the following screens appear, they are simply notifying the user that the network has been lost. This is only temporarily while the stack reloads. Click OK if prompted with the above screen. Rebooting the stack may take a few minutes longer than a single switch. After reload, the firmware upgrade is complete. To validate each switch, login again to the Web UI of the stack Master, and then go to System > File Management > Active Images.
2.5 Removing member units This section provides guidance on how to remove member units from a stack while minimizing traffic flow interruptions within the stack. Removing any member of a ring topology stack does not require a reload of any member unit in the stack.
associated with these interfaces. Use the locate switch command to blink the blue “Locator” LED on the switch unit you are trying to locate. console#locate switch 2 The blue LED locator light on switch 2 blinks for 20 seconds. The length of time the LED continues to blink can be changed using the locate time command, where ssss is the time in seconds for the LED to blink. 2.5.1.
Use the freed cables or new cables to connect the two switches together that were previously adjacent to the switch being removed. Since there are multiple interface and cable types supported for stacking, a different set of cables may be required to make this connection.
2.5.1.3 Removing units from the stack configuration Once a unit has been physically removed, it may be desired to remove it from the configuration as well, if there are no plans on restoring it later. For the above example Unit 2 was removed; however, the show switch command continues to show the removed switch in the configuration. Issuing a no member command in Stack Configuration mode deletes the removed switch from the configured stack information.
Type the switch renumber command to change the number of the switch. console(config)#switch 3 The switch will be reset and the configuration of Are you sure you want to renumber 2 to perform unit renumbering switch interfaces will be cleared. renumber? (y/n)y Allow a few minutes for the switch to renumber, then enter the following.
The plugged-in model will always be the actual model that is inserted as the stack unit. The preconfigured model must be changed to show the same. To sync the two, the Unit must be temporarily removed from the stack then added back. For this example, use the following steps to sync Unit 3. Note: This example is for a Ring topology only and will not work on a Daisy Chain topology. 1. Save the configuration of the stack by using the “write” command. 2.
5. Apply power back to the mismatched stack member. Allow the member a few minutes to load and join back into the stack. Run show switch again to validate the configurations are in sync. console#show switch SW --2 3 4 2.5.1.6 Management Status ---------Stack Mbr Stack Mbr Mgmt Sw Standby Status --------Oper Stby Preconfig Model ID ------------N4032F N4032 N4064 Plugged-in Model ID ------------N4032F N4032 N4064 Switch Status ------------OK OK OK Code Version --------6.0.1.3 6.0.1.3 6.0.1.
Note: Clicking the Configured Stack-mode column header brings all Stack ports to the top of the list. Look for interfaces shown to be in Stack mode. The logical stack unit number is represented in each interface name. The first number in the interface name is the logical stack unit. For example, interface Te2/1/1 is in logical unit number 2. A blinking LED light is generated on the back of each Dell N4xxx to locate each physical unit.
Note: Do not remove or re-route stacking cables until prompted in the last step below. Do not remove or re-route stacking cables until prompted. Disconnect all other links on the member to be removed and re-route the traffic that was going through this unit so it now goes through the ports that were prepared on the remaining stack unit members. Caution: In the next step, be sure to keep track of all ports being used for stacking that will be temporarily unplugged.
1 3 5 7 9 11 13 15 17 19 21 LNK 23 ACT 2 4 6 8 10 12 14 16 18 20 22 24 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 LNK 47 ACT 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 5 7 LNK 2 4 6 1 SWITCH REMOVED FROM STACK Figure 16 3 9 11 13 15 17 19 21 ACT LNK 1 LNK 23 ACT LNK 2 4 6 8 10 12 14 16 18 20 22 ACT 2 24 ACT 10GBASE-T MODULE Removal of a stack unit Removal of a stack unit
To remove the switch from the configuration, go to the System > Stack Management > Unit Configuration screen. Select the Switch ID of the switch to be removed, and then check the Remove Switch check box at the bottom of the bottom of the screen. Clicking Apply deletes the removed switch from the stack configuration on the running-configuration. Go back to the Stack Summary screen to verify the switch was removed.
2.5.2.4 Renumbering stack units It may be desired to renumber switches in the stack. While this feature may be used at any time, it is most commonly preferred after a switch has been removed and the remaining unit numbers are no longer consecutive. While inconsecutive numbers or numbers out of order do not impact traffic in any way, keeping an order can help quickly identify individual units and ports. A switch may be any number between 1 and 6 as long as it is not already taken by another switch.
Caution: Renumbering stack units may hinder management scripts and monitoring software that collect data based on particular port traffic. Use renumbering with caution when monitoring ports and be aware that scripts and monitoring parameters may need to be modified. In this example there are two stack members numbered 1 and 3. To change Unit 3 to Unit 2, go to System > Stack Management > Unit Configuration and select the Switch ID to change.
The switch that was Unit 3 is now Unit 2 and Unit 3 is no longer present. To remove it from the list, select it on the System > Stack Management > Unit Configuration screen, and then select the Remove Switch check box at the bottom. Clicking Apply deletes the removed switch from the stack configuration on the running-configuration. Go back to the Stack Summary screen to verify the switch was removed.
2.5.2.5 Config Mismatch When a “Config Mismatch” occurs in the Switch Status column, it indicates inconsistent entries in the “Preconfigured Model Identifier” and “Plugged-in Model Identifier” columns. This usually occurs after a renumbering of units when the stack consists of multiple models, but can also occur after a new unit is added to the stack when the stack has been preconfigured to expect a different switch model.
2.5.2.6 Configuring a removed unit to be stand-alone After removing a unit from a stack, the unit converts back to its original configuration that it had prior to adding it to the stack, retaining the IP address, VLAN, LAG, and other settings, allowing the user to immediately access the stand-alone through the out-of-band port if previously configured. Note: A duplicate address could occur on the network if another device was assigned the IP address of the device while it was a stack member.
3 Managing the standby unit If the Master unit fails or is taken offline, a Standby unit automatically takes place as Master. During this time there is no more than a 50 ms interruption in unicast connectivity. The commands below are used to manage your Standy unit. 3.1.1 Command-line interface method To find out which unit is currently in Standby status, run the show switch command.
console(config)#stack console(config-stack)#initiate failover The system has unsaved changes. Would you like to save them now? (y/n) y Configuration Saved! Management unit will be reloaded. Are you sure you want to failover to the backup unit? (y/n) y If using the CLI through the console port, the session will no longer be valid. The console port on the new Master switch will now be activated for management. The IP and MAC addresses of the stack will remain the same with the new Master.
Select the Switch ID to be assigned as the Standby. Check the “Configure as Standby” checkbox. Note: Assigning the current Stack Master to the Standby role is not allowed using this method and results in an error. Instead, use the initiate failover method described below. Click Apply. Copying the running-configuration to the startup-configuration is not required. This setting will be persistent after a reload. Verify the new Standby switch from the Stack Summary screen.
If using the CLI through the console port, the session will no longer be valid. The console port on the new Master switch will now be activated for management. The IP and MAC addresses of the stack remains the same with the new Master. If using a telnet session, log in using the same IP address as before.
4 Appendix A – Commands used in this document All CLI commands used in this document, along with a brief description, are listed below in order of appearance. Some commands are used multiple times in the document.
5 Appendix B - Network switch versions Version information for the network switches used in creating this document are as follows: Network switch Software version Dell N4000 6.0.0.8, 6.0.1.1, 6.0.1.3 About Dell Dell is a leading technology provider to commercial and public enterprises around the world.
