Cisco Broadband Local Integrated Services Solution Release 1.5 Troubleshooting Guide September 15, 2004 Corporate Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.
THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
C O N T E N T S About this Guide Audience xi xi Assumptions xi Document Organization xii Cisco Documentation Suite Document Conventions References xiii xiv xv Obtaining Documentation Cisco.
Contents Solution Features 1-25 Network Features 1-26 Route Selection 1-31 Subscriber Features 1-35 Operation, Administration, Maintenance, and Provisioning Features Cisco NMS/OSS for Broadband Services CHAPTER 2 1-48 Service Fulfillment Applications 1-50 Service Assurance Applications 1-52 Troubleshooting Overview 2-1 Troubleshooting Basics 2-1 Troubleshooting Strategy 2-2 Detailed Troubleshooting Methodology Preparing Yourself to Troubleshoot Taking Preventive Action Troubleshooting
Contents CHAPTER 4 Troubleshooting with Call Flows Understanding MGCP 4-2 MGCP Transactions Understanding SS7 4-1 4-2 4-10 Processing a Telephone Call ISUP Signaling Messages Call Flow Analysis 4-10 4-11 4-13 MGCP Key Fields 4-13 MGCP Message Correlation Call Flow Problems 4-15 Voice Quality Problems 4-22 Lost or Distorted Audio 4-13 4-23 Voice Network Troubleshooting Procedures CHAPTER 5 Troubleshooting DOCSIS Networks DOCSIS 1.
Contents CHAPTER 6 Troubleshooting MTAs 6-1 Troubleshooting EMTA Provisioning Know The Basics 6-1 Troubleshooting Tools 6-4 Troubleshooting Scenarios Motorola Surfboard 6-8 Major Features 6-8 Physical Interfaces Voice Features 6-1 6-5 6-9 6-9 Signaling, Data, Routing Features Security Features 6-10 Management Features Arris MTAs CHAPTER 7 6-10 6-10 Troubleshooting the CMTS 7-1 Console Connections 7-2 Physical Interfaces Features 6-9 7-3 7-3 Troubleshooting the Cisco uBR7246V
Contents Troubleshooting Serial Lines 8-10 Optical Signal Input/Output Problems Using Bit Error Rate Tests Using Loopback Tests CHAPTER 9 8-13 8-16 Troubleshooting Cisco Media Gateways Media Gateway Management 9-1 MGCP on Cisco IOS Software 9-2 DHCP on Cisco IOS Software Cisco MGX8850 9-1 9-2 9-2 MGX 8850 Diagnostics 9-3 Command Line Interface 9-4 Diagnostic Troubleshooting Troubleshooting Alarms 9-7 9-8 Viewing and Responding to Alarms AXSM Card Controls 9-8 9-11 RPM-PR Card Contr
Contents VISM Alarms 9-36 UNIX Snoop Trace Tool 9-36 Symptoms and Solutions 9-37 VISM Card Did Not Become Active T1/E1 Configuration Mismatch DSP Download Failure 9-37 9-37 9-39 VISM Front Card/Back Card Mismatch 9-39 Cannot Use the cc Command to Access a VISM Card VISM Card Resets Intermittently 9-40 VISM Card Does Not Accept a Firmware Download Echo Is Heard on a Voice Call 9-40 9-40 9-41 VISM Card LEDs Are Not Lighted 9-41 Firmware Does Not See the Card Insert Bit Status As Set Physi
Contents CHAPTER 11 Element Management and MIBs Cisco uBR7246vxr MIBs 11-1 11-1 CISCO-CABLE-SPECTRUM-MIB.my CISCO-DOCS-EXT-MIB.my 11-6 DOCS-CABLE-DEVICE-MIB.my DOCS-IF-MIB.
Contents Cisco Broadband Local Integrated Services Solution Troubleshooting Guide x Version 1
About this Guide This guide describes methods and procedures for troubleshooting the Residential Cable VoIP (RCVoIP) implementation of the Cisco Broadband Local Integrated Service Solution (BLISS) Release 1.5 on broadband networks with the focus on the Cisco equipment used in the solution. It also provides additional information on some of the technology issues related to VoIP over a cable network.
About this Guide Document Organization • DOCSIS Baseline Privacy Plus Interface Specification, SP-BPI+-I08-020301 • PacketCable™ MTA Device Provisioning Specification, PKT-SP-PROV-I03-011221 • PacketCable™ Network-Based Call Signaling Protocol Specification, PKT-SP-EC-MGCP-I04-011221 • PacketCable™ PSTN Gateway Call Signaling Protocol Specification, PKT-SP-TGCP-I02-011221 • PacketCable™ Electronic Surveillance Specification, PKT-SP-ESP-I01-991229.
About this Guide Cisco Documentation Suite Chapter 5 Troubleshooting DOCSIS Networks Provides information on DOCSIS and troubleshooting DOCSIS -related problems, including RF and poor performance problems. Chapter 6 Troubleshooting MTAs Provides some accumulated experience with troubleshooting MTAs. Chapter 7 Troubleshooting the CMTS Provides specific information for troubleshooting the Cisco uBR7246 or the Cisco uBR10012.
About this Guide Document Conventions Cisco MGX8850 Media Gateway documentation, including: • Cisco MGX 8850 (PXM1E/PXM45) Software Configuration Guide • Cisco MGX 8850 (PXM1E/PXM45) Command Reference • Cisco Voice Interworking Services (VISM) Configuration Guide and Command Reference Cisco uBR7246VXR Cable Modem Termination System documentation, including: • Cisco CMTS Feature Guide • Cisco Broadband Cable Command Reference Guide Document Conventions Command descriptions use the following conven
About this Guide References Caution Caution means reader be careful. In this situation, you might do something that could result in equipment damage or loss of data. Warnings use the following conventions: Warning Warning means danger. You are in a situation that could cause bodily injury. Before you work on any equipment, be aware of the hazards involved with electrical circuitry and be familiar with standard practices for preventing accidents.
About this Guide Obtaining Documentation Obtaining Documentation Cisco documentation and additional literature are available on Cisco.com. Cisco also provides several ways to obtain technical assistance and other technical resources. These sections explain how to obtain technical information from Cisco Systems. Cisco.com You can access the most current Cisco documentation at this URL: http://www.cisco.com/univercd/home/home.htm You can access the Cisco website at this URL: http://www.cisco.
About this Guide Obtaining Technical Assistance Obtaining Technical Assistance For all customers, partners, resellers, and distributors who hold valid Cisco service contracts, Cisco Technical Support provides 24-hour-a-day, award-winning technical assistance. The Cisco Technical Support Website on Cisco.com features extensive online support resources. In addition, Cisco Technical Assistance Center (TAC) engineers provide telephone support.
About this Guide Obtaining Additional Publications and Information Severity 2 (S2)—Operation of an existing network is severely degraded, or significant aspects of your business operation are negatively affected by inadequate performance of Cisco products. You and Cisco will commit full-time resources during normal business hours to resolve the situation. Severity 3 (S3)—Operational performance of your network is impaired, but most business operations remain functional.
C H A P T E R 1 Solution Overview The best possible troubleshooting tool is an extensive, in-depth understanding of the solution and all components; their installation, configuration, provisioning, operation, and maintenance in your environment. PacketCable™ is a CableLabs® -led initiative aimed at developing interoperable interface specifications for delivering advanced, real-time multimedia services over two-way cable plant.
Chapter 1 Solution Overview Architectural Overview • Support for additional resources, such as: – Enhanced compliance with CableLabs PacketCable specifications – Call Agent support for Event Messaging and DQoS – Announcement/IVR Server – Voice Mail Server – CALEA Server – Record Keeping Server (RKS) – Full Element Management layer – Flow through MTA provisioning – Fault Management This chapter provides an overview of the Cisco Broadband Local Integrated Services Solution (BLISS) Release 1.
Chapter 1 Solution Overview Architectural Overview Architectural Features The BLISS solution provides the following architectural features: • Link redundancy for traffic • Call agent redundancy • Catalyst switch redundancy • Trunk redundancy on TGW failure • SS7 load sharing • CMTS uplink load sharing • TGW WAN link load sharing • Overload call control (call gapping) • Multi-city support • Multiple dial plans • 911/0+ dialing Operational Features The BLISS solution provides the follo
Chapter 1 Solution Overview Architectural Overview • Diffserv Codepoint (DSCP) for Signalling • Diffserv Codepoint (DSCP) or TOS for Bearer • LLQ • MGCP 1.0 • DOCSIS 1.1 • NCS 1.0 • Q.
Chapter 1 Solution Overview Architectural Overview Management Features • Remote software upgrades/management/control • Non-service affecting upgrades (Call Agent) • CORBA • Support the registering of devices and pushing of device configurations via a provisioning API • Support assignment of proper IP addresses to CPE equipment based on type (via DHCP) • Support web-based GUI provisioning for the Cisco BTS10200 Call Agent Backbone Connectivity Features • Packet over SONET (POS) Trunking Gatew
Chapter 1 Solution Overview Architectural Overview Functional Architecture The generic Cisco BLISS for Cable solution architecture consists of multiple functional planes, as illustrated in Figure 1-1: • Customer Premise Equipment (CPE) layer—Includes MTA equipment and access gateways • Aggregation layer—Includes the CMTS, which aggregates traffic from all of the CPE uplinks • Core Switching layer—Provides the IP core network (packet backbone) • Trunking layer—Provides the off-net PSTN and Internet
Chapter 1 Solution Overview Architectural Overview Physical Architecture The BLISS for Cable solution architecture, as illustrated in Figure 1-2, is based on the CableLabs® PacketCable™ 1.0 architecture. The BLISS for Cable solution utilizes the CableLabs® DOCSIS™ 1.1 HFC access network architecture along with the Multi-Service over Cable (MSOC) [1] backbone architecture.
Chapter 1 Solution Overview Solution Components Solution Components The service provider’s SuperPoP/Regional Office hosts all servers needed to provide solution services. Trunking gateways are also typically located here.
Chapter 1 Solution Overview Solution Components Table 1-2 Cisco NMS/OSS for Broadband Services Components Component Version Cisco Broadband Access Center for Cable (BAC-C) 2.5 Cisco Network Registrar (CNR) 6 Cisco Resource Manager Essentials (RME) 3.5 Cisco Broadband Troubleshooter (CBT) 3.0 Cisco Information Center (CIC) 3.5 Cisco Extensible Provisioning and Operations Manager (EPOM) 1.3 and 1.
Chapter 1 Solution Overview Solution Components Multimedia Terminal Adapters Multimedia Terminal Adapters (MTA) are managed by service providers, but are physically located on the customers premises. MTAs are combined with cable modems to form the primary CPE component of the system that interfaces with the cable and provides the user with both Ethernet and telephone access. The MTA consists of a cable modem section and a telephone interface section.
Chapter 1 Solution Overview Solution Components Hybrid Fiber-Coax (HFC) Network The HFC infrastructure, which is provisioned and maintained by the Multiple Service Operator (MSO), is responsible for providing a high speed link to the headend for multiple households. Voice data packets traversing the BLISS for Cable solution initially pass over a neighborhood cable network referred to as an HFC network.
Chapter 1 Solution Overview Solution Components Diplex Filters—For coax cabling, diplex filters must be installed in the RF path between the cable modem cards in the CMTS and cablemodems and/or STBs. A diplexer has three ports: low, high, and common. The downstream cable attaches to the high port because high frequency signals flow in the downstream direction from the CMTS to cable modems and STBs..
Chapter 1 Solution Overview Solution Components Cable Aggregation Components The cable aggregation components of the network architecture for the BLISS for Cable 1.5 solution are usually located at the HFC headend at the service provider’s POP or IP Central Office (IPCO). HFC Headend The headend is where all downstream signals originate and where all upstream signals terminate. Figure 1-4 shows the headend components included in the BLISS for Cable solution, which are described in the following sections.
Chapter 1 Solution Overview Solution Components Cisco uBR7246VXR CMTS In the BLISS for Cable solution the CMTS is a Cisco Universal Broadband Router, uBR7246VXR, which integrates a CMTS with a Cisco7200 series router allowing it to terminate IP packets on its network interface. This enables the Cisco uBR7246VXR to connect cable modems on the Hybrid Fiber Coaxial (HFC) Cable network using Cisco MCxx cable modem cards.
Chapter 1 Solution Overview Solution Components • IF-to-RF Upconverters—To be compatible with cable television system frequency division multiplexing (FDM), install an external IF-to-RF upconverter that translates the IF signal to RF carrier frequency. The upconverter also allows you to maintain your existing channel lineup. Upconverters are available from many manufacturers and can be found in configurations ranging from a fixed number of ports to flexible multislot, multiport models.
Chapter 1 Solution Overview Solution Components Edge Routers Cisco GSR 12000 (optional)—the Cisco GSR 12000 Gigabit Switch Router can be used to connect CMTSs to the rest of the IP network. It can be located in a headend, in the regional backbone, or in a service providers Internet Protocol Central Office (IPCO). Trunking Components The Media Gateway Control Protocol (MGCP) 1.0 is used to control Voice over IP (VoIP) calls by external call-control elements known as call agents (CAs).
Chapter 1 Solution Overview Solution Components The Cisco MGX 8850 provides the interface to a 911 tandem, or Public Safety Answering Point (PSAP), and Operator Services by means of Feature Group D (FGD) trunks. All SS7-controlled bearer channels (Inter-Machine Trunks or IMTs) from the Public Switched Telephone Network (PSTN) terminate here.
Chapter 1 Solution Overview Solution Components The SRM/E card is another high-density bulk distribution card that removes some restrictions of the SRM card. These cards support both ANSI and ITU-T interfaces of these optical and electrical options: • opticalOptical: OC-1/OC-3, STM-0/STM-1 with APS • electricalElectrical: STS-0/STS-1, STM-0, STM-1 The VISM modules create VoIP packets which are transported out the Cisco MGX88500 via the ATM uplink on the PXM backcard.
Chapter 1 Solution Overview Solution Components Call Control Components The Cisco BLISS for Cable solution relies on interoperability with partner systems for various functions that are not directly supported. The partner systems include multimedia terminal adapters (MTAs), announcment systems, interactive voice response systems, voice mail systems, billing mediation systems and electronic surveillance systems.
Chapter 1 Solution Overview Solution Components Cisco BTS 10200 Capabilities At a high level, the Cisco BTS10200 Softswitch provides the following capabilities: • Support for MGCP call control with the TGWs and NCS signaling on the MTA side. • Call signaling capabilities including SS7/ISUP interactions with the PSTN SS7 network, which includes support for SCP database dip applications like 800 number and LNP services.
Chapter 1 Solution Overview Solution Components Figure 1-6 Cisco BTS 10200 Softswitch Alarm panel, PDU power feed 8A Filler panel PDU (DC systems) 1U 1U 1U 2U Filler panel ON APP R ALRM YES NO OFF CONS EMS B (AXi) CCN node address = 1 PDU power feed 1A/1B PW 4U PW R R ON APP ALRM OFF CONS YES NO Call agent B (AXmp) CCN node address = 2 PDU power feed 2A/2B 6U Filler panel (Expansion CA or Ext. FS) 6U 2.36 m 93 in.
Chapter 1 Solution Overview Solution Components Signaling System 7 (SS7) Links The SS7 links provides an interface between the Call Agent and the SS7 network. The signaling links carrying SS7 messages terminate on signaling interfaces in the Call Agent.
Chapter 1 Solution Overview Solution Components In terms of billing for on-net versus off-net calls, different BAF structures are generated between on-net calls, which are defined as calls completed within the service providers network, and off-net calls, which are defined as calls completed outside the service providers network, such as PSTN calls. RADIUS—is the protocol defined in Internet Engineering Task Force (IETF) Request for Comments (RFC) 2138, Remote Authentication Dial In User Service (RADIUS).
Chapter 1 Solution Overview Solution Components CALEA Server The Cisco BTS 10200 supports the Communications Assistance for Law Enforcement Act (CALEA). • The Cisco BTS 10200 provides the PacketCable EMS/RADIUS interface for the transmission of call identifying information (call data) to the CALEA delivery function (DF) server. The BTS 10200 implementation is independent of the access network application, such as packet cable or T1.
Chapter 1 Solution Overview Solution Features MTA Provisioning The Cisco Subscriber Registration Center (CSRC) v1.5 [with Cisco Network Registrar (CNR) 5.0] provides a specialized provisioning environment using CSRC component products that help you automate subscriber service provisioning, such as VoIP, e-mail, Web access, and configuration of subscriber MTAs, including DOCSIS-compliant MTAs. CSRC 1.5 works in conjunction with CNR to provide DHCP and TFTP services.
Chapter 1 Solution Overview Solution Features Network Features The network features supported by the Cisco BTS 10200 Softswitch include numbering plan and dialing procedures, tandem service, regulatory features and other network features. Some of these features are defined in Telcordia LSSGR documents. In general, Cisco BTS 10200 Softswitch features delivered via gateway clients behave identically to their PSTN counterparts.
Chapter 1 Solution Overview Solution Features Information Service Calls (900 and 976) Information service calls (ISC) provide a variety of announcement-related services on a national or local basis. The two general categories of this service are Public announcement services (PAS)-Weather, sports, horoscope, and so forth, and Media stimulated calling (MSC)-Telephone voting, radio station call-ins, and so forth. National calls are dialed as 1-900-xxx-xxxx and local calls are dialed as NPA-976-xxxx.
Chapter 1 Solution Overview Solution Features The routing of the call can vary depending on the arrangements made between the toll-free subscriber and the network service provider. These arrangements can include selective routing based on the time of day, day of week, and location from which the call originates. Vertical Service Codes (*XX) Vertical service codes (VSC) allow subscribers to activate and deactivate services from their own station.
Chapter 1 Solution Overview Solution Features ANI Screening Automatic number identification (ANI) is used for long distance access service. The ANI screening feature validates the ANI on incoming feature group D calls from the PSTN before routing. All ANIs supported by the Cisco BTS 10200 Softswitch are stored in the feature server database. If an ANI is not available, or does not appear in the feature server ANI table, the TG default data is checked to see if casual calls are allowed.
Chapter 1 Solution Overview Solution Features NPA Split Support When DNs are exhausted within an NPA, an additional NPA is assigned to the region. The new NPA may be allocated as an overlay over the existing NPA, in which case there is no major impact to the Cisco BTS 10200 Softswitch. However, when the new NPA is assigned based on a geographical split of the region, there are significant impacts. The assignment of the new NPA based on a geographical split is referred to as split-NPA.
Chapter 1 Solution Overview Solution Features Trunk Testing Trunk testing is used to determine the transmission quality of the shared trunks that interconnect switching systems. Trunk testing is extremely important in monitoring system health, because it is the only practical way to objectively determine the performance of individual trunks.
Chapter 1 Solution Overview Solution Features Figure 1-7 Route Selection Flowchart Subscriber Office code DN2Subscriber Subscriber Termination Allows routing to an alternate number Termination Route Policy NXX Subscriber or Trunk-Group Translated DN and/or carrier IntraLATA, N11, operator, route Route guide For example: Policy TOD, Policy ODR, Policy Prefix...
Chapter 1 Solution Overview Solution Features • Percentage-Based Routing, page 1-34 • Route ID, page 1-34 Least-Cost Routing With least-cost routing (LCR), the least expensive Trunk Group (TG) in a route is chosen. The TGs in a list may be provisioned in any order. The call processing function will find the relative cost of each TG in the route from the TG table and order them from least to most expensive. The least expensive idle trunk will be selected from this newly ordered TG list.
Chapter 1 Solution Overview Solution Features Percentage-Based Routing Percentage-based routing allows call distribution of a specified percentage of calls to different route groups. The percentage ranges can be specified to a granularity of 1 percent, and the ranges should cover a total of 100 percent. A random number generator is used to sort calls into the specified percentage ranges, and the percentage range is used to route each call.
Chapter 1 Solution Overview Solution Features Trunk Selection Policies in the TG Trunk selection policies are applied to each TG, and allow the service provider to control the assignment of trunks. The CA can be provisioned to select trunks in ascending order (default), select trunks in descending order, select only even numbered trunks, select only odd numbered trunks or select the least recently used trunk.
Chapter 1 Solution Overview Solution Features Remote Activation of Call Forwarding (RACF) Remote activation of call forwarding (RACF) permits the user to control their CFU functions when they are away from their phone. The service provider sets up this function for the user, and designates a DN the user should call to access IVR functions that control the RACF feature. Once set up for RACF, the user can activate CFU, deactivate CFU, and change the target DN of CFU from a remote station.
Chapter 1 Solution Overview Solution Features Calling Number Delivery Blocking (CNDB) CNDB allows the caller to control the status of their caller number privacy on a per-call basis. For all new calls, the privacy status reverts back to the PPS. Calling Name Delivery Blocking (CNAB) CNAB allows the caller to control the status of their caller name privacy on a per-call basis. For all new calls, the privacy status reverts back to the PPS.
Chapter 1 Solution Overview Solution Features Casual Call Restrictions (101XXXX) The casual call restrictions are used to allow/restrict calls dialed with a casual code prefix (101XXX).
Chapter 1 Solution Overview Solution Features Other Restrictions The service provider can provision blocking or unblocking of any or all of the following services based on customer requests: • Block 900 Calls—allows all calls of the form 1-900-XXX-XXXX to be blocked. • Block 976 Calls—allows all calls of the form 976-XXXX or NPA-976-XXXX to be blocked. • Block DA Calls—allows all calls of the form 411, 1+411 or NPA-976-XXXX to be blocked.
Chapter 1 Solution Overview Solution Features Number Blocking Number blocking prevents certain types of calls from being completed from a particular line or station. When the caller encounters a call that is blocked, the caller can receive any of several blocking treatments such as reorder tone, announcement or routing to an attendant. Number blocking is activated/deactivated and administered by the service provider on a per line or per group of lines basis.
Chapter 1 Solution Overview Solution Features Automatic Callback (AC) – Repeat Dialing AC, also called repeat dialing, allows the user to request the system to automatically call the most recently dialed number. The system will keep attempting to call the number for up to 30 minutes, and the remote station will be rung automatically when the called party becomes idle. The system alerts the user with distinctive ringing. Up to 20 AC requests can be active at any time.
Chapter 1 Solution Overview Solution Features Call Transfer (CT) CT allows a user to add a third party or second call to an existing two-party call. CT also allows the user to hang up while involved in the two calls and connect the remaining two parties in a two-way connection.
Chapter 1 Solution Overview Solution Features Multiple Directory Numbers (MDN) Multiple directory numbers (MDN) service is also known as teen service. It enables one primary DN and one or two secondary DNs to be assigned to a single line termination. A specific unique ringing pattern is assigned to each DN, so that each incoming call can be individually identified. A distinctive CW tone is also assigned to each DN so that each incoming call can be individually identified when the line is busy.
Chapter 1 Solution Overview Solution Features Selective Call Forwarding (SCF) The selective call forwarding (SCF) feature screens each incoming call to determine whether the DN is on a list of DNs, provisioned by the user (called party), to receive automatic forwarding treatment. The user also sets the forward-to number. Any incoming calls from DNs that are on the SCF screening list are forwarded to the designated number.
Chapter 1 Solution Overview Solution Features Usage-Sensitive Three-Way Calling (USTWC) USTWC allows a user to add a third party to an existing two party conversation. It provides all the functionality of TWC without requiring the user to subscribe to the service. The service provider may charge differently for the use of this service. The usage-sensitive features can be enabled/inhibited per user by turning on/off the usage-sensitive option for the user.
Chapter 1 Solution Overview Solution Features Operation, Administration, Maintenance, and Provisioning Features Cable Management and Services are at the application layer. The two most widely used are the Simple Network Management Protocol (SNMP) along with Cisco Network Registrar (CNR) described earlier. SNMP is a part of the Transmission Control Protocol/Internet Protocol (TCP/IP) protocol suite and it facilitates the exchange of management information between network devices.
Chapter 1 Solution Overview Solution Features • Alarm Generation—The Cisco BTS 10200 host generates autonomous messages to notify the operator of network problems and conditions. Alarms are stored on the Cisco BTS 10200 host and may be viewed using CLI commands and/or specialized tools. At user discretion, the autonomous alarm message stream can be addressed to single or multiple destinations, such as CMNM, HP Open View, or other third party SNMP managers.
Chapter 1 Solution Overview Cisco NMS/OSS for Broadband Services Cisco NMS/OSS for Broadband Services The Cisco NMS/OSS for Broadband Services solution is an optional offering that provides a complete end-to-end OSS solution for architectures based on PacketCable and DOCSIS standards.
Chapter 1 Solution Overview Cisco NMS/OSS for Broadband Services As shown in Figure 1-8, the Cisco NMS/OSS for Broadband Services encompasses three specific areas of network management applications for the cable operator: • Service Fulfillment Applications, page 1-50 • Service Assurance Applications, page 1-52 Figure 1-8 Cisco NMS/OSS for Broadband Services Fulfillment Cisco Broadband Access Center Flow-Through Provisioning Cisco Packet Telephony Center Network Topology and Configuration Cisco Resou
Chapter 1 Solution Overview Cisco NMS/OSS for Broadband Services Service Fulfillment Applications The following sections provide a high-level overview of the functions performed by the components of the Cisco NMS/OSS for Broadband Services that help cable operators enable service to customers: • Cisco Broadband Access Center for Cable, page 1-50 – Cisco Network Registrar, page 1-51 – Cisco CNS Address and Name Registrar, page 1-51 • Cisco Resource Manager Essentials, page 1-51 Cisco Broadband Access
Chapter 1 Solution Overview Cisco NMS/OSS for Broadband Services Cisco Network Registrar Cisco CNS Network Registrar, through its carrier-class performance (in both scalability and reliability) and advanced provisioning, simplifies IP address management.
Chapter 1 Solution Overview Cisco NMS/OSS for Broadband Services Service Assurance Applications The Cisco NMS/OSS for Broadband Services Service Assurance package helps cable operators monitor and troubleshoot their networks in a proactive manner. The Cisco NMS/OSS Service Assurance package includes the following components.
Chapter 1 Solution Overview Cisco NMS/OSS for Broadband Services New features added in Cisco Broadband Troubleshooter version 2.
Chapter 1 Solution Overview Cisco NMS/OSS for Broadband Services EMS Communications Operators, network administrators, and end users can communicate with the EMS from their workstation or PC over the interfaces shown in Figure 1-10.
Chapter 1 Solution Overview Cisco NMS/OSS for Broadband Services • Secure File Transfer Protocol (SFTP)—For bulk provisioning sessions. SSH and SFTP are always available on the Cisco BTS 10200 Softswitch, and there is no command to turn them off. The user can temporarily enable Telnet or FTP (or both); however, this can create security issues. Caution • Cisco strongly recommends that you use secure interfaces only.
Chapter 1 Solution Overview Cisco NMS/OSS for Broadband Services Cisco Broadband Local Integrated Services Solution Troubleshooting Guide 1-56 OL-5169-01
C H A P T E R 2 Troubleshooting Overview This chapter contains an overview of troubleshooting concepts for the Cisco Broadband Local Integrated Services Solution (BLISS) for Cable 1.5. It provides guidelines for troubleshooting problems on the network, including guidelines for preventing problems before they occur.
Chapter 2 Troubleshooting Overview Troubleshooting Strategy Troubleshooting Strategy Troubleshooting consists of determining the nature of a problem and then isolating the problem to a particular device or component of a device. When a problem has been isolated and identified, troubleshooting also consists of fixing the problem, usually by replacing the device, some component of the device, or changing a setting or variable in the software.
Chapter 2 Troubleshooting Overview Troubleshooting Strategy General Problem-Solving Model The complexity and crucial uptime requirements of modern telephone networks intensify the pressures to solve configuration, connectivity, and performance problems. The best way to approach a network problem is to develop a standard troubleshooting methodology. The problem-solving model presented in Figure 2-1 is one example of such a methodology.
Chapter 2 Troubleshooting Overview Troubleshooting Strategy These steps can be grouped into a small number of troubleshooting phases: • Make sure you have a clear, sufficient definition of the problem. • Gather all the relevant facts and consider the likely possibilities. • Create and implement an action plan for the most likely possibility, then observe the results. • If the symptoms do not stop, repeat the process and try another action plan (or gather more facts).
Chapter 2 Troubleshooting Overview Troubleshooting Strategy Detailed Troubleshooting Methodology The goal of this section is to establish a methodical mindset—an ordered pattern of thought to use when troubleshooting. The model described in this section takes a multistep approach to problem-solving. In the following subsections, we will examine each of these steps in detail to see how it can be used in a troubleshooting example.
Chapter 2 Troubleshooting Overview Troubleshooting Strategy One of the most important outcomes of a systematic troubleshooting approach is to narrow possibilities —remove irrelevant details from the set of items that you need to check. You can eliminate entire classes of problems associated with system software and hardware. You can eliminate several possible causes based on the facts gathered for the problem.
Chapter 2 Troubleshooting Overview Troubleshooting Strategy Step 6: Observe the Results of the Action Plan After manipulating a variable to find a possible solution to a problem, be sure to gather results based on the action plan. Generally, you should use the same method of gathering facts that you used in step 2 of the methodology. After you have analyzed the results, you must determine whether the problem has been resolved.
Chapter 2 Troubleshooting Overview Troubleshooting Strategy Preparing Yourself to Troubleshoot Troubleshooting complex networks, like those implemented in the Cisco BLISS for Cable solution, is somewhat different than troubleshooting other networks. The best way to approach any troubleshooting problem is to isolate the cause, not the symptom. Of course, the only clues you have are the syptoms, but you must use them to synthesize a possible (maybe even probable) cause.
Chapter 2 Troubleshooting Overview Troubleshooting Strategy Taking Preventive Action Monitoring the general health of the network helps avoid problems, and makes troubleshooting easier when they do occur. You have to have normal baseline data to identify when problems occur. Before you do anything that could cause additional problems, take the appropriate precautions.
Chapter 2 Troubleshooting Overview Troubleshooting Strategy Fault Analysis Troubleshooting begins with analyzing the following in the order listed: 1. Alarms and system messages (if present) 2. Call traces and log files (if availabale) 3. Software and system state(s) 4. Signaling links and destinations 5. Bearer channels and destinations 6.
Chapter 2 Troubleshooting Overview Troubleshooting Tools Troubleshooting Tools There are two types of tools available to troubleshoot problems in the Cisco Broadband Local Integrated Service Solution (BLISS) for Cable: • Software Tools, page 2-11 • Hardware Tools, page 2-15 Software Tools This section describes software tools that can help you in troubleshooting.
Chapter 2 Troubleshooting Overview Troubleshooting Tools The results of call traces are signal flow diagrams that you can use for troubleshooting. Call traces are typically used to capture system activity as part of a procedure to clear an alarm. You should have a trace of all normal call functions to use as a baseline to compare to when problems arise.
Chapter 2 Troubleshooting Overview Troubleshooting Tools Commonly Used Show Commands Some of the most commonly used Cisco IOS show commands include: • show interfaces—displays statistics for the following network interfaces – show interfaces ethernet – show interfaces fddi – show interfaces serial • show controller t1—Displays statistics for T1 interface card controllers • show running-config—Displays the router configuration currently running • show startup-config—Displays the router configuration
Chapter 2 Troubleshooting Overview Troubleshooting Tools To minimize the negative impact of using debug commands, follow this procedure: Step 1 Use the no logging console global configuration command on your router. This command disables all logging to the console terminal. Step 2 Telnet to a router port and enter the enable EXEC command. Step 3 Use the terminal monitor command to copy debug command output and system error messages to your current terminal display.
Chapter 2 Troubleshooting Overview Troubleshooting Tools Each outgoing packet can result in one of two error messages: • A "time exceeded" error message indicates that an intermediate router has seen and discarded the probe. • A "port unreachable" error message indicates that the destination node has received the probe and discarded it, because it could not deliver the packet to an application. If the timer goes off before a response comes in, the trace command prints an asterisk (*).
Chapter 2 Troubleshooting Overview Troubleshooting Tools Low-End Cable Test Equipment At the low-technology end of the spectrum of test equipment are volt-ohm meters and digital multimeters. These devices measure parameters such as AC and DC voltage, current, resistance, capacitance, and cable continuity. They can be used to check physical connectivity. Cable testers (that is, scanners) can also be used to check physical connectivity.
Chapter 2 Troubleshooting Overview Troubleshooting Tools Digital Interface Testing Tools Several test tools can be used to measure the discrete digital signals that are present at PCs, modems, printers, and other peripheral interfaces. Examples of this type of test equipment include breakout boxes, fox boxes, and bit/block error rate testers (BERTs/BLERTs). These devices can monitor data line conditions, analyze and trap data, and diagnose problems common to data communication systems.
Chapter 2 Troubleshooting Overview Troubleshooting Tools Protocol Analyzers A protocol analyzer records, interprets, and analyzes how a communication protocol operates in a particular network architecture. It captures frames as they travel across the network.
Chapter 2 Troubleshooting Overview Troubleshooting Tools Network Management Systems As networks grow larger and more complex, there is a greater chance of network failures that can disable the entire network or degrade performance to an unacceptable level. The complexity of such large networks makes the use of automated network management tools a critical factor in efficient management.
Chapter 2 Troubleshooting Overview Troubleshooting Tools Many analyzer vendors offer the capability to export the data from their analyzers into the simulation/modeling tools, thus providing a source of real network data. These simulation/modeling tools allow the network manager to see and test network performance before committing to proposed designs or changes. Cisco Supplied Tools The following tools require user registration for access. The registered sites are noted beside the link.
C H A P T E R 3 Trouble Isolation Procedures This chapter presents comprehensive trouble isolation procedures for the Cisco Broadband Local Integrated Services Solution (BLISS) for Cable.
Chapter 3 Trouble Isolation Procedures Finding System Information Determining the Source of the Problem The first step in troubleshooting your system is determining the source of your problem. There are several network elements connected to the Cisco BTS 10200 Softswitch and to the IP managment network centered on the Catalyst 6509 switch. You will usually be notified of system problems by events and alarms.
Chapter 3 Trouble Isolation Procedures Finding System Information Managing Event and Alarm Reports There are two ways to view events and alarms—by subscribing to event and alarm reports (automatic, real-time) and by retrieving event or alarm summaries from the log files by operator query.
Chapter 3 Trouble Isolation Procedures Finding System Information The elements of the reply for an alarm (or an event) are defined as follows: • Serial Number (ID)—All events and alarms have a unique, system-assigned serial number. • Type—Type is the designated category of the report: Audit, Billing, Callp, Config, Database, Maintenance, Oss, Security, Signaling, Statistics, or System. • Number—Event or alarm numbers are preset in the Cisco BTS 10200. They are not user provisionable.
Chapter 3 Trouble Isolation Procedures Finding System Information Note If the show event-log or show alarm-log commands are issued without any tokens (parameters), all events or alarms of all types and all severities for all components are displayed. Issuing the show event-log or show alarm-log commands with any combination of optional tokens limits the display to the designated subset of events or alarms.
Chapter 3 Trouble Isolation Procedures Finding System Information • If type=event-level, then value designates the severity of the events or alarms to include in the log files, which can be CRITICAL, MAJOR, MINOR, WARN, or INFO. All events or alarms whose severity is equal to or greater than the event level specified are included in the designated event or alarm log file.
Chapter 3 Trouble Isolation Procedures Finding System Information The total number of occurrences of the designated event or alarm message is determined by multiplying these two values (threshold X throttle). The system maximum number of occurrences of an event or alarm that can be reported in any 30-minute interval is 100x100 or 10,000.
Chapter 3 Trouble Isolation Procedures Troubleshooting Hardware Components Troubleshooting Hardware Components Hardware troubleshooting involves making sure all the Cisco BLISS for Cable solution components are powered on, properly connected, and communicating with one another and with the Public Switched Telephone Network (PSTN). The following sections cover procedures for isolating and remedying physical layer problems.
Chapter 3 Trouble Isolation Procedures Troubleshooting Hardware Components Step 3 Test bearer trunk connectivity. Bearer trunks carry traffic from end-users dialed into the PSTN to cable modems on the CMTS. After you have verified that you can successfully ping the CMTS, verify bearer trunk connectivity between the MGW and the PSTN.
Chapter 3 Trouble Isolation Procedures Call Traces Call Traces Traces are records of the message flows through the Cisco BTS 10200 Softswitch. Traces are useful when problem calls are reaching the Cisco BTS 10200 Softswitch basic call module (BCM). If a trace is empty, the call has not reached the BCM. A problem call can be readily identified through the presence of non-idle bearer circuits entries in the debug logs.
Chapter 3 Trouble Isolation Procedures Component States A status report for the media gateway lists both the administrative state and operational state of the link to the media gateway as well as the operational state of the gateway.
Chapter 3 Trouble Isolation Procedures Component States Signaling destination problems can result from any of the following: • SS7 traffic restart handling • SS7 STP problems • Configuration problems • Software problems • Bearer Channels Bearer channels are at the core of the solution. The goal is to ensure that a bearer channel successfully communicates between two endpoints. The state of the bearer channels is often a good indicator as to the health of the overall system.
Chapter 3 Trouble Isolation Procedures Component States • AGGR CONNECTING—the connection to the CMTS is being set up. • AGGR INITIALIZING—identifies the initial state of the CMTS before the Call Agent (CA) attempts to connect to it. This is a transitional state, which a user may rarely see. Media Gateway This example describes how to check the status of a media gateway.
Chapter 3 Trouble Isolation Procedures Component States The rules for placing an MGW into the OOS, INS, and MAINT states are shown in Figure 3-1.
Chapter 3 Trouble Isolation Procedures Component States Table 3-2 CMTS and Subscriber Termination States CMTS State Allowed Subscriber Termination States OOS OOS UEQP INS OOS MAINT INS UEQP MAINT OOS MAINT UEQP where: • OOS—out of service • MAINT—maintenance • INS—in service • UEQP—unequipped The rules for placing subscriber terminations into the OOS, INS, and MAINT states (which depend upon the CMTS state) are shown in Figure 3-2.
Chapter 3 Trouble Isolation Procedures Component States Administrative and Operational Maintenance States for CMTS If SL = NE SL = UEQP P_INS (SL) Add (SL)* If SL = UEQP Del (SL) Del (SL)* RGW = OOS SL = NP/UEQP P_OOS (SL) Warning Warning Warning If SL = NE SL = UEQP P_INS (SL) If SL = INS warn else { {if SL = OOS discover SL*2} init SL*3, SL = INS} If SL = NE SL = UEQP P_INS (SL) P_OOS = Place out of service P_INS = Place in service P_MNT = Place maintenance NP = Not provisioned SL = Susc
Chapter 3 Trouble Isolation Procedures Component States Status Command This section describes how to show the status of subscriber terminations.
Chapter 3 Trouble Isolation Procedures Component States Administrative State Token The Administrative State (admin-state) token returns the administrative state of the subscriber termination. Valid values are: • UEQP—Unequipped; resource is not commissioned. Resource is not registered. • OOS—Termination was manually controlled out of service. • INSQ—Termination was manually controlled in service, but operationally may be available or unavailable.
Chapter 3 Trouble Isolation Procedures Component States Source Token The source token specifies whether to query the Call Agent, or the EMS, for status information. It is an optional token. Valid values for the source token are: • EMS (Default)—Query the local EMS database for most current status. • AGENT—Query the remote Call Agent database for most current status.
Chapter 3 Trouble Isolation Procedures Component States Equip Command The equip command changes the administrative state of terminations that are in the UEQP state to OOS state. It ignores the terminations in the states INS, MAINT, or OOS. Step 1 Use the following example to equip a subscriber termination: CLI> equip subscriber-termination id=97_8@ipclab.cisco.com; Reply : Success: CLI change successful ID -> Subcriber ID -> 97_8@ipclab.cisco.
Chapter 3 Trouble Isolation Procedures Component States Status Command The following example shows the status of a single TG ID: CLI> status trunk-grp id=2; RESULT -> ADM configure result in success REASON -> ADM executed successful ADMIN STATE -> ADMIN_INS OPER STATE -> Trunk group in-service TGN ID -> 2 Table 3-4 lists the administrative states the system can return. Table 3-4 Example Returnable Administrative States State Definition ADMIN-INS In Service. ADMIN-OOS Out of Service.
Chapter 3 Trouble Isolation Procedures Component States • Trunk group restore establish request maintenance • Trunk group restore establish fail normal • Trunk group restore establish fail switch-over • Trunk group restore establish fail maintenance • Trunk group in maintenance state • Trunk group down session set fail soft normal • Trunk group down session set fail hard normal • Trunk group down session set fail soft maintenance • Trunk group down session set fail hard maintenance • T
Chapter 3 Trouble Isolation Procedures Component States Trunk Termination This section describes the status and control commands for trunk terminations. Either a range (for example, cic=1-24;) or a single value (for example, cic=1;) for the CIC parameter can be specified for the status and control of trunk terminations. Individual trunks and trunking groups can be placed into any of three administrative service states: INS, OOS, and MAINT.
Chapter 3 Trouble Isolation Procedures Component States Administrative and Operational Maintenance States for a Trunking Gateway Add (TGP) TGP = OOS Add (TRK, TGP) TGW = OOS TGP = NP/OOS TRK = NP/OOS if TGP = OOS add (TRK) TRK = OOS if (TRK, TGP) = OOS del (TRK, TGP) if TGP = OOS del (TGP) Del(TGP) if TGP=OOS/INS TGP=MNT all TRK=MNT bearer=MNT signaling=MNT Del (TRK, TGP) P_OOS = Place out of service P_INS = Place in service P_MNT = Place maintenance NP = Not provisioned Add (TGP) TGP=OOS P_
Chapter 3 Trouble Isolation Procedures Component States Status Command This section describes how to check trunk termination status. This section is organized as follows: • Trunk Termination Status—Basic Command • Trunk Termination Status—Optional Tokens • Trunk Termination Status—”status tt” Command Trunk Termination Status—Basic Command Note This command can be executed for one CIC (for example, cic=1;), a range of CICs (for example, cic=1-12;) or for all CICs (cic=all;).
Chapter 3 Trouble Isolation Procedures Component States Trunk Termination Status—Optional Tokens The following optional tokens can be used with the status trunk-termination command. They expand the range of useful information returned. Either all, or none of the tokens can be used, with the exception of the off-normal token, which must be used by itself (without any other tokens). • Administrative State (admin-state). Valid values are: – UEQP—Unequipped; resource is not commissioned.
Chapter 3 Trouble Isolation Procedures Component States • Static State (static-state). Valid values for the static-state token are: – UEQP—Unequipped resource is not commissioned. Resource is not registered. – LBLK—Termination is locally blocked: either manually taken OOS/MAINT (block reason can be MANUAL-OOS, MAINT-OOS), or automatically went out of service. – RBLK—Termination is remotely blocked (blocked by remote side). – ACTV—Available. – All—Returns all possible static states.
Chapter 3 Trouble Isolation Procedures Component States Table 3-7 Valid Normal Trunk Termination States State/Token ADMIN-STATE OPER-STATE STATIC-STATE DYNAMIC-STATE UNEQP UNEQP ANY UEQP IDLE MANUALLY OOS OOS ANY LBLK IDLE MANUALLY MAIN MAINT IDLE LBLK IDLE IDLE INS IDLE ACTV IDLE ACTIVE INCOMING INS IDLE ACTV IDLE ACTIVE OUTGOING INS ACTIVE ACTV OBSY TRANSIENT INCOMING INS ACTIVE ACTV IBY-TRNS TRANSIENT OUTGOING INS BUSY ACTV OBSY-TRNS Trunk Termination Sta
Chapter 3 Trouble Isolation Procedures Component States Control Command This section describes how to control trunk terminations.
Chapter 3 Trouble Isolation Procedures Component States Reset Command The reset command clears all manual and blocked states as well as any active/transient calls on a trunk termination, with the exception of SS7 trunk terminations. It brings a trunk into INS mode.
Chapter 3 Trouble Isolation Procedures Component States Equip Command The equip command changes the administrative state of terminations that are in the UEQP state to the OOS state. It ignores the terminations in the INS, MAINT, or OOS states. Use the equip command or the control trunk-grp command to change the termination state for a subscriber to a trunk. The control trunk-grp command changes all trunks in the specified trunk group to the specified state.
Chapter 3 Trouble Isolation Procedures Component States Unequip Command The unequip command changes the administrative state of terminations that are in the OOS state into the UEQP state. It ignores the terminations in the INS, MAINT, or UEQP states.
C H A P T E R 4 Troubleshooting with Call Flows This chapter suggests ways to use call flows to help diagnose network problems. It includes the following sections: • Understanding MGCP, page 4-2 • Understanding SS7, page 4-10 • Call Flow Analysis, page 4-13 • Voice Quality Problems, page 4-22 Making a phone call on a VoIP over cable network is a complex process that involves many software and hardware components.
Chapter 4 Troubleshooting with Call Flows Understanding MGCP Understanding MGCP Media Gateway Control Protocol (MGCP) is the protocol used between the Call Agent and the endpoints within the network to establish, manage, and delete connections. Endpoints are defined by the call topology: • On-net to on-net—both calling and called parties are on the same provider’s network. In this case the endpoints are MTAs.
Chapter 4 Troubleshooting with Call Flows Understanding MGCP MGCP Commands Table 4-1 describes the five commands (also called verbs) used by MGCP to make and terminate connections between endpoints. All commands except Notify are sent by the Call Agent to the gateway. Notify is sent from the gateway (which may also send a DeleteConnection). Table 4-1 MGCP Commands Verb Code Used By Action Notify NTFY Gateway Notifies the call agent of events.
Chapter 4 Troubleshooting with Call Flows Understanding MGCP Call Identifier (C) The call ID identifies the call (or session) to which this connection belongs. This parameter is unique within the network of gateways; however, connections that belong to the same call share the same call ID. The call ID can be used to identify calls for reporting and billing purposes. Connection Identifier (I) The connection ID identifies the connection within the call.
Chapter 4 Troubleshooting with Call Flows Understanding MGCP Connection Mode (M) Each connection is qualified by a connection mode parameter, which can be set to send, receive, send/receive, inactive, loopback, or continuity test. The handling of the audio signals received on these connections is determined by the parameters: Note • Audio signals incoming from connections in receive or send/receive mode are mixed and sent to the endpoint.
Chapter 4 Troubleshooting with Call Flows Understanding MGCP Table 4-4 Requested Events (continued) Event Code Continuity tone co Continuity detected (as a result of a continuity test) cv On-hook transition hu Off-hook transition hd Flash hook hf Digit collection Individual digits (e.g. “#”), timers (“T”), or ranges (e.g. “[0-9]” or “[0-9*#T]”) Each event can be qualified by a requested action or by a list of actions.
Chapter 4 Troubleshooting with Call Flows Understanding MGCP Table 4-5 Signal Requests (continued) Signal Code Dial tone dl Intercept tone it Network congestion tone cg Busy tone bz Confirm tone cf Answer tone aw Call waiting tone wt Off hook warning tone ot Preemption tone pt Continuity tone (default) co Continuity tone (single tone) co1 Continuity test (go tone, in dual tone procedures) co2 Continuity verified (response tone, in dual tone procedures) cv DTMF tones A string
Chapter 4 Troubleshooting with Call Flows Understanding MGCP The list contains the following: • Events that were accumulated (but not notified) • Events that were treated according to the digit map (but not matched yet) • The final event that triggered the detection or provided a final match in the digit map Events that have been accumulated according to the digit map are grouped in a single string.
Chapter 4 Troubleshooting with Call Flows Understanding MGCP Table 4-6 Types of Connection Parameters (continued) Connection Parameter Name Code Value Jitter JI Average inter-packet arrival jitter in milliseconds, expressed as an unsigned integer. Jitter is defined as the mean deviation (smoothed absolute value) of the difference in packet spacing at the receiver compared to the sender for a pair of packets. Detailed computation algorithms are found in RFC 1889.
Chapter 4 Troubleshooting with Call Flows Understanding SS7 Understanding SS7 Connections to the PSTN are done through ISDN user part (ISUP) trunks with a trunk gateway providing the bearer connections, and an SS7 gateway providing the signaling connections into the SS7 network. SS7 provides call setup and teardown, network management, fault resolution, and traffic management services. The SS7 network is used solely for network control, and the only data sent over it is signaling messages.
Chapter 4 Troubleshooting with Call Flows Understanding SS7 Step 8 Any call-progress indications (or other signals that can be carried in-band) are cut through the voice path as soon as an end-to-end audio channel is established.
Chapter 4 Troubleshooting with Call Flows Understanding SS7 Here is a sample call flow over the SS7 network: PSTN Switch SS7 Gateway IAM Call Agent IP Network ACM Call Proceeding CPG Alerting ANM Connect Connect ACK Bearer Connection Established REL Disconnect Release RLC Release Complete Bearer Connection Terminated Cisco Broadband Local Integrated Services Solution Troubleshooting Guide 4-12 OL-5169-01
Chapter 4 Troubleshooting with Call Flows Call Flow Analysis Call Flow Analysis The following sections discuss how to use MGCP and SS7 messages in troubleshooting problems. MGCP Key Fields The first problem in call flow analysis is to identify which messages are associated with a single call attempt. A modest amount of call activity on the network produces a relatively large volume of MGCP messages. You must be able to extract a small subset of messages for analysis, those related to the failed call.
Chapter 4 Troubleshooting with Call Flows Call Flow Analysis • The MTA has been successfully provisioned and is ready for use. • The call attempt is successful (that is, the call flow is complete without errors). • You know the approximate time that the call was made. The time of the call is used to find the likely location of the MGCP messages in the LAN analyzer trace.
Chapter 4 Troubleshooting with Call Flows Call Flow Analysis Step 5 The first CRCX to the MTA contains the Call ID parameter. The Call ID parameter is used to link connection control messages (CRCXs, MDCXs, and DLCXs) at both ingress and egress endpoints. This is important if the call is off-net, since you do not know from the facts gathered so far which trunk gateway is used to egress the call. Search the LAN analyzer trace for messages that contains the same Call ID used in the first CRCX to the MTA.
Chapter 4 Troubleshooting with Call Flows Call Flow Analysis If this is a new service, verify that the line was properly provisioned in the Call Agent, and data was input to DHCP/LDAP. Allow time for the provisioning changes to flow through. Assuming that the MTA is connected and has been working, you need to trace the call flow. See Chapter 4, “Troubleshooting with Call Flows,” for the procedure for tracing a call flow.
Chapter 4 Troubleshooting with Call Flows Call Flow Analysis Call Agent Originating MTA Terminating MTA NTFY - off-hook, xsactionID-001 ACK - xsactionID=001 RQNT - play dial tone, collect digits with map, xsactionID=002 ACK - xsactionID=002 NTFY - diaed digits = 732-345-0909, xsactionID=003 ACK - xsactionID=003 CRCX, half duplex, xsaction ID=004 ACK - xsactionID=004, SDP params A CRCX, full duplex, SDP params A, ring, ringbackxsaction ID=005 ACK - xsactionID=005, SDP params B MDCX, half duplex, SDP par
Chapter 4 Troubleshooting with Call Flows Call Flow Analysis Step 2 Check to see whether the Call Agent can reach the terminating MTA—that is, are repeated attempts to create the connection (CRCX messages) seen at the terminating side? This may indicate a problem communicating with the terminating MTA. Note This failure should normally cause the Call Agent to route the originator to an announcement. Verify L3 connectivity between the Call Agent and the terminating MTA.
Chapter 4 Troubleshooting with Call Flows Call Flow Analysis Call Agent Originating MTA Terminating MTA PSTN Switch NTFY - off-hook, xsactionID-001 ACK - xsactionID=001 RQNT - play dial tone, collect digits with map, xsactionID=002 ACK - xsactionID=002 NTFY - diaed digits = 732-345-0909, xsactionID=003 ACK - xsactionID=003 CRCX, half duplex, xsaction ID=004 ACK - xsactionID=004, SDP params A CRCX, full duplex, SDP params A, ring, ringbackxsaction ID=005 ACK - xsactionID=005, SDP params B SS7 Initial
Chapter 4 Troubleshooting with Call Flows Call Flow Analysis Step 2 Check to see whether the Call Agent can reach the terminating TGW—that is, can you see repeated attempts to send CRCX messages to the trunk gateway? This may indicate a problem communicating with the terminating TGW. Note This failure should normally cause the Call Agent to route the originator to an announcement. Verify L3 connectivity between the Call Agent and the terminating TGW.
Chapter 4 Troubleshooting with Call Flows Call Flow Analysis Call Agent Originating MTA Terminating MTA PSTN Switch SS7 Initial Address Message (IAM) CRCX, full duplex, xsactionID=006 ACK - xsactionID=005, SDP params B CRCX, full duplex, ringback, ring, SDP params B, xsaction ID=004 ACK - xsactionID=004, SDP params A MDCX, half duplex, SDP params A, xsactionsID=006 ACK - xsactionID=006 RTP Session Establishment Message (ACM) Ringbacktone 104552 SS7 Address Complete To troubleshoot the problems in
Chapter 4 Troubleshooting with Call Flows Voice Quality Problems Step 3 If the MGCP messaging looks correct, check to see if the Call Agent sent an ACM to the PSTN switch. If the Call Agent appears to have sent the MGCP messaging correctly, but there is no ACM, check L3 connectivity between the connection manager and SS7 gateway. If there is L3 connectivity, the connection manager is not communicating with the SS7 gateway. Contact Cisco technical support.
Chapter 4 Troubleshooting with Call Flows Voice Quality Problems Lost or Distorted Audio One of the most common problems is a breaking up of audio, which is often described as garbled speech or a loss of syllables within words or sentences. There are two common causes: packet loss and jitter. • Packet loss—occurs when packets are dropped or arrive at their destination too late to be useful. • Jitter—is the variation in arrival times of successive packets.
Chapter 4 Troubleshooting with Call Flows Voice Quality Problems You can also adjust the receive level so that any reflected audio is reduced. It is very important to make small adjustments at a time. Too much attenuation can make the audio impossible to hear at both ends. Alternatively, you can contact the carrier and ask to have the lines checked.
Chapter 4 Troubleshooting with Call Flows Voice Quality Problems If the LostSyncs value is high (such as once an hour), there is something physically wrong with the connection. Send a technician to recable the MTA. If T3Timeouts or T4Timeouts values are high, it is most likely an upstream problem. • Look at signal-to-noise ratio, corrected errors, and uncorrectable errors.
Chapter 4 Troubleshooting with Call Flows Voice Quality Problems Dropped Calls Dropped calls occur when a call is terminated prematurely. Dropped calls can be the result of a gateway resetting or a circuit problem. Use a protocol analyzer to determine which side is hanging up the call, and trace call flows to isolate where the problem is occurring. Codec Mismatch If a customer gets a reorder tone when going off-hook, it could be the result of codec disagreement between endpoints.
C H A P T E R 5 Troubleshooting DOCSIS Networks DOCSIS provides the bandwidth and latency guarantees needed to provide toll-quality voice, data services, and multimedia applications across a shared HFC network. It is designed to be backward compatible, enabling DOCSIS 1.0 and 1.1 modems to operate in the same spectrum on the same network. This chapter discusses: • DOCSIS 1.
Chapter 5 Troubleshooting DOCSIS Networks DOCSIS 1.0+ • When CMTS receives the REG-REQ, it creates a local database entry for the MTA. A static SID is immediately assigned to the MTA for the data service. For the phone line service, the CMTS only creates 2 deferred service flows (for subsequent activation) in the MTA's database entry. No SIDs are assigned for the phone line service during registration.
Chapter 5 Troubleshooting DOCSIS Networks Understanding Initialization States Understanding Initialization States The following sections review each of the initialization states, including any problems that may occur in that state.
Chapter 5 Troubleshooting DOCSIS Networks Understanding Initialization States To troubleshoot, use a signal or spectrum analyzer off a drop port (the final splitter to which modems are attached) to measure the digital channel power coming from the splitter to the modem. The optimal input power level at the modem is 0 dBmV. The receiver has a range of -15dBmV to +15dBmV. • If the power is too low, configure the upconverter as per Chapter 4 of the Cisco uBR7246 Hardware Installation Guide.
Chapter 5 Troubleshooting DOCSIS Networks Understanding Initialization States This ranging process occurs in the initial maintenance or broadcast regions of the MAP, because the CMTS has not assigned the modem a service identifier (SID) for unicast transmissions in the MAP. Thus, broadcast ranging is contention based and subject to collisions. To compensate for this the modems have a ranging backoff algorithm to calculate a random backoff time between RNG-REQ transmissions.
Chapter 5 Troubleshooting DOCSIS Networks Understanding Initialization States Network Layer Configuration And Above After ranging is successful, the modem needs additional configuration from the operational support servers. The collection of services they provide and configure are called operational support services (OSS).
Chapter 5 Troubleshooting DOCSIS Networks Understanding Initialization States Caution Running debug commands on a uBR with more than a handful of modems may cause the uBR to halt the system in order to keep up with the debugging. In this case, all the modems may lose sync and debugging will be useless. If no packets are seen in debug messages, check the configuration of the “ip helper-address” statement on the cable interface to which this modem is attached.
Chapter 5 Troubleshooting DOCSIS Networks Understanding Initialization States Configuration File State The main configuration and administration interface to the MTA is the configuration file downloaded from the provisioning server.
Chapter 5 Troubleshooting DOCSIS Networks Radio Frequency (RF) Issues Operational State MAC State --->>> 'operational_state' cmWriteFlashFile("CM_BOOT", 0x8109180, 0x13e) by TID 0x83049b8 (tMACCtrl) usrEraseSysFlash(1, 0x1e0900, 0x13e) by TID 0x83049b8 (tMACCtrl) If registration and baseline privacy negotiation (if required) succeed, the modem is operational and is ready to pass traffic. A modem may make it to operational state but not remain in operational state.
Chapter 5 Troubleshooting DOCSIS Networks Troubleshooting RF Problems If a digital signal employing forward error correction (FEC) is near its impairment limit, it is very susceptible to changes in signal level---on the order of 0.1 dB.
Chapter 5 Troubleshooting DOCSIS Networks Measuring RF Signals Measuring RF Signals You can use a spectrum analyzer or a digital signal level meter to gather measurements of RF signals that can help you in identifying problems and making adjustments. Measuring downstream IF and RF signals with a spectrum analyzer is normally done at setup time. However, if there are problems detected, it may be necessary to verify the quality of the IF and RF downstream signals.
Chapter 5 Troubleshooting DOCSIS Networks Measuring RF Signals Step 5 Measure the IF signal using the channel power option on your spectrum analyzer. Set your channel spacing and your channel bandwidth to 6 MHz. Your analyzer should display a signal similar to the one shown in Figure 5-2. Figure 5-2 Step 6 Select the video averaging feature. Your spectrum analyzer should display a signal similar to the one shown in Figure 5-3.
Chapter 5 Troubleshooting DOCSIS Networks Measuring RF Signals Measuring the Downstream RF Signal at the Upconverter Output To measure the downstream radio frequency (RF) signal, perform the following steps: Step 1 Disconnect the spectrum analyzer from the cable modem card downstream connector. Connect the downstream output of the cable modem card to the upconverter input connector. Connect the spectrum analyzer to the RF output of the upconverter.
Chapter 5 Troubleshooting DOCSIS Networks Measuring RF Signals Step 4 Change the spectrum analyzer settings to view the digital channel power. This setting will enable you to see if there is too much power on the upconverter output. In Figure 5-6, the upconverter output is reading +64.31 dBmV, which is beyond the typical range of +50 to +58 dBmV. Note A spectrum analyzer might become overloaded and produce false readings (such as internally generated spurs) when measuring a signal at this amplitude.
Chapter 5 Troubleshooting DOCSIS Networks Measuring RF Signals Figure 5-8 Measuring the RF Signal at the Upconverter Output---Output Adjusted to Correct Range Note Any channel frequency response problems at the headend can impair network performance or prevent a cable modem on the HFC network from operating. The specified maximum peak-to-valley measurement from a Cisco cable modem card is +/-1.5 dB across 5.6 MHz. At the output of the upconverter, the maximum tilt should not exceed +/-1.5 dB across 5.
Chapter 5 Troubleshooting DOCSIS Networks Measuring RF Signals Table 5-2 DOCSIS Cable Downstream RF Specifications DOCSIS Specifications1 Minimum Settings2 RF channel spacing (bandwidth) 6 MHz 6 MHz Transit delay3 0.800 µsec Specification Your Headend Settings System/Channel 0.
Chapter 5 Troubleshooting DOCSIS Networks Troubleshooting Slow Peformance Measuring Upstream RF Signals To measure the upstream RF signal, perform the following steps (this procedure is known as the “Zero Span Method”, which is described in greater detail in “Connecting and Configuring the Cable Headend” available on Cisco Connection Online): Step 1 Connect the spectrum analyzer to the upstream signal from your cable network at the combiner where all the cable modems connect.
Chapter 5 Troubleshooting DOCSIS Networks Troubleshooting Slow Peformance Measuring the Correct Parts of the Network There are a number of ways to gauge the speed and performance of a network, however it is important to understand exactly what parts of the network are being tested.
Chapter 5 Troubleshooting DOCSIS Networks Troubleshooting Slow Peformance Note that the FTP or HTTP based upload and download test is only reliable for testing speeds of around 3Mbps or less. At higher speeds the processing power of the CPE device, Server or Network Interface Cards may become a limiting factor in the test. For testing speeds higher than about 3Mbps, dedicated data throughput testing equipment should be used.
Chapter 5 Troubleshooting DOCSIS Networks Troubleshooting Slow Peformance While the FTP transfer is occurring, it is possible to monitor the progress of the test on the CMTS using the show interface cable X/Y sid Z counters command where cable X/Y is the cable interface the modem under test is connected to, and Z is the Service ID (SID) number of the modem under test. This command shows how many bytes are being transferred from or to a particular cable modem.
Chapter 5 Troubleshooting DOCSIS Networks Troubleshooting Slow Peformance Potential Reasons for Poor Performance This section describes the common reasons for slow performance and suggested resolutions, including: • Performance being restricted by the limits in the DOCSIS configuration file • Bursty or inconstant download performance caused by using a sub-optimal rate limiting scheme on the cable modem termination system (CMTS) • Upstream and downstream channel congestion • Backhaul network or Inte
Chapter 5 Troubleshooting DOCSIS Networks Troubleshooting Slow Peformance Using a Sub-optimal Method for Rate Limiting When an end user is trying to download data from the Internet at a rate greater than their cable modem's DOCSIS configuration file allows, the CMTS must rate limit the traffic being sent to that user to ensure that the user does not consume more than their allowed share of bandwidth.
Chapter 5 Troubleshooting DOCSIS Networks Troubleshooting Slow Peformance You can view how severely the CMTS is rate limiting traffic to a particular cable modem by using the show interface cable X/Y sid counters command, where cable X/Y is the cable interface which the cable modem is connected to, and Z is the SID number of the modem being observed.
Chapter 5 Troubleshooting DOCSIS Networks Troubleshooting Slow Peformance Upstream Channel Congestion Note The measures discussed in this section will not significantly increase the performance of an already uncongested network. The upstream channel is normally the most precious resource in a cable network. At present, most cable service providers use a 1.6MHz channel width and Quadrature Phase Shift Keying (QPSK) modulation in the upstream path. This equates to approximately 2.
Chapter 5 Troubleshooting DOCSIS Networks Troubleshooting Slow Peformance Ways of relieving upstream congestion include: • Reducing the number of cable modems per upstream - If there are too many cable modems connected to a particular upstream, or if users on a particular upstream are heavy users of upstream bandwidth, then the best solution is to move some users on the congested upstream port to an under utilized upstream port, or to a completely new upstream port.
Chapter 5 Troubleshooting DOCSIS Networks Troubleshooting Slow Peformance Table 5-3 shows the total available downstream bandwidth associated with the four possible downstream modulation schemes available in DOCSIS networks.
Chapter 5 Troubleshooting DOCSIS Networks Troubleshooting Slow Peformance The third component to note is the output rate which shows the average downstream throughput rate in bits per second. If this number consistently exceeds approximately 75% of the available downstream bandwidth during the peak usage time then end users will start to experience slower Internet access and higher latency. By default, these statistics are calculated over a five minute moving average.
Chapter 5 Troubleshooting DOCSIS Networks Troubleshooting Slow Peformance If performance is still slow, then there is a performance problem in the network that is not related to the CMTS or the cable segment. If performance from the Local CMTS network segment is significantly better than for users connected to cable modems then you should focus your efforts back on the CMTS and the cable segment.
Chapter 5 Troubleshooting DOCSIS Networks Troubleshooting Slow Peformance Invalid BW Requests= 0x102 Minislots Requested= 0x65B74A2 Minislots Granted = 0x65B74A2 Minislot Size in Bytes = 16 Map Advance (Dynamic) : 2809 usecs UCD Count = 23068 In the example above, the estimated SNR reading is 28.628dB. This is adequate for QPSK upstream operation.
Chapter 5 Troubleshooting DOCSIS Networks Troubleshooting Slow Peformance See the Cisco Cable Modem Termination System Command Reference for more information about the show cable flap-list command. • Cable modems displaying a * or a ! in the output of a show cable modem or show cable flap-list. A * indicates a cable modem that is rapidly varying it's upstream power levels.
Chapter 5 Troubleshooting DOCSIS Networks Troubleshooting Slow Peformance . . . . . . . 89 3304 90 12 92 0 93 40 81013 769 385 13058 40 15 0 3 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0 0 0 0 PIM Process CEF Scanner DHCPD Timer DHCPD Database In the example show above, the current CPU load on the CMTS is 45%/21%. This means that the total CPU utilization is at 45% of the capacity of the system. In addition 21% of the CPU is being used to service interrupts.
Chapter 5 Troubleshooting DOCSIS Networks Troubleshooting Slow Peformance Conclusion A DOCSIS cable network is a sophisticated network that requires proper planning and maintenance. Most performance issues in DOCSIS cable networks are a direct result of the proper planning and maintenance not being performed.
C H A P T E R 6 Troubleshooting MTAs Troubleshooting EMTA Provisioning Provisioning PacketCable Embedded Media Terminal Adapters (EMTAs) is a relatively complex process; however, with the right tools and ‘tricks of the trade,’ getting EMTAs operational is a fairly straightforward process. This chapter assumes that the Cisco Network Registrar (CNR) and Broadband Access Center for Cable (BACC) are both in use; however, much of the information would also apply for other deployments.
Chapter 6 Troubleshooting MTAs Troubleshooting EMTA Provisioning Embedded Media Terminal Adapter The EMTA is a cable modem (CM) and a Media Terminal Adapter (MTA) in one box, with a common software image. The CM and MTA each has its own MAC-address and each performs DHCP to get its own IP address. The EMTA contains, at minimum, 2 certificates. One certificate is a unique MTA certificate, sent by the MTA to authenticate itself to the key distribution center (KDC).
Chapter 6 Troubleshooting MTAs Troubleshooting EMTA Provisioning Call Management Server The CMS is essentially a softswitch, or call-agent, with additional PacketCable functionality to control QoS on a cable network, among other things. The MTA sends a network call signaling (NCS) restart in progress (RSIP) message to the CMS upon successful PacketCable provisioning. Key Variables This section describes the key variables that you need to know to provision an EMTA correctly.
Chapter 6 Troubleshooting MTAs Troubleshooting EMTA Provisioning Scope Selection Tag(s) In most scenarios, the BACC will be involved in processing all DHCP requests from scopes which have scope selection tags that match the selection criteria as specified in the DHCP criteria configuration tab in the BACC GUI. Client Class can also be used to tie scopes to BACC processing. Make sure you make this association before you attempt to provision devices.
Chapter 6 Troubleshooting MTAs Troubleshooting EMTA Provisioning Logs The following log files contain information as listed: • CNR logs name_dhcp_1_log and name_dns_1_log contain the most recent logging entries from CNR. Look here for DHCP or DNS related problems. • $BPR_HOME/kdc/logs/KDC.log shows all KDC interactions with MTAs, and KDC interactions with the provisioning server (BACC DPE). • $BPR_HOME/dpe.
Chapter 6 Troubleshooting MTAs Troubleshooting EMTA Provisioning Table 6-1 Trobleshooting Scenarios (continued) Problem Possible Causes MTA does not accept DHCP offer (continually cycles thru DHCP steps) a. Invalid DHCP options configured b. Offer came from DHCP server other than indicated in CM portion’s Option 122 suboption 1 MTA never contacts KDC (as indicated by KDC.log, or ethereal trace KDC reports failure at Step 9 (Kerberos AS-Request) Remedies a.
Chapter 6 Troubleshooting MTAs Troubleshooting EMTA Provisioning Table 6-1 Trobleshooting Scenarios (continued) Problem Possible Causes KDC reports success at step 9 (AS-Request/Reply), but MTA never moves past step 9, and continually reprovisions up to that step a. Telephony cert mismatch between telephony root loaded/enabled on MTA, and that loaded on KDC a. Check certs on MTA and KDC. b. Corrupted telephony cert chain (unlikely) b. Ensure correct cert is loaded/enabled on MTA.
Chapter 6 Troubleshooting MTAs Motorola Surfboard Table 6-1 Trobleshooting Scenarios (continued) Problem Possible Causes Remedies MTA reports success at step 25, but proceeds to contact KDC again for ‘cms’ service a. MTA config file points to incorrect CMS a. Correct config file, or reconfigure BTS to use FQDN listed in the config file b.
Chapter 6 Troubleshooting MTAs Motorola Surfboard Physical Interfaces Motorola Surfboard SBV4502 contains the following interfaces: • Type-F female connector for connectivity to HFC cable system. • One RJ-45 connectors for Ethernet 10BaseT connectivity • Two FXS voice ports. • An access-port for the reset button. Voice Features The voice features of the SBV4502 are as follows: • Voice Encoding • G.711 (A-law, Mu-law), G.726 (32kbps), G.728, and G.729A codecs.
Chapter 6 Troubleshooting MTAs Arris MTAs • Upstream Channel Change (DCC supported for DOCSIS1.
Chapter 6 Troubleshooting MTAs Arris MTAs Step 3 At the Arris console prompt, enter the following command: [ 1] Console> callp m 1 The Arris MTA repeats the command you entered and displays the result.
Chapter 6 Troubleshooting MTAs Arris MTAs Cisco Broadband Local Integrated Services Solution Troubleshooting Guide 6-12 OL-5169-01
C H A P T E R 7 Troubleshooting the CMTS The Cisco Universal Broadband Router (uBR7246VXR, Figure 7-1) is a modular, standards-based Cable Modem Termination System (CMTS), as defined in the DOCSIS 1.1 specification, with an integrated router that offers carrier-class availability and advanced IP routing capabilities for mid to large cable headends or distribution hubs. Connectivity to the service provider (SP) for PSTN access is through a CMTS.
Chapter 7 Figure 7-1 Troubleshooting the CMTS Cisco uBR7246VXR—Front View 2 3 1 4 12 5 uBR - CLK-T1 SEC 11 LOS 6 ACTIVE 5 uBR - MCI6 5 uBR - MCI6 5 uBR - MCI6 5 uBR - MCI6 S S D 4 U S U S 3 U S 2 U S 1 U D LE FAULT S S D U S 2 U S D 1 U U LE 0 7 S S D U S 2 U S D 1 U U LE 0 S D S 2 U S 1 U S U D LE U 0 EN AB S 9 72308 EN AB S EN AB S EN AB 0 U 10 S FREERUN ER 8 Figure 7-2 Cisco uBR7246VXR—Rear View 1 2 6 5
Chapter 7 Troubleshooting the CMTS Physical Interfaces HFC interfaces are provided by up to 4 plug-in modem cards. A variety of modem cards are supported, including UBR-MC11C, UBR-MC12C, UBR-MC14C, UBR-MC16C, and UBR-MC16S. All the cable modem cards have one downstream cable port. The number of upstream cable ports vary with the type of card. For example MC16C has 6 upstream ports. The MC16S has spectrum managemnent support. The spectrum manager continuously monitors the noise in unused upstream channels.
Chapter 7 • Integrated Time Of Day server • Integrated DHCP server • Per SID Bandwidth Request /Grant counters • QoS for voice: LLQ, TOS • Point to Point Protocol (PPP) Troubleshooting the CMTS Performance Redundancy/Availability/Compatibility Features The Cisco uBR7246VXR has a carrier class architecture with redundant components, modular design and hardening facilitates maximum performance and uptime in mission critical applications.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR • Weighted Fair Drop (WFD) enhancements • DOCSIS 1.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR The Cisco uBR7246VXR software provides commands to help diagnose problems with: • Media Terminal Adapters (MTAs) and the cable plant—Cisco uBR7246VXR software includes a flap list that helps isolate problems between the cable plant (such as ingress noise or incorrect power levels) and specific MTAs. A “flap” is defined as an MTA being registered on the Cisco uBR7246VXR, de-registering, and then immediately reregistering.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Table 7-1 Note State Values (continued) State Value Description init (t) TOD exchange started init(o) Option file transfer started online Cable modem registered, enabled for data online(d) Cable modem registered, but network access for the cable modem is disabled online(pk) Cable modem registered, BPI enabled and KEK assigned online(pt) Cable modem registered, BPI enabled and TEK assigned reject (pk) KEK modem key as
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Tuner: status=0x00 Rx: tuner_freq 529776400, symbol_rate 5361000, local_freq 11520000 snr_estimate 166(TenthdB), ber_estimate 0, lock_threshold 26000 QAM not in lock, FEC not in lock, qam_mode QAM_64 (Annex B) Tx: tx_freq 27984000, symbol rate 8 (1280000 sym/sec) power_level: 6.0 dBmV (commanded) 7 (gain in US AMP units) 63 (BCM3300 attenuation in .4 dB units) ...
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR 2d18h: 239198.524 CMAC_LOG_WILL_SEARCH_DS_FREQUENCY_BAND 2d18h: 239198.528 CMAC_LOG_WILL_SEARCH_DS_FREQUENCY_BAND 2d18h: 239198.528 CMAC_LOG_WILL_SEARCH_DS_FREQUENCY_BAND 2d18h: 239198.528 CMAC_LOG_WILL_SEARCH_DS_FREQUENCY_BAND 2d18h: 239198.532 CMAC_LOG_WILL_SEARCH_DS_FREQUENCY_BAND 2d18h: 239198.532 CMAC_LOG_WILL_SEARCH_DS_FREQUENCY_BAND 2d18h: 239198.532 CMAC_LOG_WILL_SEARCH_DS_FREQUENCY_BAND 2d18h: 239198.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR 4d00h: 345781.280 CMAC_LOG_RNG_RSP_MSG_RCVD 4d00h: 345781.282 CMAC_LOG_RNG_RSP_SID_ASSIGNED 3 4d00h: 345781.284 CMAC_LOG_ADJUST_RANGING_OFFSET 2288 4d00h: 345781.288 CMAC_LOG_RANGING_OFFSET_SET_TO 11898 4d00h: 345781.292 CMAC_LOG_ADJUST_TX_POWER 7 4d00h: 345781.294 CMAC_LOG_POWER_LEVEL_IS 24.0 dBmV (comma) 4d00h: 345781.298 CMAC_LOG_STATE_CHANGE ranging_2_state 4d00h: 345781.302 CMAC_LOG_RNG_REQ_QUEUED 3 4d00h: 345782.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Once a usable UCD is found, the cable modem will begin to listen to MAP (Bandwidth Allocation Map) messages which contain the upstream bandwidth allocation map of time. A section of time is mapped out into mini-slots and assigned to individual modems. There are also regions in the MAP for broadcast, contention based initial maintenance (or broadcast) ranging.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR 1w3d: 871160.686 CMAC_LOG_RNG_REQ_QUEUED 6 1w3d: 871161.690 CMAC_LOG_RNG_REQ_TRANSMITTED 1w3d: 871161.690 CMAC_LOG_RNG_RSP_MSG_RCVD 1w3d: 871161.694 CMAC_LOG_ADJUST_TX_POWER -36 1w3d: 871161.694 CMAC_LOG_RANGING_CONTINUE 1w3d: 871162.698 CMAC_LOG_RNG_REQ_TRANSMITTED 1w3d: 871162.898 CMAC_LOG_T3_TIMER 1w3d: 871163.734 CMAC_LOG_RNG_REQ_TRANSMITTED 1w3d: 871163.934 CMAC_LOG_T3_TIMER 1w3d: 871164.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR DHCP - init(d) state The next stage after successful ranging is acquiring network configuration via DHCP. Below is a an output display of show cable modem showing a cable modem in init(d), which indicates that the DHCP request was received from the MTA: #show cable modem Interface Prim Sid Online State Timing Offset Rec Power QoS CPE IP address MAC address Cable2/0/U0 7 init(d) 2811 0.25 0 10.1.1.20 0030.96f9.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Caution Running debug commands on a uBR (Universal Broadband Router) with more than a handful of modems may cause the Cisco uBR7246VXR to halt the system in order to keep up with the debugging. In this case, all the cable modems may lose sync and debugging will be useless.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Since the Cable Modem has reached as far as init(i) we know that it has got as far as obtaining an IP address. This can be clearly shown in the output display of the debug cable-modem mac log verbose command from the cable modem below: 3d20h: 334402.548 CMAC_LOG_RANGING_SUCCESS 3d20h: 334402.548 CMAC_LOG_STATE_CHANGE dhcp_state NOTE—IP address Assigned to CM: 3d20h: 334415.492 CMAC_LOG_DHCP_ASSIGNED_IP_ADDRESS 10.1.1.20 3d20h: 334415.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Problems getting TOD (Time of Day) or Timing Offset would also result in the cable modem not achieving online status: 3d21h: 3d21h: 3d21h: 3d21h: 3d21h: 3d21h: 3d21h: 3d21h: 3d21h: 3d21h: 3d21h: 3d21h: 3d21h: 3d21h: 3d21h: 3d21h: 338322.500 338334.260 338334.260 338335.424 338335.424 338335.424 338335.424 338335.424 338335.424 338335.428 338335.428 338335.428 338335.428 338335.432 338335.432 338336.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR 3d23h: 345297.596 CMAC_LOG_RNG_REQ_QUEUED 7 3d23h: 345297.596 CMAC_LOG_REGISTRATION_OK 3d23h: 345297.596 CMAC_LOG_STATE_CHANGE establish_privacy_state 3d23h: 345297.596 CMAC_LOG_PRIVACY_NOT_CONFIGURED 3d23h: 345297.596 CMAC_LOG_STATE_CHANGE maintenance_state 133.CABLEMODEM.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR 2d01h: 177960.616 CMAC_LOG_TOD_REQUEST_SENT 172.17.110.130 2d01h: 177960.712 CMAC_LOG_RNG_REQ_TRANSMITTED 2d01h: 177960.716 CMAC_LOG_RNG_RSP_MSG_RCVD 2d01h: 177961.716 CMAC_LOG_RNG_REQ_TRANSMITTED 131.CABLEMODEM.CISCO: 2d01h: %UBR900-3-TOD_FAILED_TIMER_EXPIRED:TOD failed, but Cable Interface proceeding to operational state 2d01h: 177986.616 CMAC_LOG_TOD_WATCHDOG_EXPIRED 2d01h: 177986.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Registered: Privacy Established: FALSE FALSE When you see init(o) in the show cable modem output, running the debug cable-modem mac log verbose will not tell you if it is a corrupt configuration file or a TFTP server failure that is the cause. The debug point to both of them. An example of invalid Configuration Parameters in the DOCSIS CPE Configurator is invalid or missing Vendor ID or Vendor Specific Information.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Cable2/0/U0 5 online(pt) 2290 0.25 5 0 10.1.1.25 0050.7366.2223 Cable2/0/U0 6 online(d) 2815 0.00 6 0 10.1.1.27 0001.9659.4461 Online(d) is typically caused by disabling the Network Access option under Radio Frequency info in the DOCSIS CPE Configurator. The default for Network Access is enabled.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Assigned SID: Max Downstream Rate: Max Upstream Rate: Upstream Priority: Min Upstream Rate: Max Upstream Burst: Privacy Enable: 4 10000000 1024000 7 0 0 FALSE [Rest of display been omitted] Online means the cable modem has come online and is able to communicate with the Cisco uBR7246VXR.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Reject(pk) and Reject(pt) state The following display of output from show cable modem on the Cisco uBR7246VXR shows a reject(pk) state: #show cable modem Interface Prim Sid Online State Timing Offset Rec Power QoS CPE IP address MAC address Cable2/0/U0 2 reject(pk) 2812 0.00 0 10.1.1.20 0030.96f9.65d9 6 01:58:51: %UBR7200-5-UNAUTHSIDTIMEOUT: CMTS deleted BPI unauthorized Cable Modem 0030.96f9.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Below is a sample debug output on the cable modem when there is an authorization failure: 6d02h: 527617.480 CMAC_LOG_CONFIG_FILE_PROCESS_COMPLETE 6d02h: 527617.480 CMAC_LOG_STATE_CHANGE registration_state 6d02h: 527617.484 CMAC_LOG_REG_REQ_MSG_QUEUED 6d02h: 527617.488 CMAC_LOG_REG_REQ_TRANSMITTED 6d02h: 527617.492 CMAC_LOG_REG_RSP_MSG_RCVD 6d02h: 527617.492 CMAC_LOG_COS_ASSIGNED_SID 1/2 6d02h: 527617.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Registration - reject (c) state A cable modem that fails registration due to bad class of service (COS) has a state of reject(c).
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Monitoring the Cisco uBR7246VXR Flap List The Cisco uBR7200 series maintains a database of flapping cable modems to assist in locating cable plant problems. The flapping cable modem detector tracks the upstream and downstream performance of all cable modems on the network, without impacting throughput and performance, or creating additional packet overhead on the broadband network.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Tips In Cisco IOS Release 12.0(7)XR2, Cisco IOS Release 12.1(1a)T1, and higher, the system supports automatic power adjustments. The show cable flap-list and show cable modem commands now indicate when the Cisco uBR7200 series has detected an unstable return path for a particular modem and has compensated with a power adjustment.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Sample: MAC ID___ CableIF 0010.7b6b.60ad C3/0 U0 0010.7b6b.65a3 C3/0 U0 0010.7b6b.6b9d C3/0 U0 0 0 0 Ins _Hit__ Miss 14386 1390 14503 1264 14060 1726 CRC 1 1 3 P-Adj 38 33 40 Flap ___Time____ 41 Nov 24 21:34:24 37 Nov 24 21:28:09 43 Nov 24 21:18:36 Table 7-2 Flap List Statistics Description Statistic Description MAC ID This is the MAC-layer address of a cable modem.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Table 7-2 Flap List Statistics Description (continued) Statistic Description Hit and Miss The HIT and MISS columns are keepalive polling statistics between the Cisco uBR7200 series and the cable modem. The station maintenance process occurs for every modem approximately every 25 seconds. When the router receives a response from the cable modem, the event is counted as a Hit.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Table 7-2 Flap List Statistics Description (continued) Statistic Description Power Adjustments (P-Adj) The station maintenance poll in the Cisco uBR7246VXR constantly adjusts the cable modem transmit power, frequency, and timing. The P-Adj column indicates the number of times the router instructs the cable modem to adjust transmit power more than 3dB.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR • Condition 3: Relatively high power adjustment counter. Analysis: Indicates the power adjustment threshold is probably set at default value of 2 dB adjustment. the cable modem transmitter step size is 1.5 dB, whereas the headend may command 0.25 dB step sizes. Tuning your power threshold to 6 dB is recommended to decrease irrelevant entries in the flap list.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Setting Cable Flap List Insertion Time You can set the cable flap list insertion time. When a cable modem makes an insertion request more frequently than the amount of insertion time defined by this command, the cable modem is placed in the flap list. The valid range is from 60 to 86400 seconds. A cable modem will not “flap” more than once in each insertion time interval.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Note For underground HFC networks with 4 amplifier cascade length, a typical threshold value should be 3 dB. For overhead HFC networks with 4 amplifier cascade length, a typical threshold value should be 4 dB. Longer coax cascades without return path thermal gain control and sites with extreme daily temperatures will have larger threshold ranges.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Verifying Cable Flap List Miss Threshold To verify the cable flap list miss threshold, enter the show cable flap list command and note the values in the Miss column: show cable flap list Mac Addr 0010.7b6b.5d1d 0010.7b6b.5e15 0010.7b6b.5e27 0010.7b6b.5d29 0010.7b6b.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Syntax Description: mac-addr (Optional) MAC address. Specify the 48-bit hardware address of an individual cable modem. all (Optional) Remove all cable modems from the flap-list table. No default behavior or values. Cable modems are removed from the flap list table after the number of days (between 1 and 60) specified by the cable flap-list aging global configuration command.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Similar to the show cable flap-list display, the * symbol in the show cable modem output indicates that the Cisco uBR7246VXR is using the power adjustment method on this modem. The ! symbol indicates that the cable modem has reached maximum transmit power.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Setting Frequency Threshold to Affect Power Adjustment To control power adjustment methods by setting the frequency threshold, use the cable upstream freq-adj averaging interface configuration command. To disable power adjustments, use the no form of this command.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR The following example confirms that the cable modem at 0050.7366.2223 is connected to the network and is operational: # ping docsis 0050.7366.2223 Queueing 5 MAC-layer station maintenance intervals, timeout is 25 msec: !!!!! Success rate is 100 percent (5/5) Following are the keys used for the ping response messages: Tips f The ping message failed. . The ping message timed out without getting a response.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Generating PRBS Test Signals To configure Pseudo Random Bit Stream (PRBS) test signals, perform the steps in Table 7-6 in the configure interface mode. Table 7-6 Procedure to Generate PRBS Test Signals Step Command Description 1 Router(config-if)# cable downstream if-output prbs Generates a PRBS test signal on the downstream channel. The interface is shut down.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR The following command displays output for the verbose keyword extension for all SIDs on the specified interface: router# show interface c3/0 sid counters verbose Sid Input packets Input octets Output packets Output octets BW requests received Grants issued Rate exceeded BW request drops Rate exceeded DS packet drops Sid Input packets Input octets Output packets Output octets BW requests received Grants issued Rate exceeded BW request
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR The following command displays the counters of the SIDs connected to the specified interface: router# show interface c3/0 sid counters Sid Inpackets Inoctets Outpackets Outoctets 1 2 3 40 0 0 16586 0 0 31 0 0 9160 0 0 Ratelimit BWReqDrop 0 0 0 Ratelimit DSPktDrop 0 0 0 Table 7-7 describes the fields shown in the output for the show interface cable sid displays.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Displaying Type of Service (ToS) Specifications The following command displays ToS specifications: uBR7200#show cable qos profile Service Prio Max class upstream bandwidth 1 0 0 2 0 64000 3 0 1000 4 3 256000 5 5 1000000 6 3 256000 Guarantee upstream bandwidth 0 0 0 0 0 0 Max downstream bandwidth 0 1000000 1000 512000 10000000 512000 Max tx TOS TOS Create burst mask value by 0 0 0 0 0 0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Table 7-8 describes the fields shown in the show interface cable upstream display. Table 7-8 Show Interface Cable Upstream Command Field Descriptions Field Description Cable Indicates the location of the upstream interface. Upstream is up/...administratively down Indicates the administrative state of the upstream interface. Received broadcasts Number of broadcast packets received through this upstream interface.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Table 7-8 Show Interface Cable Upstream Command Field Descriptions (continued) Field Description Current Total Bandwidth Reserved Total amount of bandwidth reserved by all modems sharing this upstream channel that require bandwidth reservation. The Class of Service for these modems specifies some non-zero value for the guaranteed-upstream rate.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Table 7-8 Show Interface Cable Upstream Command Field Descriptions (continued) Field Description Init Mtn IEs The counter of Initial Maintenance IEs; that is, counters for each type of upstream slot scheduled in the MAPs for this upstream channel. For example, “Init Mtn IEs 800” means that the MAC scheduler has added 800 initial maintenance information elements (slots) at the time the show command was issued.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Using uBR MIBs Refer to Chapter 11, “Element Management and MIBs,” for a description of useful MIBs for uBRs. Also refer to Chapter 3, “Trouble Isolation Procedures,” for suggestions on using specific MIB objects in troubleshooting. Using Debug Commands Using CMTS debug commands on a large, in-service network is not recommended. These commands can generate large amounts of output, and dramatically affect router performance.
Chapter 7 Troubleshooting the CMTS Troubleshooting the Cisco uBR7246VXR Cisco Broadband Local Integrated Services Solution Troubleshooting Guide 7-46 OL-5169-01
8 C H A P T E R Troubleshooting the Cisco Catalyst 6509 Cisco Catalyst 6509 Ethernet switches, deployed in a redundant configuration to provide high reliability, are used in the BLISS for Cable solution to provide Layer 2 connectivity for the Cisco BTS 10200, Layer 3 functionality for routing signaling packets to edge and trunking gateways, and to interconnect all servers within the SuperPOP.
Chapter 8 Troubleshooting the Cisco Catalyst 6509 Catalyst 6509 Ethernet switches are used to aggregate traffic from Trunk Gateways, Network Servers, Call Agent and the NMS server. The connections are Fast Ethernets. The key features of the Catalyst 6500 IP switch router include wirespeed Layer 3 IP, IP multicast, and forwarding across Ethernet, Fast EtherChannel (FEC), and Gigabit EtherChannel (GEC) capability.
Chapter 8 Troubleshooting the Cisco Catalyst 6509 Redundancy • Redundant Switch Fabrics (Catalyst 6500 Series only) • Redundant power supply and cooling • Redundant System Clocks • Redundant Supervisors • Redundant Uplinks Troubleshooting the Switch There are many ways to troubleshoot a switch. As the features of switches grow, the possible things that can break also increase.
Chapter 8 Troubleshooting the Cisco Catalyst 6509 complicated to troubleshoot because they often take advantage of special features like trunking and EtherChannel. The rest of the ports are significant as well because they connect the actual users of the network. Many things can cause a port to be non-functional: hardware issues, configuration issues, and traffic issues. Let's look at these categories a little deeper.
Chapter 8 Troubleshooting the Cisco Catalyst 6509 Copper Make sure you have the correct cable for the type of connection you are making. Category 3 cable can be used for 10MB UTP connections, but category 5 should be used for 10/100 connections. A straight-through RJ-45 cable is used for end-stations, routers or servers to connect to a switch or hub. An Ethernet crossover cable is used for switch to switch, or hub to switch connections. Below is the pin-out for an Ethernet crossover cable.
Chapter 8 Troubleshooting the Cisco Catalyst 6509 Make sure that the administrator has not shut down the ports involved (as mentioned earlier). The administrator could have manually shut down the port on one side of the link or the other. This link will not come up until you re-enable the port; check the port status. Some switches, such as the Catalyst 4000/5000/6000, may shut down the port if software processes inside the switch detect an error.
Chapter 8 Troubleshooting the Cisco Catalyst 6509 If you have link and the ports show connected, but you cannot communicate with another device, this can be particularly perplexing. It usually indicates a problem above the physical layer: layer 2 or layer 3. Try the following things. • Check the trunking mode on each side of the link. Make sure both sides are in the same mode.
Chapter 8 Troubleshooting the Cisco Catalyst 6509 Before exchanging the switch hardware you might try a few things: • Reseat the module in the switch. If you do this with the power on, make sure the module is hot swappable. If in doubt, turn the switch off before reseating the module or refer to the hardware installation guide. If the port is built in to the switch, ignore this step. • Reboot the switch. Sometimes this causes the problem to disappear; this is a workaround, not a fix.
Chapter 8 Troubleshooting the Cisco Catalyst 6509 Cisco GSR-12000 Series Gigabit Switch Router Cisco GSR-12000 Series Gigabit Switch Router The Cisco GSR-12000 Series Gigabit Switch Router (GSR) is a data aggregation point that passes packet streams on to, and receives packet streams from, the local IP backbone network in the Cisco BLISS for Cable solution.
Chapter 8 Troubleshooting the Cisco Catalyst 6509 Troubleshooting Serial Lines Gigabit/Fast Ethernet (GE/FE) As bandwidth demands increase in the Internet, service providers must cost effectively scale their networks and maintain network simplicity. Ethernet interfaces on the Cisc0 12000 Series help meet this critical requirement.
Chapter 8 Troubleshooting the Cisco Catalyst 6509 Troubleshooting Serial Lines Fiber-Optic Connections An optical signal I/O problem can be caused by • Incorrect type of fiber • Defective fiber • Transmit (TX) and Receive (RX) fibers reversed • Insufficient power budget on the optical link • Receiver overload on the optical link Be sure to use single-mode fiber for a single-mode interface and multimode fiber for a multimode interface.
Chapter 8 Troubleshooting the Cisco Catalyst 6509 Troubleshooting Serial Lines The difference between the power budget and the link loss (LL) is called the power margin (PM). If the power margin is zero or positive, the link should work. If it is negative, the signal may not arrive with enough power to operate the receiver.
Chapter 8 Troubleshooting the Cisco Catalyst 6509 Troubleshooting Serial Lines Using Bit Error Rate Tests This section discusses problem isolation using bit error rate (BER) tests. The topics discussed are: • Configuring a BER Test on a T1 Line, page 8-13 • Sending a BER Test Pattern on a T1 Line, page 8-13 • Viewing the Results of a BER Test, page 8-14 • Terminating a BER test, page 8-16 Configuring a BER Test on a T1 Line BER test circuitry is built into the CT3 line card.
Chapter 8 Troubleshooting the Cisco Catalyst 6509 Troubleshooting Serial Lines t1 t1-line-number bert pattern pattern interval time [unframed] where: • t1-line-number is 1–28. • time is 1–14400 minutes. • pattern is: – 0s, repetitive test pattern of all zeros (as 00000...) – 1s, repetitive test pattern of all ones (as 11111...) – 2^11, pseudorandom test pattern (2,048 bits long) – 2^15, pseudorandom O.151 test pattern (32,768 bits long). – 2^20-O153, pseudorandom O.
Chapter 8 Troubleshooting the Cisco Catalyst 6509 Troubleshooting Serial Lines Examples: • This example is for a CT3 line card: Router# show controllers T3 1/0/0 T3 1/0/0 is up. C2T3 H/W Version : 3, C2T3 ROM Version : 0.79, C2T3 F/W Version : 0.29.0 T3 1/0/0 T1 1 No alarms detected. Clock Source is internal.
Chapter 8 Troubleshooting the Cisco Catalyst 6509 Troubleshooting Serial Lines Terminating a BER test To terminate a BER test, type no t1 t1-line-number bert where t1-line-number is 1–28. Example: • Terminate the BER test running on T1 line 10 on the CT3 line card.
Chapter 8 Troubleshooting the Cisco Catalyst 6509 Troubleshooting Serial Lines Configuring a T3 Controller to Respond to Remote Loopback Commands The equipment customer loopback command allows a port to respond to loopback commands from remote T3 equipment. The equipment network loopback causes a controller to ignore remote T3 loopback commands.
Chapter 8 Troubleshooting the Cisco Catalyst 6509 Troubleshooting Serial Lines Both line and internal do the following • Loop any data received at the OC-12 POS card’s network interface back into the network • Loop any data received at the OC-12 POS card’s network interface back into the PRE card Use the no form of the command to stop the loopback test.
C H A P T E R 9 Troubleshooting Cisco Media Gateways Media Gateway Management The Cisco IOS software installed on the media gateways provides an array of network management capabilities, including: • SNMP and RMON Support—The media gateways are fully manageable using the Simple Network Management Protocol (SNMP) and imbedded Remote Monitoring (RMON) capabilities: SNMP provides for the collection of information about each controller and interface, which can be polled through any SNMP-compatible network ma
Chapter 9 Troubleshooting Cisco Media Gateways MGCP on Cisco IOS Software MGCP on Cisco IOS Software The Media Gateway Control Protocol (MGCP) runs on Cisco IOS software and the Cisco BTS 10200 Softswitch. The Cisco BTS 10200 Softswitch uses MGCP to control media gateways from external call control elements. A media gateway is a network element that provides conversion between audio signals carried on telephone circuits and data packets carried over the Internet or other packet network.
Chapter 9 Troubleshooting Cisco Media Gateways MGX 8850 Diagnostics • 512 Meg SRAM • MGX-RJ45-FE - FE card for RPM • RPM 1-TO-N redundancy The second configuration—MGX8850/PXM45/RPM-XF/VISM-PR/SRM-E—includes the following: • 19-inch rack mount. Redundant PXM1 (1.2.01) • 24 I/O+4 SRM slot, PXM1, PXM-UI, • MGX-VISM-8T1 (2.2.
Chapter 9 Troubleshooting Cisco Media Gateways Command Line Interface Command Line Interface The PXM45 system diagnostics CLI provides the ability to configure diagnostics, to initiate the On-line or Off-line diagnostics, and to display diagnostics status. The commands are described in the following paragraphs: Note All parameters are required unless specifically labeled (optional). cnfdiag This command configures diagnostics.
Chapter 9 Troubleshooting Cisco Media Gateways Command Line Interface dspdiagcnf This command displays the diagnostics configuration. Parameters None dspdiagstatus This command displays the diagnostics status. Parameters None dspdiagerr This command displays the diagnostics error. Parameters None. Diagnostic Figures The following figures show the diagnostics configuration, status, and error screens.
Chapter 9 Troubleshooting Cisco Media Gateways Command Line Interface Figure 9-2 Diagnostics Status Screen Figure 9-3 Diagnostics Error Screen Shellcon commands There are no shell commands for the diagnostic modules since most of the code is run from the CLI.
Chapter 9 Troubleshooting Cisco Media Gateways Command Line Interface Diagnostic Troubleshooting The following sections describe different diagnostic troubleshooting scenarios for the Cisco MGX 8850. Are “OnLine” Diagnostics Running To determine if the “online” diagnostics are running, complete the following steps: Step 1 Ensure that the diagnostics are enabled on the slot. This can be done by using the dspdiagcnf command.
Chapter 9 Troubleshooting Cisco Media Gateways Troubleshooting Alarms “Offline” Diagnostics Task is Taking Too Long to Complete or is Hung The offline diagnostics task is designed with different levels of coverage. The table below shows the types of coverages and the expected time for completion.
Chapter 9 Troubleshooting Cisco Media Gateways Troubleshooting Alarms Table 9-2 LED Indicators for PXM45 (continued) LED Label Color Meaning System Status Green Blinking green indicates that the card is in the active state. Yellow Slow blink yellow indicates that the card is in the standby state. Fast blink yellow indicates that the card is in the boot state. Red Solid red indicates that the card is in the Reset state, the card has failed, or a back card is missing.
Chapter 9 Troubleshooting Cisco Media Gateways Troubleshooting Alarms Figure 9-4 PXM45 Front Card Controls Controller port CNTLR Port Critical alarm (blue) Major alarm (red) Minor alarm (yellow) DC power A (green) DC power B (green) Alarm cut-off (yellow) History (green) Ethernet LAN control port (green) CR MJ MN DC-A DC-B ACO HIST ENET Alarm cut-off History Green = active Red = major alarm Yellow = minor alarm ACO HIST PXM45/B System status 38656 SYSTEM STATUS Blinking green = active Slow
Chapter 9 Troubleshooting Cisco Media Gateways Troubleshooting Alarms AXSM Card Controls Figure 9-5 shows the LEDs available on the front of the AXSM card. Table 9-3 describes these LEDs.
Chapter 9 Troubleshooting Cisco Media Gateways Troubleshooting Alarms Table 9-3 LED Indicators for AXSM Card LED Color Description Active Green Card is active. Standby Yellow Card is in standby mode. Fail Red Failure detected on card. Line Green The line is active and there are no alarms. Red The line is active, but a local alarm has been detected. Yellow The line is active, and a remote alarm has been detected.
Chapter 9 Troubleshooting Cisco Media Gateways Troubleshooting Alarms Displaying Alarm Reports in the CLI This section provides summary information about alarms on a shelf. The alarm manager also sets the alarm LEDs on the PXM card and sends a ShelfIntegratedAlarm Trap. You can use a CLI session to view the status of switch alarms. A set of CLI commands is provided for debugging hardware problems.
Chapter 9 Troubleshooting Cisco Media Gateways Troubleshooting Alarms dspndalms A node alarm report displays a summary report of all alarms on the node. To display node alarms, enter the following command: pop20two.7.PXM.a > dspndalms The following is an example of the node alarm report. pop20one.7.PXM.
Chapter 9 Troubleshooting Cisco Media Gateways Troubleshooting Alarms The next example shows a card alarm report for an AXSM card in slot 1: pop20one.7.PXM.a > dspcdalms 1 Card Alarm Summary Alarm Type ---------Hardware Alarm Card State Alarm Disk Alarm Line Alarm Port Alarm Feeder Alarm Channel Alarm Critical -------0 0 0 0 0 0 0 Major ------0 0 0 0 0 0 0 Minor ------0 0 0 0 0 0 0 Displaying Clock Alarms The MGX 8850 switch monitors the quality of its clock sources.
Chapter 9 Troubleshooting Cisco Media Gateways Troubleshooting Alarms Top Top Top Top Top Top Top Top Fan Fan Fan Fan Fan Fan Fan Fan Bottom Bottom Bottom Bottom Bottom Bottom Bottom Bottom Bottom Tray Tray Tray Tray Tray Tray Tray Tray Fan Fan Fan Fan Fan Fan Fan Fan Fan Tray Tray Tray Tray Tray Tray Tray Tray Tray +5V Input +3.3V Input +2.
Chapter 9 Troubleshooting Cisco Media Gateways Troubleshooting Alarms When the MGX 8850 reports xbar alarms, you can use the following troubleshooting commands to collect more information: • dspxbar • dspxbaralms • dspxbarerrcnt • dspxbarerrthresh • dspxbarmgmt • dspxbarstatus For more information on these commands, refer to the MGX 8850 Command Reference. PXM45x Alarm Issues Crossbar Alarm Commands • dspxbarerrthresh – Display the threshold for the different alarms.
Chapter 9 Troubleshooting Cisco Media Gateways Troubleshooting Alarms • dspfdr IF # • dsplmistat IF # to display LMI alarms Channel Alarm Commands • dspconinfo • dspcons –state fail - on PXM will show conditioned SPVCs • cc - The following commands are executed on the axsm • dspcons – filt option – displays more information for port / channel alarms.
Chapter 9 Troubleshooting Cisco Media Gateways Troubleshooting Alarms Step 2 Step 3 Slot Centric De-bugging a. Use dspdevalms XBARCORE -pslot to display the core alarm for the specific slot. b. Use dspdeverr XBARCORE -pslot and dspdeverrhist XBARCORE -pslot to display the current and cumulative error counters for the different error types. The different error types that were in alarm in a. will have corresponding error counters. Plane centric de-bugging a.
Chapter 9 Troubleshooting Cisco Media Gateways Troubleshooting the Gateways Displaying Log File Information Log files record switch events such as operator login and command entry. To view the contents of the current log, enter the following command: pop20two.7.PXM.a > dsplog [-sl ] [-mod CLI] To limit the log display to the events for a single slot, use the -sl option and replace slot with the appropriate slot number.
Chapter 9 Troubleshooting Cisco Media Gateways Troubleshooting the Gateways The lsdlsps command displays all DLSAP profiles. This command should never fail. However, it will not display anything of there are no DLSAPs profiles added. The deldlsp command deletes the DLSAP profile. This command could fail for the following reasons: 1. If the DLSAP index specified in the command is NOT added. 2. If the DLSAP index is out of range.
Chapter 9 Troubleshooting Cisco Media Gateways Troubleshooting the Gateways deldchan Command Fails The deldchan command deletes the specified D-channel. This command could fail if: 1. Specified D-channel is not added. 2. Specified parameters are out of range/illegal. 3. Logical slot specified in the command is not ACTIVE. D-Channel Is Down The lsdlsapstatus command displays the DLSAP status. If the status displays dataXfer, the D-channel is up and is ready to send and receive Q.931 PRI messages.
Chapter 9 Troubleshooting Cisco Media Gateways Data Path Troubleshooting Data Path Troubleshooting Data path debugging and troubleshooting is required when any of the following cases occur: • No Voice • BERT Test Fails • Tone Detection Fails There Is No Voice Fault Isolation 1. Make sure that T1/E1 lines and channel bank are configured correctly. 2. End point on NSC: determine the type of call: dspvoiceparm , 3.
Chapter 9 Troubleshooting Cisco Media Gateways Data Path Troubleshooting Step 3 Enter this command a few times. The PacketCnt should increase and the PacketDrpCnt should stay the same in normal case (for both ingress and egress). Step 4 If various PacketDrpCnt increase, check the ethernet port configuration on SCC: lsethlns (on SCC, list ether lines) Line IP Address Subnet Mask Status Gateway Addr ======================================================== 9.1 192.168.4.123 lsethln 9.1 255.255.
Chapter 9 Troubleshooting Cisco Media Gateways Data Path Troubleshooting • lsevt 0 When the end point is on BSC: Step 1 Find out the DSP channel on which NSC is assigned to provide the service for this connection: dspDspRscAll (on SCC) dspDspRscAll 0 DspChan 0: SRC SLOT 12 PORT 0 --> VIA SLOT 0 ECN (257, 11) --> DST ... DspChan 1: SRC SLOT 12 PORT 1 --> VIA SLOT 0 ECN (769, 11) --> DST ...
Chapter 9 Troubleshooting Cisco Media Gateways Data Path Troubleshooting Fax/Modem Fails Before starting your Fax/Modem, make sure that the connection is setup correctly (if possible, test the connection by talking on the phone). If connected to channel bank, make sure that the channel bank is configured properly with the DS1 lines (robbitedBit, source of the clock). If any switchover occurs while Fax/Modem is in progress, the Fax/Modem calls will be dropped. This is an expected result.
Chapter 9 Troubleshooting Cisco Media Gateways Data Path Troubleshooting Line Interface and Switching Path Troubleshooting Line interface and switching patch troubleshooting is necessary when problems occur with any of the following: • DS1 line interface • E1 line interface • DS3 line interface • OC3 line interface • ATM switching path components Most line interface symptoms are caused by improper hardware configuration, such as cabling.
Chapter 9 Troubleshooting Cisco Media Gateways Call Control Special case 3: DS1 line disappears/comes up with wrong configuration after BSC/NSC switch over/back Check event log for DS1 and DBM related error events; issue the following DBM debug command on SCC: dbmdisptab (63+slot #), 0, 1 Example: For service card in slot 2 (1-based): MMS.9.
Chapter 9 Troubleshooting Cisco Media Gateways Call Control Table 9-5 Debug Commands (continued) Command Description prtOn 28 Displays all the backhaul messages received or transmitted by the service card to IBPMUX module on the SCC. It also displays all the LAPD messages received/transmitted on all the D-channels.
Chapter 9 Troubleshooting Cisco Media Gateways Media Gateway Errors a long time to automatically logout an idle telnet session and the telnet session is left unattended for unauthorized personel to access the system. It is therefore recommended that the timeout value be restored to its original value. The mscpPrtOn command turns debugging on for Protocol layer messages prints. It takes a single parameter value (2,3 or 4) which scales the amount of debug info to print.
Chapter 9 Troubleshooting Cisco Media Gateways Troubleshooting VISM Cards Typically, both SCCs run the same software image. However, it is possible for SCCs not to run the same software image when there is only 1 SCC in the node and you tftp a different image to the SCC, and then insert another SCC which has an old image. In this case, the active SCC will check the image file checksum and transfer the image file from Active to Standby if they are different.
Chapter 9 Troubleshooting Cisco Media Gateways Troubleshooting VISM Cards VISM Card LEDs The VISM card uses the following three card status LEDs (see Figure 9-6) to indicate certain states: • ACT—Green indicates the active state. • STBY—Orange, or blinking orange, indicates one of the following: – VISM is in the standby state. – VISM is in the mismatch state. – VISM card DSPs are currently involved in the VISM card bootup.
Chapter 9 Troubleshooting Cisco Media Gateways Troubleshooting VISM Cards Figure 9-6 Card status LEDs VISM Front Card LEDs Active Standby Fail ACT STBY FAIL PORT 1 PORT 2 PORT 3 Line LEDs PORT 4 PORT 5 PORT 6 PORT 7 VISM 8E1 27981 PORT 8 E1 Front Card Cisco Broadband Local Integrated Services Solution Troubleshooting Guide OL-5169-01 9-33
Chapter 9 Troubleshooting Cisco Media Gateways Troubleshooting VISM Cards VISM and PXM Display, Log, and Diagnostic Loopback Path CLI Commands You can use the following commands to troubleshoot your VISM card: Note • The VISM dspcd command • The PXM dsplog command • PXM diagnostic loopback commands Refer to the Cisco MGX 8000 Series platform command reference guides for more information on PXM commands.
Chapter 9 Troubleshooting Cisco Media Gateways Troubleshooting VISM Cards PXM Display Log CLI Command Use the PXM dsplog command to display useful information for troubleshooting your VISM card. The log is maintained by the PXM.
Chapter 9 Troubleshooting Cisco Media Gateways VISM Alarms VISM Alarms Table 9-6 describes VISM T1 and E1 card alarms. Table 9-6 VISM T1 and E1 Card Alarms Error Alarm Type 1 Link Failure—receive LOS1 LOS Receive RAI3 Yellow Receive LOF 4 Receive AIS2 Down stream Up Stream (ATM side) (TDM side) AIS 2 RAI 3 Comments RAI3 returned on the transmit line. RAI3 None 2 3 RAI3 returned on the transmit line. RAI3 RAI3 returned on the transmit line. — AIS AIS2 AIS2 RAI — 1.
Chapter 9 Troubleshooting Cisco Media Gateways Symptoms and Solutions Symptoms and Solutions This section includes possible solutions to the following possible symptoms: • “VISM Card Did Not Become Active” section on page 9-37 • “T1/E1 Configuration Mismatch” section on page 9-37 • “DSP Download Failure” section on page 9-39 • “VISM Front Card/Back Card Mismatch” section on page 9-39 • “Cannot Use the cc Command to Access a VISM Card” section on page 9-40 • “VISM Card Resets Intermittently” sec
Chapter 9 Troubleshooting Cisco Media Gateways Symptoms and Solutions Use the PXM dspsmcnf command to identify a T1/E1 configuration mismatch, as follows: NODENAME.1.7.PXM.a > dspsmcnf slot Card Rate Channel MIB Feature No.
Chapter 9 Troubleshooting Cisco Media Gateways Symptoms and Solutions DSP Download Failure Use the PXM dsplog command to determine if the minimum number (five) of the DSPs failed to download. The terminal displays results similar to the following: NODENAME.1.7.PXM.a > dsplog 01/01/2001-00:02:10 05 tDspmDl VISM-6-9193 DSPM task errors : 6 DSPs failed to download If the number of DSPs (six in the above case) is greater than five, the card will fail to be in the active state.
Chapter 9 Troubleshooting Cisco Media Gateways Symptoms and Solutions 1.9 1.10 1.11 1.12 1.13 1.14 1.15 1.16 1.17 1.18 1.19 Empty Empty Empty Empty Empty Empty Empty Empty Empty Empty Empty Clear Clear Clear Clear Clear Clear Clear Clear Clear Clear Clear Use the VISM dspcd command to display the following types of information: NODENAME.1.5.VISM8.s > dspcd ModuleSlotNumber: 5 FunctionModuleState: Mismatch FunctionModuleType: VISM-8E1 FunctionModuleSerialNum: CAB0246014P FunctionModuleHWRev: 0.
Chapter 9 Troubleshooting Cisco Media Gateways Physical Indicators Echo Is Heard on a Voice Call Ensure that the call has the ECAN feature enabled. If the echo delay is longer than the provision tail length, ECAN does not work. Use the VISM cnfecantail command to configure a larger value for the tail length. VISM Card LEDs Are Not Lighted The VISM card may not be inserted completely in the slot.
Chapter 9 Troubleshooting Cisco Media Gateways Physical Indicators Table 9-8 VISM Card Line LEDs LED COLOR INDICATION Green The line has been added and there is no alarm on that line. Orange The line has been added and there is a YELLOW alarm condition on the line. Red The line has been added and there is a LOS condition (RED alarm condition) on the line..
Chapter 9 Troubleshooting Cisco Media Gateways Physical Indicators Card Does Not Respond to Ping from Router Symptom Cannot ‘CC’ to VISM card. Action Check whether the card is in the ACTIVE/STANDBY state, using the “dspcds” CLI on PXM. If the VISM card is not in the ACTIVE/STANDBY state, “cc” to that slot is not possbile. CiscoView Does Not Show VISM Card Configuration Symptoms Card constantly reboots -- becomes Active and then resets again. Card resets occasionally.
Chapter 9 Troubleshooting Cisco Media Gateways Physical Indicators Verify that the lines between the Cisco BTS 10200 and the VISM are in service. Verify that you can ping the VISM from the Cisco BTS 10200. If this fails, determine whether the routing tables on the Cisco BTS 10200 have been configured correctly. Verify that there is an entry for destination 10.0.0.0. Verify that the Cisco BTS 10200 IP address has been added to the VISM’s list.
Chapter 9 Troubleshooting Cisco Media Gateways Physical Indicators Voice Is Distorted Find out the compression DSP that the call is on. Use the shellConn command dspm_get_err to display error statistics on the DSP. If the error count is non-zero, and keeps increasing as this command is entered again, then this is a source of the problem. VISM Card Has All Leds Turned Off After Insertion In A Chasis Cause The VISM card is not inserted completely in the slot.
Chapter 9 Troubleshooting Cisco Media Gateways Physical Indicators Cisco Broadband Local Integrated Services Solution Troubleshooting Guide 9-46 OL-5169-01
C H A P T E R 10 Troubleshooting the Cisco BTS 10200 Softswitch This chapter describes various troubleshooting techniques for the Cisco BTS 10200 Softswitch. It includes the following sections: • Architecture, page 10-1 • Components, page 10-5 • Troubleshooting, page 10-10 • Cisco BTS 10200 Failure, page 10-17 • Operating System Failure, page 10-17 Architecture This section briefly describes the Cisco BTS 10200 Softswitch architecture from an external components perspective.
Chapter 10 Troubleshooting the Cisco BTS 10200 Softswitch Architecture Call Agent Call processing is performed with MGCP 1.0/NCS call control signaling with the Cisco BTS10200 call agent deployed in an active/stand mode. Feature servers, which provide the logic for enhanced services, are part of this complex. An EMS system which manages the call agent and the feature servers and provides subscriber/network provisioning functions is also part of the call agent infrastructure residing in the SuperPOP.
Chapter 10 Troubleshooting the Cisco BTS 10200 Softswitch Architecture Interfaces In the Cisco BLISS for Cable solution, an interface is any device that the solution components use to accomplish internetworking. The relevant interfaces in the solution are the following: • Trunking Gateway Interface—Trunking gateways (TGW) are high-density PSTN to packet network gateways.
Chapter 10 Troubleshooting the Cisco BTS 10200 Softswitch Architecture • Media Gateways Interface to the PSTN—The Cisco BTS 10200 Softswitch provides both originating and terminating two-wire and four-wire continuity check (COT), a necessary feature within the PSTN for automatic monitoring of circuit quality. An IOS image that supports the control protocol is required for use in conjunction with the call agent software.
Chapter 10 Troubleshooting the Cisco BTS 10200 Softswitch Components Components This section describes the Cisco BTS 10200 Softswitch components: • Rack Configuration, page 10-6 • Feature Server Architecture, page 10-8 • EMS Architecture, page 10-8 • Continuous Computing AXi, page 10-9 • Reference Documentation, page 10-9 The Cisco Broadband Local Integrated Services Solution uses the Cisco BTS 10200 Softswitch, Figure 10-2, as a call agent (CA).
Chapter 10 Troubleshooting the Cisco BTS 10200 Softswitch Components Table 10-2 Cisco BTS 10200 Softswitch Features and Call Types Feature Support for... EMS Mediation device between an NMS1 and one or more CAs. It also supports OAMP2 functions, which include the following element management areas: • Billing management • Traffic management • Fault management • Event management • Configuration management (status and control) • Security management CA Serves as a CMS3 and MGC4.
Chapter 10 Troubleshooting the Cisco BTS 10200 Softswitch Components Figure 10-3 Typical Cisco BTS 10200 Softswitch Rack Configuration Alarm panel, PDU power feed 8A Filler panel PDU (DC systems) 1U 1U 1U 2U Filler panel ON APP R ALRM YES NO OFF CONS EMS B (AXi) CCN node address = 1 PDU power feed 1A/1B PW 4U PW ON R APP ALRM OFF CONS YES NO Call agent B (AXmp) CCN node address = 2 PDU power feed 2A/2B 6U Filler panel (Expansion CA or Ext. FS) 6U 2.36 m 93 in.
Chapter 10 Troubleshooting the Cisco BTS 10200 Softswitch Components Feature Server Architecture The purpose of the FS is to provide access to features through a well-defined interface. The Cisco BTS 10200 Softswitch architecture separates FS (which provides feature control) from CA (which provides call control) with a clear interface, Feature Control Protocol (FCP), defined between them.
Chapter 10 Troubleshooting the Cisco BTS 10200 Softswitch Components Cisco BTS 10200 Softswitch Controller Hosts Each telephony controller host supports a minimum of 49K DS0 channels with a throughput of 100 DS0 calls (POTS) per second (setup and tear-down), with an average hold time of three minutes. It provides at least 99.9994 percent availability.
Chapter 10 Troubleshooting the Cisco BTS 10200 Softswitch Troubleshooting Troubleshooting This section presents procedures for troubleshooting the Cisco BTS 10200 Softswitch. It includes the following subsections: • Verify Running Processes, page 10-10 • Verify Current Status, page 10-10 Verify Running Processes Perform the following steps to open Unix shells on the primary and secondary EMS platforms and on the primary and secondary CA/FS platforms.
Chapter 10 Troubleshooting the Cisco BTS 10200 Softswitch Troubleshooting The call processing logic in the Cisco BTS 10200 Softswitch is based on the IN Capability Set 2 (CS2) half call model. The interface between the basic call module (BCM), which provides the core call processing logic, and the signaling interface adapters (SIA) is protocol independent. To verify the current status of the Cisco BTS 10200 Softswitch you can enter commands using the command line interface (CLI).
Chapter 10 Troubleshooting the Cisco BTS 10200 Softswitch Troubleshooting Step 2 To display the media gateway test menu and perform specific tests, enter the command diag mgw. You should receive a response similar to the following example: Reply: Diagnostic MGW Menu (1) MGW Network Connectivity Test (2) MGW MGCP Connectivity Test (3) ALL Test 1 verifies that there is a path to the device by pinging it, test 2 verifies that the device has MGCP connectivity, and test 3 performs both tests 1 and 2.
Chapter 10 Troubleshooting the Cisco BTS 10200 Softswitch Troubleshooting SS7-controlled Trunk Groups When an SS7-controlled trunk group’s administrative state is set to out-of-service (OOS), the trunk group cannot be used for outgoing calls; however, incoming calls on that trunk group are still honored. • Trunk states are not changed. • The operational state of that trunk group is not changed or affected.
Chapter 10 Troubleshooting the Cisco BTS 10200 Softswitch Troubleshooting TEST-DURATION -> 0 RESULT -> TEST-SUCCESS REASON -> PASSED: Reason: AUEP-NACK received with RespCode = 510 Reply: Diagnostic command executed. Step 4 Run any additional tests that might be needed to determine that all SS7 signaling links are reachable and in service.
Chapter 10 Troubleshooting the Cisco BTS 10200 Softswitch Troubleshooting You should receive a response similar to the following example: TRUNK DIAGNOSTIC TEST EXECUTED -> diag trunk TG-NUM -> 17 CIC -> 1 TEST-TYPE -> ADM-TERM-CONNECTION-TEST TEST-DURATION -> 0 RESULT -> TEST-SUCCESS REASON -> PASSED: Reason: AUEP-NACK received with RespCode = 510 Reply: Diagnostic command executed.
Chapter 10 Troubleshooting the Cisco BTS 10200 Softswitch Troubleshooting Step 3 To perform test 1 on CAS trunk group 64, enter the following command: diag cas-trunk-termination test=1; tgn-id=64; cic=1; You should receive a response similar to the following: Reply: Diagnostic CAS Trunk Group Menu.
Chapter 10 Troubleshooting the Cisco BTS 10200 Softswitch Cisco BTS 10200 Failure SIP-controlled Trunk Groups When a SIP-controlled trunk group’s administrative state is set to out-of-service (OOS), the trunk group cannot be used for outgoing calls and incoming calls on that trunk group are rejected. SIP-controlled trunk groups do not have physical resources; therefore, no mechanism can be used to indicate to the remote switch the non-availability of the trunk group.
Chapter 10 Troubleshooting the Cisco BTS 10200 Softswitch Operating System Failure Cisco Broadband Local Integrated Services Solution Troubleshooting Guide 10-18 OL-5169-01
C H A P T E R 11 Element Management and MIBs This chapter provides an overview of the performance monitoring features and useful SNMP MIB files related to the Cisco uBR7246vxr, Cisco GSR10012, and MTAs. Cisco uBR7246vxr MIBs This section includes descriptions of the following useful SNMP MIBs for the uBR7246VXR: • CISCO-CABLE-SPECTRUM-MIB.my • CISCO-DOCS-EXT-MIB.my • DOCS-CABLE-DEVICE-MIB.my • DOCS-IF-MIB.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs There are 3 flap detectors defined. The flap count (ccsFlapTotal) is increased when any one of the flap detectors is triggered. (1) Registration Flap: A CM may fail the registration process due to not being able to get an IP address. When that happens the CMTS will receive the Initial Maintenance packet from the CM sooner than expected and the CM is considered a flapping modem.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-1 CISCO-CABLE-SPECTRUM-MIB.my Objects (continued) Object Description ccsFlapUpstreamIfIndex The ifIndex of the Cable upstream interface whose ifType is docsCableUpstream(129). The CMTS detects a flapping MTA from its Cable upstream interface. ccsFlapDownstreamIfIndex The ifIndex of the Cable downstream interface whose ifType is docsCableDownstream(128).
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-1 CISCO-CABLE-SPECTRUM-MIB.my Objects (continued) Object ccsFlapMisses Description The number of times the CMTS misses the Ranging request from the MTA. The CMTS issues a Station Maintenance packet every 10 seconds and waits for a Ranging request from the MTA. If the CMTS misses a Ranging request within 25 msec then the Misses count are increased.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-1 CISCO-CABLE-SPECTRUM-MIB.my Objects (continued) Object Description ccsFlapPowerAdjustments The number of times the MTA upstream transmit power is adjusted during station maintenance. When the adjustment is greater than the power adjustment threshold the counter will be increased. The power adjustment threshold is chosen in an implementation- dependant manner.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-1 CISCO-CABLE-SPECTRUM-MIB.my Objects (continued) Object Description ccsFlapCreateTime The time that this entry was added to the table. If an entry is removed and then later re-added, there may be a discontinuity in the counters associated with this entry. This timestamp can be used to detect those discontinuities. ccsFlapRowStatus Controls and reflects the status of rows in this table.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Scheduler QoS Control Group To ensure Quality of Service and fairness, the scheduler needs to control the traffic. This group includes attributes that user can configure how the scheduler controls the traffic and attributes showing the current status of the scheduler admission and rate control. For each Service ID, there is one Quality of Service profile associated with it.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs This table extends the attributes for CMTS MAC interface. It includes attributes of the MTA notification enabling/disabling and the interval of MTA notification sent by the CMTS for a MTA that the Mac interface supports. • CMTS cable modem channel override operation table (cdxCmtsCmChOver) This table may be used to perform downstream/upstream load balancing or failure recovery.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs • CMTS Upstream Channel Table (cdxIfUpChannel) This table contains the additional upstream channel attributes. The additional configurable objects for automated Spectrum Management are the modulation profile and channel width needed for the frequency hop algorithm used for noise mitigation. Another upstream channel attribute is the number of MTAs.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Cisco DOCS Extension MIB Notifications Table 11-2 CISCO-DOCS-EXT-MIB.my Objects Object Description cdxQosCtrlUpAdmissionCtrl The admission control status for minimum guaranteed upstream bandwidth scheduling service requests for this upstream.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-2 CISCO-DOCS-EXT-MIB.my Objects (continued) Object Description cdxQosCtrlUpReservedBW The current total reserved bandwidth in bits per second of this upstream interface. It is the sum of all MTAs' minimum guaranteed bandwidth in bits per second currently supported on this upstream. cdxQosCtrlUpMaxVirtualBW The maximum virtual bandwidth capacity of this upstream interface if the admission control is enabled.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-2 CISCO-DOCS-EXT-MIB.my Objects (continued) Object Description cdxQosIfRateLimitExpWt Weight for exponential moving average of loss rate for weighted excess packet discard algorithm to maintain. The higher value of the weight makes the algorithm more sensitive to the recent bandwidth usage by the Sid.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-2 CISCO-DOCS-EXT-MIB.my Objects (continued) Object Description cdxQosIfRateLimitShpMaxDelay The maximum shaping delay in milliseconds. That is, the maximum amount time of buffering the CMTS will allow for any rate exceeded flow. If the max buffering delay is large, the grants/packets of the flow will be buffered for a longer period of time even though the flow is rate exceeded.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-2 CISCO-DOCS-EXT-MIB.my Objects (continued) Object Description cdxQosIfRateLimitShpGranularity The width in milliseconds of each element in shaping delay queue, that is, the shaping granularity. The shaping granularity is only applied to rate limit algorithm: Token bucket algorithm with shaping. It controls how accurately the algorithm quantizes the shaping delay for a rate exceeded flow.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-2 CISCO-DOCS-EXT-MIB.my Objects (continued) Object Description cdxQosMaxUpBWExcessRequests The number of upstream bandwidth requests which exceeds the maximum upstream bandwidth allowed for a service defined in the Quality of Service profile associated with this Sid. The request which exceeds the maximum upstream bandwidth allowed will be rejected by the upstream's rate limiting process using one of the rate limiting algorithm.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-2 CISCO-DOCS-EXT-MIB.my Objects (continued) Object Description cdxBWQueueMaxDepth The maximum number of requests/packets which the queue can support. cdxBWQueueDepth The current number of requests/packets in the queue. cdxBWQueueDiscards The number of requests/packets discarded because of queue overflow (queue depth > queue maximum depth). cdxCmCpeMacAddress The Mac address to identify a MTA or a Customer Premises Equipment.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-2 CISCO-DOCS-EXT-MIB.my Objects (continued) Object Description cdxCmtsCmStatusValue Current MTA connectivity state. The object extends states in docsIfCmtsCmStatusValue in more details. The enumerations are: offline(1): modem considered offline. others(2): states is in docsIfCmtsCmStatusValue. initRangingRcvd(3): modem sent initial ranging. initDhcpReqRcvd(4): dhcp request received.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-2 CISCO-DOCS-EXT-MIB.my Objects (continued) Object Description cdxIfCmtsCmStatusOnlineTimes The number of times that the modem changes the connectivity state from 'offline' to 'online' over the time period from the modem's first ranging message received by CMTS until now.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-2 CISCO-DOCS-EXT-MIB.my Objects (continued) Object Description cdxIfCmtsCmStatusAvgOnlineTime The average period of time the modem stayed ‘online’ [‘offline’] over the time period from the modem's first ranging message received by CMTS until now.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-2 CISCO-DOCS-EXT-MIB.my Objects (continued) Object Description cdxCmtsCmOnOffTrapInterval The interval for cdxCmtsCmOnOffNotification sent by CMTS for one online/offline state change if cdxCmtsCmOnOffTrapEnable is true.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-2 CISCO-DOCS-EXT-MIB.my Objects (continued) Object Description cdxCmtsCmChOverTimeExpiration The time period to expire a CMTS channel override operation. Within the time period, if the CMTS cannot send out a RNG-RSP message with channel override fields to a MTA specified in the operation, the CMTS will abort the operation. The possible reason is that the MTA does not repeat the initial ranging.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-2 CISCO-DOCS-EXT-MIB.my Objects (continued) Object Description cdxCmtsCmChOverState The status of the specified channel override operation. The enumerations are: • messageSent(1): the CMTS has sent a RNG-RSP message with channel override to the MTA. • commandNotActive(2): the command is not in active mode due to this entry's row status is not in active yet.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-2 CISCO-DOCS-EXT-MIB.my Objects (continued) Object Description cdxCmtsCmMaxCpeNumber The maximum number of permitted CPEs connecting to the modem. The value -1 means to use the default value of maximum hosts per modem in the CMTS cable interface which the modem connects to and the value is defined in cdxCmtsCmDefaultMaxCpes in the cdxCmtsMacExtTable. The value 0 means no maximum limit.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-2 CISCO-DOCS-EXT-MIB.my Objects (continued) Object Description cdxCmtsCmOnOffNotification This notification indicates that the MTA is coming online and going offline. A notification is sent from the CMTS for a MTA status changing to online or offline within the interval specified in cdxCmtsCmOnOffTrapInterval.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs • Link Level Control Filtering group (docsDevFilterLLC) LLC (Link Level Control) filters can be defined on an inclusive or exclusive basis: CMs can be configured to forward only packets matching a set of layer three protocols, or to drop packets matching a set of layer three protocols. Typical use of these filters is to filter out possibly harmful protocols.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs This says that a packet which matches a filter with policy id 1, first has TOS policy 1 applied (which might set the TOS bits to enable a higher priority), and next has the IPSEC policy 3 applied (which may result in the packet being dumped into a secure VPN to a remote encryptor). Policy ID 0 is reserved for default actions and is applied only to packets which match no filters in docsDevIpFilterTable.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-3 DOCS-CABLE-DEVICE-MIB.my Objects (continued) Object Description docsDevNmAccessIpMask The IP subnet mask of the network management stations. If traps are enabled for this entry, then the value must be 255.255.255.255. docsDevNmAccessCommunity The community string to be matched for access by this entry. If set to a zero length string then any community string will match. When read, this object SHOULD return a zero length string.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-3 DOCS-CABLE-DEVICE-MIB.my Objects (continued) Object Description docsDevSwAdminStatus If set to upgradeFromMgt(1), the device will initiate a TFTP software image download using docsDevSwFilename. After successfully receiving an image, the device will set its state to ignoreProvisioningUpgrade(3) and reboot.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-3 DOCS-CABLE-DEVICE-MIB.my Objects (continued) Object Description docsDevServerBootState If operational(1), the device has completed loading and processing of configuration parameters and the CMTS has completed the Registration exchange. If disabled(2) then the device was administratively disabled, possibly by being refused network access in the configuration file.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-3 DOCS-CABLE-DEVICE-MIB.my Objects (continued) Object Description docsDevEvThrottleAdminStatus Controls the transmission of traps and syslog messages with respect to the trap pacing threshold. unconstrained(1) causes traps and syslog messages to be transmitted without regard to the threshold settings.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-3 DOCS-CABLE-DEVICE-MIB.my Objects (continued) Object Description docsDevEvPriority The priority level that is controlled by this entry. emergency(1) alert(2) critical(3) error(4) warning(5) notice(6) information(7) debug(8) These are ordered from most (emergency) to least (debug) critical. Each event with a CM or CMTS has a particular priority level associated with it (as defined by the vendor).
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-3 DOCS-CABLE-DEVICE-MIB.my Objects (continued) Object Description Vendors will provide their own enumerations for the following objects. The interpretation of the enumeration is unambiguous for a particular value of the vendor's enterprise number in sysObjectID. docsDevEvId For this product, uniquely identifies the type of event that is reported by this entry.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-3 DOCS-CABLE-DEVICE-MIB.my Objects (continued) Object Description docsDevFilterIpStatus Controls and reflects the status of rows in this table. Specifying only this object (with the appropriate index) on a CM is sufficient to create a filter row which matches all inbound packets on the ethernet interface, and results in the packets being discarded. docsDevFilterIpIfIndex (at least) must be specified on a CMTS to create a row.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-3 DOCS-CABLE-DEVICE-MIB.my Objects (continued) Object Description docsDevFilterIpDmask A bit mask that is to be applied to the destination address prior to matching. This mask is not necessarily the same as a subnet mask, but 1's bits must be left-most and contiguous. docsDevFilterIpProtocol The IP protocol value that is to be matched. For example: icmp is 1, tcp is 6, udp is 17. A value of 256 matches ANY protocol.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-3 DOCS-CABLE-DEVICE-MIB.my Objects (continued) Object Description docsDevFilterPolicyPtr This object points to a row in an applicable filter policy table. Currently, the only standard policy table is docsDevFilterTosTable. Per the textual convention, this object points to the first accessible object in the row. E.g.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs If implemented as read-only, entries are created based on information in REG-REQ MAC messages received from MTAs (CMTS implementation), or based on information extracted from the TFTP option file (cable modem implementation). In the CMTS, read-only entries are removed if no longer referenced by docsIfCmtsServiceTable.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-4 DOCS-IF-MIB.my Objects Object Description docsIfDownChannelId The CMTS identification of the downstream channel within this particular MAC interface. If the interface is down, the object returns the most current value. If the downstream channel ID is unknown, this object returns a value of 0. docsIfDownChannelFrequency The center of the downstream frequency associated with this channel.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-4 DOCS-IF-MIB.my Objects (continued) Object Description docsIfDownChannelPower At the CMTS, the operational transmit power. At the CM, the received power level. May be set to zero at the CM if power level measurement is not supported. If the interface is down, this object either returns the configured value (CMTS), the most current value (CM) or the value of 0.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-4 DOCS-IF-MIB.my Objects (continued) Object Description docsIfUpChannelTxBackoffStart The initial random backoff window to use when retrying transmissions. Expressed as a power of 2. A value of 16 at the CMTS indicates that a proprietary adaptive retry mechanism is to be used. See the associated conformance object for write conditions and limitations.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-4 DOCS-IF-MIB.my Objects (continued) Object Description docsIfSigQSignalNoise Signal/Noise ratio as perceived for this channel. At the CM, describes the Signal/Noise of the downstream channel. At the CMTS, describes the average Signal/Noise of the upstream channel. docsIfSigQMicroreflections Total microreflections including in-channel response as perceived on this interface, measured in dBc below the signal level.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-4 DOCS-IF-MIB.my Objects (continued) Object Description docsIfCmStatusLostSyncs Number of times the CM lost synchronization with the downstream channel. docsIfCmStatusInvalidMaps Number of times the CM received invalid MAP messages. docsIfCmStatusInvalidUcds Number of times the CM received invalid UCD messages. docsIfCmStatusT1Timeouts Number of times counter T1 expired in the CM.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-4 DOCS-IF-MIB.my Objects (continued) Object Description docsIfCmtsUcdInterval The interval between CMTS transmission of successive Upstream Channel Descriptor messages for each upstream channel at this interface. docsIfCmtsMaxServiceIds The maximum number of service IDs that may be simultaneously active. docsIfCmtsInvitedRanging Attempts The maximum number of attempts to make on invitations for ranging requests.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-4 DOCS-IF-MIB.my Objects (continued) Object Description docsIfCmtsCmStatusTiming Offset A measure of the current round trip time for this CM. Used for timing of CM upstream transmissions to ensure synchronized arrivals at the CMTS. Units are in terms of (6.25 microseconds/64). Returns zero if the value is unknown. docsIfCmtsCmStatusEqualization Equalization data for this CM.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-4 DOCS-IF-MIB.my Objects (continued) Object Description docsIfCmtsServiceQosProfile The index in docsIfQosProfileTable describing the quality of service attributes associated with this particular service. If no associated docsIfQosProfileTable entry exists, this object returns a value of zero. docsIfCmtsServiceCreateTime The value of sysUpTime when this entry was created.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Table 11-4 DOCS-IF-MIB.my Objects (continued) Object Description docsIfCmtsModScrambler Indicates if the scrambler is employed. docsIfCmtsQosProfile Permissions This object specifies permitted methods of creating entries in docsIfQosProfileTable. CreateByManagement(0) is set if entries can be created using SNMP. UpdateByManagement(1) is set if updating entries using SNMP is permitted.
Chapter 11 Element Management and MIBs Cisco uBR7246vxr MIBs Cisco Broadband Local Integrated Services Solution Troubleshooting Guide 11-46 OL-5169-01
