Avaya™ Interactive Response Release 2.0 Troubleshooting issue 1.0 Publication Date: April 2006 Issue 1.
© 2005 - 2006 Avaya Inc. All Rights Reserved. Notice While reasonable efforts were made to ensure that the information in this document was complete and accurate at the time of printing, Avaya Inc. can assume no liability for any errors. Changes and corrections to the information in this document might be incorporated in future releases. Documentation disclaimer Avaya Inc.
Troubleshooting overview Be aware that there might be a risk of unauthorized intrusions associated with your system and/or its networked equipment. Also realize that, if such an intrusion should occur, it might result in a variety of losses to your company (including but not limited to, human/data privacy, intellectual property, material assets, financial resources, labor costs, and/or legal costs).
Contents Troubleshooting ..................................................................................................................... 6 Troubleshooting overview ............................................................................................... 6 Requirements for successful operations....................................................................... 6 Possible malfunctions and errors..................................................................................
Contents Troubleshooting An Avaya IR system interacts with other systems and relies on them for critical functions. Consequently, troubleshooting may involve testing connections and checking other systems where databases, speech functions, and host services reside. This section guides you in resolving many Avaya IR system problems and includes information on basic LAN, server, and host troubleshooting.
Contents • An IP-enabled DEFINITY or MultiVantage system receives calls from the PSTN, converts them to packet-based signals, and sends them to the IR system over a Local Area Network (LAN) connection. • Calls come from the PSTN directly to the IR system through digital connections. For successful voice response operations, all of the connections described above and the telephony network that supports them–including central offices–must be working and free from errors.
Contents • Digital lines and LAN connections that bring calls in from and send calls out to the public switched telephone network (PSTN) • Connections between any MultiVantage systems and the IR system • Connections from the back of the IR system to other devices, and to the LAN • LAN connections between the IR system and servers that provide speech functions, database information, or both A breakdown in any of these connections can affect voice response operations.
Contents • TCIP/IP connections between the IR system and a server might be set up incorrectly, so that required data is not available to callers. Application errors Applications manage voice response functions, so errors can be devastating to operations. For instance, an application may call for the playing of recorded speech that does not exist, or try to access the wrong server for speech.
Contents • LAN overloading may result from competition with other processes for LAN capacity. The result can be delays and breaks in the availability of required data and functions. If the call load increases beyond the capacity of the IR system, call handling problems are likely to occur. A new system, or re-routing of calls, is generally required. Speech The IR system provides information to callers through recorded or generated speech.
Contents Inadequate or expired feature licensing Note: Renaming the IR system may cause loss of feature licensing. Calls not terminating on the IR system. Poor communication or no communication with required servers across the LAN Affected features are not functioning, or are not functioning as expected. Callers hear a fast busy signal.
Contents Troubleshooting procedure If a problem develops with voice response operations, follow this general procedure to resolve it: 1. Go to the Message Log Report screen (Reports > Message Log Report) and check for messages about events related to the problem. Events may generate alarms. The level of the alarm (critical, major, or minor) indicates the severity of the problem. 2. Follow repair procedures for any events related to the problem.
Contents Using IR system events The first step in the general troubleshooting procedure is to check for messages about events related to the problem. This topic provides more information about troubleshooting based on events. Events on the Avaya IR system are logged, and alarms are generated when those events cause or might cause a problem with voice response operations. To troubleshoot using Avaya IR system alarms and errors: 1.
Contents • Services (voice response applications) assigned to channels • Service state The following table shows state descriptions and their meanings. State Meaning INSERV In service MANOOS Manually out of service FOOS Facility out of service BROKEN Not functioning, possibly needing replacement Monitoring live operations Use the sysmon command to observe voice response operations as they occur. You can see calls coming in, digits entered by callers, and line conditions, such as off-hook.
Contents • Validation Test System (VTS) to test and validate major hardware components • OpenBoot Diagnostics system to perform root cause failure analysis on various IR devices • PROM Diagnostics to check system processes, such as the error rate and type for Ethernet packets Sun Fire V210 and V240 Servers Administration Guide explains how to run these diagnostic tests on the Sun Fire V240 platform.
Contents Investigating operations problems Problems central to voice response functions can affect business operations and may result in missed calls and caller frustration. Most of the problems described in this section require prompt attention. To investigate these problems, you should have a good understanding of the Requirements for successful operations on page 6. Call-handling problems Call handling problems include issues related to responding to and transferring voice and fax calls.
Contents 4. If cards are not in service, either try to restore them or contact your Avaya support representative. See Restoring cards and channels on page 48 for procedures that may bring cards back into service. 5. If cards are in service, go to the Channel Services screen (Voice Equipment > Voice Services > Channel Services) and verify that required voice response applications are assigned to the appropriate channels. 6.
Contents Look for messages that occurred just before and at the time when calls began dropping. If calls are handled by VoIP, look for the messages VOIP_DISABLED_CALL_PROC or VOIP_CALL_FORCE_CLEARED. 2. Type who -rpb and press Enter to display a log of system processes. 3. Search for different time stamps on the processes. A recent date different from most of the others may indicate that the process respawned. 4.
Contents No DTMF tones (WINK protocol) If voice response applications are not responding correctly to caller input, you may suspect that DTMF tones (the tones that identify the called number) are missing. For calls handled with the WINK start protocol, run a trace to determine if the NMS card is receiving the tones. To set up the trace: 1. Set the following values in the cta.cfg file: ― Tracemode=1 ― Tracemask=allevt ― Tracefile=cta.log 2. Stop and start the voice system. 3. Review the ctdaemon file.
Contents MESG: Tue May 13 17:08:48 2003 | pid=580 tid=578 ctahd=80000001 (CTATEST) uid=0 tag=14001 sev=0 | ? Msg:854D Ch#01 Obj:0000 Np=16 Nb=0 0033 0000 0000 0000 0000 0000 ... MESG: Tue May 13 17:08:48 2003 | pid=580 tid=578 ctahd=80000001 (CTATEST) uid=0 tag=4006 sev=0 | DISPEVT: ? NCCEVN_RECEIVED_DIGIT (1c201d) (val=33) objhd=1 src=1c dst=2000 time=235a9bd7 uid=0 size=0 buf=0 This trace output shows that digits were sent.
Contents • There is no fax machine at the remote number. Once you have checked these two possibilities, troubleshoot fax problems using the information in this section. Locating fax errors For internal errors: 1. Check for fax errors by taking one of the following actions: ― Go to the Message Log Report (Reports > Message Log Report) and check for fax errors or ― Type trace date FAXOOC sbFaxProc NMSIP chan # area all level all and press Enter. 2. Provide the output to technical support.
Contents -20 Subprog to sbFaxHpr failed (internal). -21 IRAPI call failed (internal). -23 Wrong subprog message was received (internal). -24 Max. sbFaxHpr instances was reached (internal). Reviewing fax repair procedures When problems arise with fax operations, the Message Log Report screen might display various events related to fax operations. The Explain text and help topics for these events include suggested repair procedures.
Contents No response for application with host interface A voice response application that relies on a host system for data might receive no answer intermittently or consistently. To resolve the problem: 1. Go to the Message Log Report screen (Reports > Message Log Report) and check for events related to the trouble. The event HOST001 (HOST_NORESP) should appear in the log. 2. Follow the repair procedure for the event.
Contents 1. Go to the Speech Resource Status screen (Features > Speech and DPR Administration > Display Status > Speech Resource Status), select a speech resource type from the drop-down list, and select Submit. The configuration listing for the selected type of speech resource is displayed. 2. Review the speech resource listing to verify that the resource is administered and is in the in service (IN SERV) state. 3. Repeat the previous steps with all types of speech resources. 4.
Contents Go to the Speech Recognition or DPR Configuration screen (Feature Packages > Speech and DPR Administration > Administration > Speech Proxy Administration > Speech Recognition Administration > Speech Recognition or DPR Configuration) and administer a server. Speech server not running For earlier releases of OSR, run the command swisvc -start to restart the speech server after every Avaya IR system re-start. Starting with OSR 2.
Contents • If you want to use the OSR 1.x confidence score scale, set the attribute swirec_backward_compatible_confidence_scores to 1.1 in the OSR server user configuration file. Refer to the Scansoft OSR 2.x reference manual for further information. • If you want to use the OSR 2.x confidence score but you want to increase the recognition rate, try decreasing the confidencelevel property in the /vs/data/vxml/defaults.xml file on the Avaya Interactive Response system, then restart the voice system.
Contents Checking the server connection See Troubleshooting speech server disconnections on page 37 for a complete procedure on checking and testing the connection. Checking for errors To check for errors: 1. Go to the Message Log Report screen (Reports > Message Log Report) and check for messages related to the trouble. 2. Enter the following commands as needed to analyze operations: ― Type trace tsm chan all | tee /tmp/trace.out and press Enter to trace all levels of operations for the call.
Contents • RTP uses a base port of 10,000 The MRCP configuration file Installing the MRCP feature on Avaya IR creates a configuration file (/vs/sproxy/cfg/mrcp.cfg) and the default settings shown in the following table. The configuration file sets parameters for the ports used for signaling and the timing intervals that Avaya IR uses to determine if it has lost touch with speech resource servers. Under most circumstances, there is no need to change these default settings.
Contents RTSPNoActivityTimeout The number of seconds Avaya IR will wait for a response from an MRCP server before sending a DESCRIBE message as a test of the connection. 15 If a second timeout interval elapses with no response, Avaya IR decides that the server or the connection is out of service. It then places the server into the maintenance state called FOOS (facility out-of-service), logs the event, and raises an alarm.
Contents BuiltinLocale A setting that indicates the language to be used for en-US built-in ASR grammars such as numbers or phone numbers. In Avaya IR release 1.2.1 and release 1.3, the IBM WVS supports only one language at a time but does offer several different languages. Refer to the IBM WVS documentation for the full range of language options. Change this setting if the MRCP server supports a language other than US English.
Contents ASR Servers Server Name Base Port Numb er IBM WVS 554 /media/reco gnizer Scansoft SWMS /media/spe echrecognizer 4900 Nuance MRCP Server /recognizer 554 TTS Servers Server Name Base Port Numb er IBM WVS /media/synt hesizer 554 Scansoft SWMS /media/spe echsynthesizer 4900 Nuance MRCP Server /synthesizer 554 Note: The suffixes above are automatically appended when the above media servers are configured through Web Admi
Contents Text-to-Speech output is not heard If TTS output does not play in an application, use the following procedure to investigate the problem. To investigate the problem: 1. Check for allocation errors (TTS008) in the error log. This error indicates that a request for a TTS resource was denied, and the associated TTS will not be heard. This error occurs when there are no TTS resources in service.
Contents Text-to-Speech problems This section describes how to find and resolve problems related to the Text-to-Speech (TTS) feature. Speechify or RealSpeak TTS port state stays INSERV when LAN is down The SpeechWorks Speechify or RealSpeak proxy software on the Avaya IR system does not detect when the LAN connection to the speech server is inactive and leaves the TTS port state as INSERV. As a result, callers hear silence, since the port on the IR system is still in service.
Contents d) The system displays the Component Services window. e) On the Tree tab, select Services (Local). f) Select the Text-to-Speech service from the list of services in the lower-right window. g) From the Action menu, select Stop. h) The Status of the service is blank. i) From the Action menu, select Start. j) The Status of the service is started. k) Close the Component Services window.
Contents Database problems If you are using databases on the LAN, communications problems with those databases may affect voice operations. Troubleshooting database server disconnections on page 38 covers what to do when the IR system is not communicating with the database server at all. The following topics explain how to check on less serious problems with databases. Checking JDBC operations Use the following commands to check JDBC function: • netstat -a lists port usage.
Contents 3. Redefine the database table storage, if necessary. Note: Contact your internal database administrator or your database vendor for help with this and other database tasks. A word about the Tomcat server log In Avaya IR Release 2.0, the Web Administration tool uses Tomcat as both the Web server and servlet engine. Tomcat periodically writes data to a log on the Avaya IR server.
Contents network is to enable the DEFINITY or MultiVantage system and the Avaya IR system to communicate with each other using the UDP and TCP protocols on the network.
Contents a) Go to the appropriate proxy configuration screen and make the required corrections. To correct speech recognition configuration errors, go to the Speech Recognition or DPR Configuration screen (Feature Packages > Speech and DPR Administration > Administration > Speech Proxy Administration > Speech Recognition or DPR Configuration).
Contents The connection between the Avaya IR system and the database server is tested and the results are reported. If the connection is not working, the related error message is included in the output. 5. Continue checking settings, testing connections, and making corrections for all DIPs that communicate with the database server. 6.
Contents Pinging server connections The ping command indicates whether a remote host can be reached. It can also display statistics about packet loss and delivery time. The ping command is available through the Solaris operating system. Use it with the attributes shown in the table that follows. Attribute Function -d Set the SO_DEBUG socket option. -l Send the packet to the given host and back again. -L Turn off loopback of multicast packets. -n Display the network addresses as numbers.
Contents • When traffic on the LAN is very heavy, some packets might be lost because the server cannot keep up with the flow. To better understand the results, you might want to seek support from your Avaya support representative when running the LAN utilities. Detecting incorrect IP addresses (arp) The arp command provides information about Ethernet/IP address translation. You can use the command to detect systems on the LAN that are configured with an incorrect IP address.
Contents -r Display the routing tables. -s Display the per-protocol statistics. -v Display additional information for the sockets and the routing table. -I interface Display the state of a particular interface. -M Display the multicast routing tables. -P protocol Limit the display of statistics or state of all sockets to those applicable to protocol. Displaying packet route (traceroute) Use the traceroute command to display the route that packets take when going to a remote system.
Contents Checking hardware Hardware failures and malfunctions can stop or interfere with voice system operations. This section explains how to check various types of hardware connections and components. Resolving a problem when the monitor does not display It has been observed on Sun Fire headless systems (such as the 280R and the V240), that when the console is left disconnected for an extended period of time, the system can stop sending output to the port where the console monitor would be connected.
Contents Label Function and troubleshooting considerations 1 Power connector - Cable provides power to the IR system. Disconnection from the plug results in loss of power and function. 2 PCI card slots - Cables connect NMS cards to the MultiVantage (DEFINITY) system or to digital telephony lines. Problems here may interfere with receiving and handling calls. 3 USB connectors (four) - Two of these connectors are reserved for the keyboard and mouse that are part of the country kit.
Contents Sun Fire 280R cable connections The following figure shows where cables connect to the back of the back of the Sun Fire 280R platform. Label Function and Troubleshooting Considerations 1 PCI card slots - Cables connect NMS cards to the MultiVantage (DEFINITY) system or to digital telephony lines. Problems here may interfere with receiving and handling calls. 2 Serial connector a - The cable connects the video monitor to the IR system.
Contents 8 Twisted-pair Ethernet connector - Cable connects the IR system to the LAN. Problems here may interfere with access to voice response applications, databases, proxy speech servers, and other IR system components that reside on servers on the LAN. If VoIP is in use, a loose connection here may cause problems with call processing. 9 FC-AL 10, 12 Power connectors - Cable provides power to the IR system. Disconnection from the plug results in loss of power and function.
Contents Checking NMS card configuration Check the configuration of the NMS card or cards with the commands described in the table that follows. Commands Functions nmsboards Identifies NMS digital telephony boards, their types (E1 or T1), and their slot numbers. pcidev Identifies board type and communicates with PCI files. Can be run without having to configure an NMS board or run ctdaemon.
Contents Performing root cause failure analysis The Sun OpenBoot Diagnostics system performs root cause failure analysis on various IR devices by testing internal registers, confirming subsystem integrity, and verifying device functionality. Refer to the service manual for your platform: • Sun Blade 150 Service Manual • Sun Fire 280R Server Service Manual These documents are available in Avaya IR System Help (under "Print documents") or from the Sun Web site (http://www.sun.com).
Contents Restoring NETOOS cards and channels The NETOOS state indicates that the card or channel was taken out of service by some network or physical channel error. This state refers only to channels that are defined as PRI protocol. To restore a card in the NETOOS state: 1. Go to the Message Log Report screen (Reports > Message Log Report) and review any messages related to the particular card or channel. 2.
Contents 2. Check connections and indicators on the back of the IR system and reseat the connection, if necessary: a) Check the physical connection to the card and determine if it is seated correctly. The card should not have worked its way out of the connection. See Checking cable connections on page 43 for more information. VoIP may use a network interface (NIC) card that is different from the one used by other web-based processes for the Avaya IR system.
Contents 3. Check the connection for the BROKEN card or channel on the back of the IR system. The card or channel should not have worked its way out of the connection. See Checking c on page 43able connections for more information. 4. If the connection is loose, re-seat it. 5. If you have reseated the connection, go to the Display Equipment screen (Configuration Management > Voice Equipment > Display Equipment) to see if the card or channel is now in the INSERV state. 6.
Contents 4. If the card is still not in service, type display card card _number and press Enter and verify that the card is found. Note:i If there are problems with licensing, there may be no usable channels on the card, and the card will not be found when the display card command is run. 5. If the card is not found, go to the Feature Licensing screen (Configuration Management > Feature Licensing) and verify that there are enough Right-to-Use licenses (RTUs) for the channels supported by the card. 6.
Contents Gathering data on card operations Follow the procedures in this section to learn more about the exact nature of the problem. Then contact your Avaya support representative for assistance in troubleshooting the card. Displaying data on NMS cards To display information about NMS cards: 1. Type trunkmon-b card# and press Enter to display information about a specific card. ― card# represents the number of the card you want to check.
Contents Performing root cause failure analysis The Sun OpenBoot Diagnostics system performs root cause failure analysis on various IR devices by testing internal registers, confirming subsystem integrity, and verifying device functionality.
Contents a a a a W W W W W W W W p p p p p p p p p p p p luo luo luo luo l l l l l l l l 4152 5186 6220 7254 16 1050 2084 3118 4152 5186 6220 7254 1034 1034 1034 1034 1034 1034 1034 1034 1034 1034 1034 1034 /dev/dsk/c1t0d0s4 /dev/dsk/c1t0d0s4 /dev/dsk/c1t0d0s4 /dev/dsk/c1t0d0s4 /dev/dsk/c1t1d0s4 /dev/dsk/c1t1d0s4 /dev/dsk/c1t1d0s4 /dev/dsk/c1t1d0s4 /dev/dsk/c1t1d0s4 /dev/dsk/c1t1d0s4 /dev/dsk/c1t1d0s4 /dev/dsk/c1t1d0s4 Where: o u l c p m W a M D F S R = = = = = = = = = = = = = Replica active prior
Contents The system displays detailed information for each metadevice (virtual device) similar to the following sample output: d1: Mirror Submirror 0: d11 State: Okay Submirror 1: d12 State: Okay Pass: 2 Read option: roundrobin (default) Write option: parallel (default) Size: 4202688 blocks d11: Submirror of d1 State: Okay Size: 4202688 blocks Stripe 0: Device Spare c1t0d0s1 d12: Submirror of d1 State: Okay Size: 4202688 blocks Stripe 0: Device Spare c1t1d0s1 Start Block 0 Start Block 0 Dbase State No
Contents 2. At the command prompt, enter stop_vs. The voice system stops. 3. Enter mirror_admin detach. The system displays the following prompt: Prompt for hard disk drive to detach 4. Type the number that corresponds to the hard disk drive to detach and press Enter. The system detaches all submirrors from the failed hard disk drive. 5. Physically remove the failed hard disk drive.
Contents Restoring the system if both hard disk drives fail If both hard disk drives fail, after replacing the hard disk drives, you must do one of the following: • Rebuild your system from CD media. • Rebuild your system by restoring the system from a backup, which may be preferable if you have applications, data, and feature administration that also needs to be restored.
Contents Issue 1.
F Index Fax problems • 20 Fax text or file not found • 22 G Gathering data on card operations • 52, 53 Gathering information on a problem • 13 A H A word about the Tomcat server log • 36 Activating disk mirroring for a new hard disk drive • 57 All calls dropped • 17 All ports BROKEN on speech server • 24 Application errors • 9 Automatic Speech Recognition fails • 32 Hardware malfunctions and failures • 8 Host interaction problems • 16, 17, 22 Host sessions recover repeatedly • 22 C Call-handling pro
Index Restoring BROKEN NMS and VoIP cards • 50 Restoring cards and channels • 17, 47, 48 Restoring FOOS cards and channels • 49 Restoring MANOOS cards and channels • 48 Restoring NETOOS cards and channels • 49 Restoring the system if both hard disk drives fail • 58 Reviewing fax repair procedures • 22 Reviewing the Display Equipment screen • 13 Running the metastat command • 55 S Speech • 10 Speech and TTS problems • 23 Speech delayed or cut off • 23 Speech not playing • 26 Speech recognition not availabl
