Cisco Performance Visibility Manager 1.0 Technical Implementation Guide Corporate Headquarters Cisco Systems Inc. 170 West Tasman Drive San Jose, Ca 95134-1706 USA http://www.cisco.com/en/US/products/sw/netmgtsw/index.
Corporate Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 526-4100 European Headquarters Cisco Systems International BV Haarlerbergpark Haarlerbergweg 13-19 1101 CH Amsterdam The Netherlands www-europe.cisco.com Tel: 31 0 20 357 1000 Fax: 31 0 20 357 1100 Americas Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA www.cisco.
TECHNICAL IMPLEMENTATION GUIDE Cisco Performance Visibility Manager 1.0 INTRODUCTION ............................................................................................................................................................................4 DATA COLLECTION AND TRAFFIC ANALYSIS ..........................................................................................................................4 TRAFFIC ANALYSIS INCLUDING TOP-N ANALYSIS .................................................
INTRODUCTION The Cisco® Performance Visibility Manager is an enterprise-level, centralized network management tool that enhances the Cisco Network Analysis Module (NAM) for Cisco Catalyst® 6500 Series switches, 7600 Series routers and Branch Routers series. Cisco PVM provides a centralized and integrated End-to-End (E2E) network view, by aggregating and correlating information from multiple NAMs that are strategically deployed in the network.
NAM GUI Drill down Cisco PVM provides you the ability to drill down into the NAM GUI for more detailed and efficient troubleshooting, once it has been identified with the help of PVM’s traffic analysis and monitoring components. CiscoWorks and LDAP Integration Cisco PVM provides integration with CiscoWorks DCR and LDAP directories and you can efficiently administer your network equipment, users and credentials. Cisco Internal Use Only Copyright © 2006 Cisco Systems, Inc. All rights reserved.
DEPLOYMENT CONSIDERATIONS AND PLANNING Cisco PVM works in conjunction with the Cisco Network Analysis Modules (NAMs) to provide you in-depth visibility into your network traffic. In Cisco PVM 1.0, traffic statistics are collected from the NAMs and their associated Switches/Routers. Technologies such as RSPAN and Netflow can be used to gather data from other devices. Cisco PVM communicates with the NAMs and their associated devices through SNMP.
5. Create the appropriate datasource groupings of NAMs and/or Switch/Routers to aggregate data from. 6. Use the Traffic Analysis features of Cisco PVM to identify or troubleshoot the problem. Basic questions about the Problems to be addressed To decide how to deploy Cisco PVM and Cisco NAM-1/NAM-2 in the network, first answer some questions that address the purpose and needs of the administrator and how Cisco PVM and the Cisco NAMs can provide an accurate analysis.
• Access Layer: Place Cisco NAMs at the access layer only if critical clients are required to be monitored. IP phones, for example, can be monitored for latency or for adequate response to and from Cisco CallManagers. • WAN Edge: Place Cisco NAMs at the WAN edge to gather WAN statistics from Optical Services Module (OSM) or FlexWAN interfaces, or to collect NetFlow statistics on remote NetFlow-enabled routers.
Create the Datasource Groups (DSGs) in Cisco PVM Since Cisco PVM collects information from multiple NAMs, and each NAM can be monitoring multiple datasources, you have to group these datasources together in Cisco PVM to view useful aggregated data. This is essential for aggregation, and also a requirement to perform traffic analysis and view reports in Cisco PVM. This is described in detail in the PVM Setup section.
USAGE SCENARIOS After following the workflow of installing Cisco PVM, the next step is to understand the usage of the statistics provided by Cisco PVM so you can utilize it to monitor your network. This section provides details on Cisco PVM and NAM setup from configuring NAM to setting up the data source groups using the Cisco PVM GUI. This section also provides you with scenarios to help you understand and use PVM.
e. Log into the Web application, configure the SPAN sessions, and enable data collection such as applications, hosts and conversations. See the following user guide for Cisco NAM.: http://www.cisco.com/en/US/products/sw/cscowork/ps5401/products_user_guide_list.html Tip: Note that Cisco PVM can only collect traffic information that the NAMs are collecting. To see traffic data for a given NAM, ensure that the NAM is collecting the statistic that you require.
To add a user, click Add and fill in the appropriate information in the window shown. For more details on configuring users through the GUI, see the User Guide. User management through LDAP Cisco PVM provides the user the ability to manage authentication and authorization through a LDAP server. When configured, Cisco PVM will use the LDAP protocol to communicate with the LDAP server whenever user authentication or authorization is necessary in PVM.
• • • • ldap.auth.scheme=ssl ldap.account.name=admin ldap.server.name=ware.trendium.com ldap.server.port=636 Note: The parameters such as ldap.account.name, ldap.server.name and ldap.server.port are relative to the test environment. The PVM administrator needs to obtain these parameters from LDAP administrator. For SSL communication with the LDAP server, you need to import the public key from the LDAP server.
Cisco PVM allows the user to add an individual NAM and its associated device through the GUI. Click the Setup Tab and select the NAMs menu item to see the list of NAMs. Click Add to add a NAM and its device. Note: 1. You can add only the NAM and add the Switch/Router later. In this case, Cisco PVM collects information from the NAM and display traffic statistics for the NAM datasources. No information is collected from the associated Switch/Router till you add the Switch/Router. 2.
set to be 50 ms. This might not be appropriate for your network topology. Ensure that this value is appropriate based on your knowledge of the network. 5. While Cisco PVM automatically determines the type of NAM being added, it relies on the user’s specification of the Switch/Router device type. Ensure that you select the appropriate resource type. Select NM_ROUTER for ISR’s, NAM_ROUTER for the 7600 Series router and NAM_SWITCH for the 6500 Series switch. 6.
5. Any problems encountered during the import process are reported in the Alerts window. 6. Remember to refresh the Alerts window periodically as well to see the latest list of alerts. Import File Formats Cisco PVM supports the DCR v3 Export file format and a user-defined format that is based on the tokens found in the DCR v3 export file.
Note: 1. Remember to wait for at least one minute for the devices to be imported. Also, remember to refresh the NAM list page to view the latest list of NAMs in Cisco PVM. 2. When using a user-defined csv file to do the import, it is critical to include the header line. Without the header, Cisco PVM cannot make sense of the values in the file. 3. Through the GUI, Cisco PVM does not allow the user to add a Switch/Router without adding a NAM.
NAM Type DSG: This type of DSG allows the user to group NAM datasources Switch/Router Type DSG: This type of DSG allows the user to group Switch/Router datasources Depending on the type of DSG you select, the devices and datasources for the appropriate type are shown. You can select the device, click the right arrow to list the datasources for that device, and then select the datasources to add to the group.
Usage Scenarios: Gather Statistics and Test Monitoring and Troubleshooting After completing the deployment planning and configuration for the Cisco NAMs and Cisco PVM, you can gather statistics and test their monitoring and troubleshooting capabilities. The statistics to gather depend on your goals. To optimize the monitoring capacity of Cisco PVM, enable statistics collections only for the areas of interest rather than enabling all collections at once on the Cisco NAM.
c d f e g h Step 1 (Contd) c. Type the Name d. Select the NAM Type in the Type dropdown e. Select the device. f. Click the right arrow to see the datasources from the device. g. Select the appropriate datasource and click the down arrow to add the datasource to the group. h. After you have added all the datasources, click OK to create the datasource group. Cisco Internal Use Only Copyright © 2006 Cisco Systems, Inc. All rights reserved.
When you click the Monitor Tab, a Network Overview report is automatically launched for the first DSG in the list. Use the drop downs to select the appropriate DSG, View and Report a b c d The report shows the Cumulative Rates for the Applications. If you would like to view the TopN, you can select TopN from the drop down list and click on Refresh. You can also click the arrows next to the metric to sort the table. Step 2. Go to the Monitor tab and select the Applications Report Suite. a.
a The list of hosts that were using this application protocol is shown to aid in monitoring. b d c Step 3. If you find a protocol that is using excessive bandwidth, you can find out who is using it. a. Click the Protocol of interest. b. PVM lists the hosts that were using the protocol and the amount of traffic they generated. c. You can analyze the host in detail by clicking the Host IP. d.
a b c d e f Step 4. You can also schedule these reports to be run at a given time for later perusal. a. Click the Reports Tab. By default the Applications Report page is shown. b. Select the report period. c. Select the datasource group and specify the view type. d. Type a report name. e. Schedule the report to run at a given time. f. Click Run. Cisco Internal Use Only Copyright © 2006 Cisco Systems, Inc. All rights reserved.
b a c d e Step 4. (Contd.) a. Scheduled reports are visible from the View Schedules menu item. b. Click the Schedules tab to see scheduled reports. c. Reports that have already been run are available in the Completed tab. d. Click the report hyperlink to view it. e. You can also view reports of a particular type by selecting the appropriate report suite in the Archived Reports menu section. Cisco Internal Use Only Copyright © 2006 Cisco Systems, Inc. All rights reserved.
Scenario 2: Proactive Monitoring Network engineers receive calls to troubleshoot user issues. They would like to proactively monitor the network and troubleshoot issues before users become aware of them. Assume that users are complaining of intermittent slow response times from a particular server.
c d e f g i Step 1. (Contd.) c. Type a Name for the ART Group. d. Type the appropriate Report Interval e. Select the NAM from the list. f. Select the appropriate datasource from the NAM g. Click Add h. Repeat steps to add all the datasources. i. Click OK. Cisco Internal Use Only Copyright © 2006 Cisco Systems, Inc. All rights reserved.
a b c Step 2. Create a Threshold for the Application Response Time of the server you are interested in. a. Click the Setup tab b. Click the Thresholds menu item. c. Click Add Cisco Internal Use Only Copyright © 2006 Cisco Systems, Inc. All rights reserved.
f d e i j Step 2. (Contd.) d. Type the Name for the Threshold. e. Select the severity of the Alert to be issued. f. Select ART from the Statistics list. g. Select the Average Response Time metric. h. Select the ART Group you just created. i. Type the IP Address of the server you want to monitor. j. Click OK to create the Threshold. g h Cisco Internal Use Only Copyright © 2006 Cisco Systems, Inc. All rights reserved.
a b Step 3. PVM will now start base-lining the Average Response time for that server. If the thresholds are crossed, it issues an alert. a. To view alerts, click Alerts tab. b. To view a specific alert click the severity hyperlink of the appropriate alert. Cisco Internal Use Only Copyright © 2006 Cisco Systems, Inc. All rights reserved.
c Step 3. (Contd.) c. PVM displays the details of the threshold violation. Cisco Internal Use Only Copyright © 2006 Cisco Systems, Inc. All rights reserved.
Scenario 3: Troubleshooting You are base-lining your response times from the server. An alert is issued that a critical corporate server has a very high response time when compared to the baseline. You want to find out if the apparent slow response time is due to the network or the application. Once you find that the problem is indeed with the network, you want to know where the problem is and correlate the response time problem in context with other traffic in your network.
a c d b The Link Utilization is very high and close to maximum. Step 2. Verify the link utilization on the client Branch Router. a. Click the Monitor tab b. Click Switch/Router -> Interface c. In the Parameters Pane, select the appropriate DSG and time period. d. Click Refresh Cisco Internal Use Only Copyright © 2006 Cisco Systems, Inc. All rights reserved.
a b c Step 3. Using the PVM Single Sign-On feature, logon to the Branch Router NM-NAM for further troubleshooting. a. Click the Setup tab b. Verify the appropriate NAM c. Click Connect d. PVM takes you to the NAM Overview page. If an application is utilizing extra bandwidth on the branch router, you can use the Single Sign-On feature of PVM to logon to the NM-NAM on the branch router and check for the applications that are using that particular link.
a b c d Step 3. Using the PVM Single Sign-On feature, logon to the Branch Router NM-NAM for further troubleshooting. a. Click Monitor Æ Apps b. Select the appropriate datasource c. Top protocols are displayed. Select the FTP and click details to view the hosts using that protocol. Cisco Internal Use Only Copyright © 2006 Cisco Systems, Inc. All rights reserved.
Overview of PVM functionality You now have an understanding of usage of Cisco PVM with some of the scenarios mentioned earlier This section will explain all the features of Cisco PVM to provide a thorough overview of Cisco PVM capabilities. Traffic Analysis using Cisco PVM Cisco PVM provides two ways to perform traffic analysis. For active monitoring of network traffic, use the Monitoring feature. For historical traffic analysis, use the Reporting feature.
Clicking the right arrow or the green Monitor bar on the left toggles the display of the Generate Reports menu. Similarly clicking the down arrow or the green Monitor bar toggles the display of the parameters pane. Cisco Internal Use Only Copyright © 2006 Cisco Systems, Inc. All rights reserved.
From the Parameters pane shown select the appropriate DSG, view and report type and the time frame for which to run the report. Then select the appropriate report to run from the Generate Reports menu and Cisco PVM displays report. Note: 1. After login, Cisco PVM automatically displays the Monitoring Tab and runs the Network Overview Report for the first DSG it finds. If the DSG has not been created, a popup asking you to create a DSG is displayed. 2.
Datasource – This scheme allows the user to view traffic statistics per datasource. Aggregated – This scheme allows the user to view aggregated traffic statistics for all datasources in the DSG. All NAM – This scheme allows the user to view traffic statistics aggregated per NAM in the DSG. These aggregation schemes are available for all report suites. For the Switch/Router report suite, the All NAM scheme is called All Device. Note: 1.
The “T” hyperlink provides a trend report which displays all the data points for the given time period and shows a trend line for the statistic of choice. Note: 1. Real-time data is gathered from a chosen datasource once every 5 seconds. 2. Data gathered by the real-time feature is not stored in the database. 3. When you click the “R” hyperlink, a window is displayed with all the datasources in the DSG and the available statistics for the type of report from which the “R” hyperlink was clicked.
Report Name Drill-Down Reports Available Overview • Host Details • Application Details • DSCP Applications Applications Application Details Hosts(IP) Host Details Conversations Host Details DSCP Applications Application details DSCP Host Host Details Note: 1. To provide the single sign-on feature, Cisco PVM has to communicate with an applet. This requires the proper support from the client side browser.
The menu pane on the left pane lists the reports suites. You can click any of the reports, and then select the appropriate parameters in the right pane to either schedule the report or to run it right away. The parameters are mostly self-explanatory and are similar to the ones you select in the Monitoring tab. From the Scheduled Reports menu subsection, you can select the View Schedules menu item to view the schedules of reports to be run.
Clicking the version number displays the report. Note: 1. 2. Reports generated from the Reports Tab are automatically archived. These archives are versioned each run of a particular report. If you schedule numerous reports to be run at high frequency, the archive can balloon into a huge list. Currently all archived reports have to be deleted individually, so exercise caution when scheduling reports. 3. The report archive is independent of the data from which the reports were generated.
Note: 1. You can include multiple datasources from multiple NAMs in an ART group. Select the NAM of interest from the list and then add the datasources from those NAMs to the chosen list of datasources. 2. The Report Interval parameter is an artifact of the ART MIB, which defines when the ART MIB consolidates the response time statistics and starts a new collection cycle. The default value for this parameter is 1800 seconds (30 minutes).
Server Response Time (SRT) – Gives you the server side latency statistics from the Server side NAMs. Client/Server Response Time (CSRT) – Gives you the total roundtrip response time from the client side NAMs.
Note: 1. Pre-filtering: Cisco PVM depends on traffic statistics gathered by the NAM to perform its reporting. Cisco PVM performs some prefiltering of the data is has collected for the various ART Groups. If no data is available for a given time period, you will not be able to see any ART groups and hence see a report. In this case you will see a message that says that no ART Groups were found for the given period. If you see this message, try changing the parameters. 2.
From the Scheduled Reports menu subsection, you can select the View Schedules menu item to view the schedules of reports to be run. Cisco PVM lists the scheduled, running, pending and completed reports. Baselining and Alerts in Cisco PVM Cisco PVM provides you with the ability to proactively monitor the network using thresholds. You can identify problems and trouble spots before they impact users. Threshold violations result in alerts, which can be viewed in the Alert Viewer.
Select the type of statistics and the particular metric that are required. You also need to specify which DSG you want to monitor (In the case of ART statistics, you will need to specify an ART group). Depending on the type of statistic required more fields will appear and you can further tailor the threshold. For example, when you select Host Statistics, an IP Address field and an Application field appear.
Baseline Period (Default is 1 day) – The amount of time over which the moving average baseline is calculated. Note: 1. Collected data is aggregated with the frequency specified by the Aggregation Period. Alerts are issued if the aggregated value exceeds the previous cycles calculated value of Baseline + Standard Deviation and Baseline – Standard Deviation. 2.
trapCommunity = public trapDestination = 172.16.11.161 trapPort = 162 You can define multiple [snmptarget] blocks, one for each destination. Alerts in Cisco PVM Cisco PVM generates alerts in various circumstances and these alerts can be viewed in the Alert Viewer. The Alert Viewer can be accessed by clicking on the Alerts Tab. By default, any alerts over last hour are displayed. You can change the time period and view alerts over that time period. Alerts in Cisco PVM are color coded.
Note: 1. NAM Alarms are obtained by Cisco PVM from the RMON MIB in the NAM. Since the MIB information does not specify a severity level, Cisco PVM always designates a NAM alarm as Minor severity level. 2. NAM Alarms can also be raised due to threshold violations in the NAM. If the NAM alarm was raised as a result of a threshold violation in the NAM, the Description field in the alert detail denotes the name of the threshold in the NAM that was violated.
Cisco PVM Requirements and Sizing Cisco PVM is a network monitoring software that runs on Linux. The minimum recommended hardware and software configurations are as follows: Minimum Server Requirements Hardware: • 2 Intel Xeon CPU – 3.4 GHz • 2 GB RAM • 4 GB HD space available for the application and third–party software • 70 GB HD space available in the host installation directory (This depends on the number of NAMs you want to monitor.
Cisco PVM supports a maximum of 200 NAM–2s, or the equivalent of 100 NAM–2s plus 300 NM–NAMs. The hardware requirements for Cisco PVM installations differ depending on the number of NAMs the system is intended to support.
Note: 1. If the environment you are installing in is configured for NIS, ensure that ‘oracle’ and ‘pvmadm’ users are not created. Cisco PVM will create these users. Install Procedure: This section describes the steps necessary to install PVM. 1. Insert DVD into DVD drive. 2. Open a command shell and go to the DVD drive root (login as root). Ex: cd /dev/cdrom 3. Start the installation: $./installpvm Follow the prompts and change the install directories if necessary. 4.
Remember to generate and install the SSL certificate as ‘pvmadm’ user Troubleshooting Tips: 1. Verify the log files for any signs of trouble. Cisco PVM places the installation log files in the $PVM_BASE/installlogs directory. The main Cisco PVM install log file is named ‘sp_installMM.DD.YY.hh.mm.ss.log’. From there you can glean enough information to look at the other log files and find the issue. If unable to do so, contact Cisco TAC. 2.
$su - oracle $export ORACLE_SID=cnam $sqlplus /nolog sqlplus>connect /as sysdba sqlplus>shutdown immediate sqlplus>quit $export ORACLE_SID=spdw $sqlplus /nolog sqlplus>connect /as sysdba sqlplus>shutdown immediate sqlplus>quit $lsnrctl stop 4. Shutdown any rogue Oracle processes $ps –ef | grep ora If you see any oracle process, kill them manually. At this point, you should check any semaphores or queues that might be left open by the terminated oracle processes.
restart. If Cisco PVM is manually stopped, it will have to be restarted manually as well. 2. When Cisco PVM is started manually, ensure that you are starting it as ‘pvmadm’ user. Also, Cisco PVM starts both the Oracle processes and the Cisco PVM server processes when the pvm start command is issued. To do this it uses the ‘sudo’ process available in Linux to start Oracle as the ‘oracle’ user.
Cisco PVM is a database intensive software application. Similar to any database-driven application, Cisco PVM has some maintenance activities that the user can perform to ensure good performance and trouble free use. The most important aspect of the maintenance activities is database management. Database Management Cisco PVM uses 2 database instances to store the data it collects. Raw traffic statistics that are collected by the system are stored in the OLTP database.
$archive –p -f [-I] [-[{T|H}]C] {start|stop} The I flag runs the archive process immediately instead of scheduling it. If the I flag is not specified, the archive process will be scheduled as a cron job which runs daily at 3:00 AM. The H flag indicates to Cisco PVM to include historical information in the archive. The T flag indicates to Cisco PVM to include transactional (raw statistics) information in the archive.
CONCLUSION This guide attempts to make the deployment of Cisco PVM on your network easier to plan and execute. The tasks that you need to perform to successfully deploy and use Cisco PVM are explained in detail. If you want in-depth understanding of Cisco PVM, see the User Guide and other documentation for Cisco PVM. This guide also attempts to look at some of the scenarios in daily network management and how Cisco PVM can help you accomplish your objective.
APPENDIX Deployment Q&A Q. What login permissions are required to install Cisco PVM? A. Cisco PVM installation requires root-level access to the Linux server that has been configured to run Cisco PVM. Q. Why does Cisco PVM not overwrite an existing Cisco PVM installation? A. When Cisco PVM detects an existing installation, the install routine notifies the user and stops the installation. The application must be uninstalled before re-installation is permitted.
A. The installation software requires a minimum of 4.0 GB of disk space to install Cisco PVM and third-party software. Cisco PVM requires that a minimum of 70 GB of disk space be available in the data storage directory. For more information about storage requirements based on the number of NAMs you intend to support, see the Cisco Performance Visibility Manager Installation Guide. Q. What are the ports and protocols used by Cisco PVM? A. See Table 1-1. Q. How do I obtain a license file for Cisco PVM? A.
Q. Does Cisco PVM provide tracing capability? A. The Cisco PVM collection framework provides tracing capabilities from the Cisco PVM server. The trace level can be configured which dictates the detail of the trace information. The trace information is logged into separate files for each collector. Some of the security and system logs that appear in the Cisco PVM GUI are also useful to troubleshoot certain problems. Filtering can be applied to Security Logs in the Admin GUI to view specific logs.
Deployment Troubleshooting Symptom - During the Cisco PVM installation, I receive the error message “Not enough free disk space”. Possible Cause - The Cisco PVM installation directory has insufficient disk space. Recommended Action - Check the Cisco Performance Visibility Manager Installation Guide for disk space requirements, free up the required space in the installation directory, and repeat the installation process.
Symptom - After successfully installing the Cisco PVM product, when I invoke the web page to access the PVM GUI, I get the error message “The page cannot be displayed” from the web browser. Possible Cause - You did not generate the SSL key file. This file is necessary to run the Cisco PVM GUI. Recommended Action - Generate the SSL key file. See the Cisco Performance Visibility Manager Installation Guide for more information on how to generate the SSL key file.
For More Information Release Notes for Cisco Performance Visibility Manager, 1.0 (OL-8615-01) http://www.cisco.com/en/US/products/ps6768/prod_release_note09186a0080640a00.html Cisco Performance Visibility Manager User Guide (OL-8620-01) http://preview.cisco.com/en/US/products/ps6768/products_user_guide_book09186a008063d44f.html Cisco Performance Visibility Manager Installation Guide (OL-8614-01) http://www.cisco.com/en/US/products/ps6768/products_installation_guide_book09186a008063d41d.
THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
