HPjmeter 4.
© Copyright 2005-2011 Hewlett-Packard Development Company, L.P. Document Notice The information in this document is subject to change without notice. Hewlett-Packard makes no warranty of any kind with regard to this manual, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose.
Table of Contents About This Document.......................................................................................................10 Intended Audience................................................................................................................................10 Typographic Conventions.....................................................................................................................10 Additional HPjmeter Documents.................................................
Monitoring Applications ............................................................................................36 Controlling Data Collection and Display.............................................................................................36 Setting Data Collection Preferences......................................................................................................36 Managing Node Agents................................................................................................
Profiling with -Xeprof......................................................................................................................61 Profiling with Zero Preparation......................................................................................................62 Profiling with -agentlib:hprof..........................................................................................................63 Naming Profile Data Files.....................................................................
Open and Cached Sessions........................................................................................................96 Time Slice Entries.......................................................................................................................96 Saving Data................................................................................................................................96 Console Tool Bar Buttons...........................................................................
Guidelines...........................................................................................................................125 Details..................................................................................................................................125 Unfinalized Objects..................................................................................................................126 Guidelines.......................................................................................
Live Objects (Bytes)..................................................................................................................145 Live Array Sizes........................................................................................................................145 Unfinalized Objects..................................................................................................................146 Reference Graph Tree.............................................................................
Reset Interval to Default Settings Across All Metrics.........................................................184 Changing Time Interval in Monitoring Visualizers.................................................................184 Change Color Selection for Histogram Display..................................................................................185 9 Understanding How HPjmeter Works ....................................................................
About This Document This document provides updated information about product features, known problems, and workarounds for the 4.2 release of HPjmeter. Intended Audience This document is intended for application administrators and software developers responsible for monitoring Java application performance and operation on HP-UX. Users are expected to have some knowledge of concepts and/or commands for the HP-UX operating system, usage of the Java Virtual Machine on HP-UX, and Java application performance.
Publishing History The document publication date and part number indicate this document's current edition. The publication date changes when a new edition is printed. Document updates may be issued between editions to correct errors or document product changes. To ensure that you receive the updated or new editions, subscribe to the appropriate product support service. See your HP sales representative for details.
1 Introducing HPjmeter 4.2.00.00 HPjmeter is a performance analysis tool for deployed Java applications. It will help you identify performance problems in your production environment as well as during development. NOTE: You cannot use HPjmeter to monitor Java applets in a production environment. HPjmeter helps you diagnose many types of Java application problems that occur only after a product is deployed.
• • • • Interactive call graph with call count, or with CPU or clock time, if available Per thread, per thread group, or per process display Allocated and residual objects with object reference graph Comparison capability for performance improvement tracking HPjmeter provides these metrics for garbage collection (GC) analysis: • Details and graphical display of object creation rate, changes in cumulative memory allocation and in heap usage as related to GC events and types, and duration of GC events in th
2 Completing Installation of HPjmeter NOTE: This chapter assumes that you have installed the HPjmeter components on the appropriate server(s) and are ready to configure the agents. For information on initial installation, see HPjmeter Release Notes and Installation Guide. Platform Support and System Requirements Although the agents and console can execute on the same system, this document assumes they are installed on separate systems—the recommended configuration.
Table 2-2 Console Requirements (continued) Patches and updates HPjconfig can help you determine which Java patches are recommended or required for best operation of the HPjmeter agent on HP-UX. Processor and memory • Minimum 500 MHz processor is recommended. • Minimum 256 MB memory is required.
To Take Advantage of Dynamic Attach: Check that JM_JAVA_HOME in $JMETER_HOME/config/hpjmeter.conf is set to a Java 6 directory (6.0.03 or later) to be able to later dynamically attach to a running JVM from the HPjmeter console without first setting HPjmeter options on the command line. When HPjmeter is installed on a system that has Java 6 installed in the usual location, the installation procedure will automatically store the JDK location in hpjmeter.conf configuration file. If the Java 6.
$ /opt/java1.4/bin/java -Xms256m -Xmx512m -Xbootclasspath/a:/opt/hpjmeter/lib/agent.jar -Xrunjmeter myapp NOTE: With the addition of JVMTI in JDK 5.0, you should use the -agentlib switch to take advantage of improvements in JDK 5.0 and reduce the impact of data sampling on application performance. While -Xrunjmeter can still be used to specify the JVM agent with Java 5 versions, the impact on application performance is significantly greater than when using -agentlib. Starting with Java 5.
include=filter1[:filter2]..., exclude=filter1[:filter2]... Creates a colon-separated list of classes to specify which classes or packages are instrumented. Method-level filtering is not supported. • If a class is not instrumented, the JVM agent metrics that use bytecode instrumentation (BCI) do not provide any output related to the class methods. To see the list of filters in • • • effect while the data was collected, including default agent filters, click the icon when you see it in the console.
• • The default filters always include the following packages. They cannot be overriden with an exclude option: sun.jdbc.odbc. weblogic.jdbc.informix4. org.apache.jsp. weblogic.jdbc.mssqlserver4. org.apache.jasper. weblogic.jdbc.oci. weblogic.jdbc.rowset. com.bea.p13n. com.bea.netuix. oracle.jdbc. oracle.sql. oracle.evermind.sql. com.bea.medrec. com.ibm.samples. com.sun.j2ee.blueprints. com.sun.ebank. You can change the default behavior by using the include option.
noalloc Reduces dormant overhead by skipping bytecode instrumentation that applies to object allocation metrics: Allocated Object Statistics by Class and Allocating Method Statistics. The noalloc option makes those metrics unavailable. Example usage: -agentlib:jmeter=noalloc nohotspots Reduces dormant overhead by skipping the bytecode instrumentation that supports Java Method HotSpots.
JVM Options Usage Examples Here are some examples of JVM agent option usage: $ $ $ $ -agentlib:jmeter=noalloc,appserver_port=7001,public,include=weblogic:com.bea -agentlib:jmeter=nohotspots,owner_private,verbose:file=bcifilters.txt -agentlib:jmeter=version -agentlib:jmeter Security Awareness Securing Communication Between the HPjmeter Node Agent and the Console IMPORTANT: The data stream between the HPjmeter console and agents is not protected from tampering by a network attacker.
IMPORTANT: If you choose to open a port through a firewall to enable communication between a node agent and a console, secure the tunneling port using HP-UX Secure Shell or HP-UX IPSec. Configuring User Access The node agent must be started by either the same user or group as the running JVM (recommended) or root to establish contact. IMPORTANT: Setting access for owner or group should not be considered a security solution because node agent to JVM communications are not secured by default—see below.
3 Getting Started Are You Monitoring an Application or Analyzing Collected Data? HPjmeter helps you to monitor a live run of your application. You can also view saved data captured using HPjmeter during a live session, by using profiling or garbage collection options when you start your application, or if you are running HP JDK/JRE 5.0.04 (or higher), by starting and stopping profiling data collection from the command line.
1. Choose Connect from the File menu or click Connect to server [ 2. In the Connect to Server dialog box, type the complete host name (server.domain.com) on which a Java application and a corresponding node agent are running. Optionally, in the Optional Port box, specify the same port number that was used when the node agent was started. 3. 4. ] Click Connect.
1. Choose Connect from the File Menu or click Connect to server [ 2. In the Connect to Server dialog box, type the complete host name (server.domain.com) of the system that is the designated host for the SSH tunnel (the SSH delegate machine). Optionally, in the Optional Port box, specify a port number through which information can pass from the SSH delegate to the node agent. Check the box labeled Use SSH Options. 3. 4. ] The Connect to Server dialog expands to reveal required SSH attributes. 5. 6.
9. Repeat the previous steps if you want to connect to several node agents. Set Session Preferences 1. Double-click on the JVM icon in the data pane for the application that you want to monitor. This opens the Session Preferences dialog box.
NOTE: When the Optional Port Box Is Not Used The console automatically attempts to use a port on the remote system between 9505 and 9515 when arranging a port for its server socket. If it is unable to successfully use a port from this range, it will use an ephemeral port number. 2. By clicking the tabs in this dialog, you can see the default settings for metrics, filters, and alerts. Enable or disable the settings as desired.
Some alerts require collection of specific metrics such that even if a metric is de-selected, it may be activated by selection of a related alert. 3. Once all selections are made, click OK. The preferences window will close and the new Open Session will be visible, marked by this icon: Though it is possible to begin viewing application activity immediately in the monitoring visualizers, you may want to wait for a short period of time before viewing some of the metrics.
behavior and performance. A longer collection time gives you greater accuracy in the results. For details, see: Setting Monitoring Session Preferences (page 100) . Changing Session Preferences During a Session With HPjmeter 4.0 and later, it is possible to change session preferences during a session. 1. Double-click the “Open Session” line in the console for the session that you want to change. The Session Preferences window opens. 2.
NOTE: If you are running the HP JDK/JRE 5.0.04 or later, you can send a signal to the running JVM to start and stop a profiling data collection period with zero preparation and no interruption of your application. See Profiling with Zero Preparation (page 62). 1. Configure your application.
6. Click among the tabs to view available metrics. Use the Metrics or Estimate menus to select additional metrics to view. Each metric that you select opens in a new tab. Mousing over each category in the cascading menu will reveal the relevant metrics for that category.
4. Click File→Open File to browse for and open the data file, or drag and drop the file onto the console. 5. A GC viewer window opens and displays a set of tabs containing summary and graphical metric data. Click among the tabs to view available metrics. 6.
$ cd $JMETER_HOME/demo $ ../bin/run_simple_jvmagent -jar sample_program Use the file name of the specific sample you want to run in place of sample_program. 4. 5. 6. In the console main window, click Connect and type in the host name of the machine that runs the sample application. If you specified a port number when starting the node agent, use the same port number. Otherwise, leave the port number box empty. An icon representing the host appears in the main window.
Figure 3-1 Heap Display Showing Possible Memory Leak Behavior See also Heap Usage Notification (page 111). The Pi sample program uses a simple Java program that allocates some arrays, thereby exhibiting the behavior of a memory-leaking application. The demo application is available from the HPjmeter installation directory: Source: $JMETER_HOME/demo/Pi.java Binary: $JMETER_HOME/demo/Pi.jar Use -jar Pi.jar with the run_simple_jvmagent script to start the sample.
The example creates pairs of threads every 30 seconds, stopping at 50 threads, which synchronize work using shared locks. Occasionally, the program reverses the order in which locks are taken, eventually causing a deadlock, which generates a Thread Deadlock Alert (page 115). The demo application is available from the HPjmeter installation directory: Source: $JMETER_HOME/demo/DL1.java Binary: $JMETER_HOME/demo/DL1.jar Use —jar DL1.jar with the run_simple_jvmagent script to start the sample.
4 Monitoring Applications In addition to using HPjmeter when you notice a problem, you can also use it to prevent problems. For example: • You can use HPjmeter to check the efficiency of your applications, even if you don't notice a problem. To make it easier to detect problems, you should periodically review key metrics for your application to establish a baseline profile. To save metrics information to a file that you can review later, see Saving Monitoring Metrics Information (page 38).
Managing Node Agents Each managed node requires an HPjmeter node agent that manages communication between the console and JVM agents. Once the Session Preferences window closes, and after waiting for a few moments, you should see the running JVM agents listed below the connected server host name. If no JVM appears, check that a node agent is running on that server. If no node agent is running, the console cannot show the JVM agents.
Start the node agent from the command line by typing: $ JMETER_HOME/bin/nodeagent where $JMETER_HOME has the default value of JMETER_HOME=/opt/hpjmeter You can run the node agent as a background process.
NOTE: Saved data files can be quite large. If you expect to keep the files or you plan to transfer the files, compress the files for speedier transfer. Compressing the files also reduces the chances that the file could be corrupted during a transfer. You can compress various profile data files; for example: .vgc, .hprof (ascii text and binary format), .eprof, and .hpjmeter. The preferred compression format is gzip. HPjmeter 4.1 and higher can read gzip compressed files, without having to expand them.
Typically, when the top entry is represented by a single-digit percentage, you should not see application CPU performance issues unless the entry describes an obscure method that you did not expect to see. Related Topics • • • Java Method HotSpots (page 118) Process CPU Usage Alert (page 113) System CPU Usage Alert (page 114) Viewing the Application Load Use the Heap Monitor display. This metric shows how busy the application is.
You can verify this using a system tool like HP GlancePlus. The possible remedies for thrashing: • Decrease the maximum heap size, which corresponds to a decrease of the maximum load supported by the application. • Eliminate other load on the system. • Install more physical memory. Related Topics • • • Heap Monitor (page 122) Garbage Collections (page 123) Heap Usage Notification (page 111) Identifying Excessive Calls to System.gc() Use the Heap Monitor display.
Figure 4-1 Example Metric: Percentage of Time Spent in Garbage Collection When Application Behavior is Normal This next example shows an application with a potential memory leak. • A rising trend in the percentage of time spent in garbage collection.
• • Array Leak Locations Alert (page 112) Class Loaders (page 150) Checking for Proper Heap Sizing Efficiencies in program performance can be obtained by allocating the optimal amount of memory for the heap according to the needs of the program and the operation of the JVM garbage collection routine. Checking activity in the heap and comparing to garbage collection frequency and duration can help you determine optimal heap size needed for best performance from your application.
The Heap Monitor and Garbage Collections data help you determine if an observed application termination could have been caused by running out of heap space. The Heap Monitor gives the idea about the overall heap limit, while the Garbage Collections display shows the size of heap after each garbage collection. If the heap size shown by Garbage Collections converges toward the heap limit, the application will, or has, run out of memory.
Threads are relatively costly to create and dispose; usually a more efficient solution is through thread pooling. Related Topics • • Thread Histogram (page 131) Identifying Abnormal Thread Termination (page 44) Identifying Excessive Lock Contention Use the Thread Histogram. There is no simple answer to how much lock contention is excessive. A multi-threaded application normally exhibits some lock contention.
The Thread Histogram shows the number of threads existing in the observed period of time. By selecting the smallest time interval, one minute, you can see an estimate of the number of simultaneously active threads. Most operating systems have a limited capacity for the number of threads a single process may create. Exceeding this capacity may cause a crash.
Related Topics • • • Excessive Compilation Alert (page 110) Method Compilation Count (page 134) Method Compilation Frequency (page 135) Identifying Too Many Classes Loaded Use the Loaded Classes display. This display can be used to determine if the pool of classes loaded into memory stabilizes over time to a constant value, which is normal.
Figure 4-4 Main Console Showing an Expanded JMX Server Node in an Open Session To access this summary data, double-click on the summary JMX server node. The summary JMX Viewer opens. For information on this part of the JMX Viewer, see Understanding the JMX Summary View (page 48). To manipulate some JVM and MBean functions on this server, expand the summary node and double-click the entry labeled Modify JMX MBean data. The server JMX Viewer opens.
Figure 4-5 Appearance of the Summary JMX Viewer When First Opened JMX Memory Tab Fluctuations in memory usage in heap and non-heap areas are graphically displayed and periodically updated for the duration of the open session. Mouse over the features of the graph to learn what spaces are represented, as well as to learn what each of the markers designates. The following image demonstrates some of the features available on this tab.
Figure 4-6 JMX Viewer with Summary Memory Tab Selected Select a region of interest Click the color bar in the graph to select the memory space of interest. The bar will become outlined in blue and the Region Details will update in the text area. Consult the Region Details for updated information on memory usage in that region, total count of garbage collections, and cumulative duration of GC events. Set a usage threshold Where usage threshold can be set, a “0” (zero) appears in the Usage Threshold box.
On pressing Enter, a red marker will appear on the region usage bar to mark the point at which the threshold is reached. The bar color turns red when the threshold is reached and a notification is generated. Start verbose GC At any point, you can click the verbose GC check box to start collection of verbose GC data by the JVM. The JVM usually writes this information to stdout. Uncheck the box to stop logging verbose GC data.
See details for a selected thread Click a thread name in the left pane. Details on thread activity and current state appear in the right pane. Apply a filter This filtering text box is useful when the thread list is long. It allows you to reduce the list size according to the starting letters of the thread name. To apply a filter, start typing the first few distinguishing letters of the thread names that you are interested in. The list will immediately be trimmed to entries starting with those letters.
Figure 4-8 Default View When Server JMX Viewer is Opened JMX MBean list filter Click this button to see a list of the filters that you can apply to the MBean drop-down menu items at . JMX server drop-down menu The server from which this JMX viewer was launched is shown in the “JMX Servers” drop-down menu. JMX MBean drop-down menu This drop-down menu lists the viewable MBeans for which data can be displayed in the viewer. This list can get quite long when viewing data for application servers.
A small window opens that gives you the following options for sorting the MBeans: • • • by domain (for example, java.lang, com.bea) by name (for example, ClassLoadingImpl, MBeanServerDelegate) by type (for example,GarbageCollector, MemoryPool) Select the filter type that you want, and click the Use Filter button. Use the MBean drop-down menu to see the resulting list and to select an MBean to view. The MBean Attribute Tab The Attributes tab lists the contents of the selected MBean.
Clickable value Names or values appearing in boldface type can be opened to display details in the Detail Viewer area. Double-click the boldface text to open a tab in the Detail Viewer as in . The tab remains open until you click the closure box on the tab or until you close the viewer. Editable value Values appearing in blue type can be changed. Click the blue text to open a text box. Type the change, and press Enter to immediately apply the change.
Operational detail tab The detail tab remains available until you click the closure box on the tab or until you close the viewer. Editable text box Values appearing in text box can be changed. Type the change, and press the Operation_Name button to immediately apply the operational change. In this example, the operational button name is getThreadCpuTime . To view the effect of the change, choose an appropriate monitoring metric.
Figure 4-12 MBean Information Tab Open for Display View classification information for the selected MBean.
5 Profiling Applications HPjmeter allows you to process profile data from Java virtual machines. Separating the profile data collection step from the analysis step has the following advantages: • The data analysis can be done at a different time and on a different platform than was used to run the application. For example, it can be done on a desktop system or on a laptop. • A non-interactive profiling agent will often impose less overhead than an interactive one.
Generally, a metric is a mapping that associates numerical values with program static or dynamic elements such as functions, variables, classes, objects, types, or threads. The numerical values may represent various resources used by the program. For in-depth analysis of program performance, it is useful to analyze a call graph. Call graphs capture the “call” relationships between the methods.
There are two important rules to remember when modifying programs to improve performance. These might seem obvious, but in practice they are often forgotten. • Don't put performance above correctness. When you modify the code, and especially when you change some of the algorithms, always take care to preserve program correctness. After you change the code, you'll want to test its performance. Do not forget to test its correctness.
See also For information on how to compare data files, see Comparing Profiling Data Files (page 71) and Scaling Comparison Data (page 72). Collecting Profile Data To take full advantage of HPjmeter functionality, you can gather profiling data using -Xeprof for performance tuning and -agentlib:hprof for memory tuning when you run your application. NOTE: If you are running the HP JDK/JRE 5.0.
Table 5-1 Supported -Xeprof options (continued) time_slice=sigusr1|sigusr2 Specifies which signal will cause profiling termination and the profile data output. • The signals for profiling start and profiling termination can be the same. Specifying a signal and a timeout at the same time is possible by repeating the time_slice option — termination of profiling occurs when the first qualifying event takes place. The application will continue running.
3. Start the profiling interval: send a signal to the JVM by typing: kill -USR2 pid You will see the following message: eprof: starting profiling Let the profiling collection generated by the JVM continue for the length of time that you think will be meaningful. 4. Stop the profiling interval by sending the same signal to the JVM: kill -USR2 pid You will see the following message: eprof: terminating profiling writing profile data to ./filename.
To see the availability of HPjmeter metrics from hprof data collection, see –Xeprof and –agentlib:hprof Profiling Options and Their Corresponding Metrics (page 65). For additional information on collecting heap dump data, see the Java™ Troubleshooting Guide for HP-UX Systems. NOTE: With the addition of JVMTI in JDK 5.0, the -agentlib switch is the preferred way to activate tools such as hprof. With -agentlib, you can take advantage of improvements in JDK 5.
Table 5-2 Supported -agentlib:hprof options (continued) depth=size Controls the depth of the stack trace for each stack trace taken. • Increasing this value will make the profile data file larger, but it enables HPjmeter to better estimate inclusive method times and generate more complete call graphs. • If you measure application performance by sampling, the default value of 4 is too small for most applications; use values between 6 and 12.
Table 5-4 Available Metrics or Features from -Xeprof and -agentlib:hprof Data (continued) Feature -Xeprof -agentlib:hprof cpu=samples 66 cpu=times Yes 1,2 heap=dump heap=sites N/A N/A Inclusive Method CPU Times (page 142) Yes Inclusive Method Clock Times (page 142) Yes No No N/A N/A Threads/Groups Histogram Yes Yes 3 Yes 3 Yes 3 Yes 3 Objects Created by Method (page 145) Yes 2,4 N/A Yes 2 No Yes Created Objects (Count) (page 145) Yes 2,4 N/A Yes 2 No Yes Created Objects (
Table 5-4 Available Metrics or Features from -Xeprof and -agentlib:hprof Data (continued) Feature -Xeprof -agentlib:hprof cpu=samples cpu=times heap=dump heap=sites Average Inclusive Method Clock Times* (page 143) Yes No No N/A N/A Starvation by Method* (page 143) Yes No No N/A N/A Starvation Ratio* (page 144) Yes No No N/A N/A Methods with Loops* (page 144)2 Yes No Yes N/A N/A Lock Contention Ratio by Method* (page 155) Yes No No N/A N/A Average Exclusive Method Lock Del
Approaches to Analyzing Performance Data The metrics you choose to view depend on your application domain, your code, the demands for the application, and the operating environment. Look at all of the available ones, and analyze at least a few of the topmost entries. Whenever possible, the console visualizers present all metrics sorted in decreasing order of resources used. This places the largest resource users at the top of the screen. Evaluate them knowing your application and its characteristics.
Inclusive Versus Exclusive Time Exclusive time is the accumulated total time used by a method, but not including the time used by the methods that were called from it. Inclusive time is the accumulated total time used by all invocations of the method and all methods that were called from it directly or indirectly. Inclusive times are useful if you want to see the cost of a particular task performed by the application, and you can associate this task with a single method.
Related Topics • • Viewing Profiling or GC Data in HPjmeter (page 137) Understanding the Summary Presentation of GC Data (page 88) Adjusting Scope The console can display metrics for the whole application, for a single thread, or for a single thread group, provided the profile data has been appropriately collected. These three sectors of analysis are called scopes. To change the current scope, do the following. 1.
Comparing Profiling Data Files It can be useful to compare session data in order to understand the effect of differences in activity at different times of day, week, or month. Comparisons can be especially useful when running performance testing benchmarks. (See Preparing a Benchmark (page 60).) To compare data files, do the following: 1. From the main console, open two data files of the same type that you want to compare.
A viewer will open for each data representation (see Using HPjmeter to Analyze Profiling Data (page 29) for an example profiling viewer). 2. 3. Click on one of the viewers to make it active. Two ways exist to create a combined set of comparable data: • In one viewer, click File→Compare , or • click the Compare button . The Compare window opens, and you can now see the other file to compare your first file against. 4. Click the file name that you want to compare against. 5.
menu bar. HPjmeter automatically recalculates the call count for each method in the peak data with the result that each value is normalized to the ratio of the call counts for the selected method. HPjmeter then sorts the data. By viewing the information that appears in the newly sorted list, you can compare differences in activity between the two data files. If you want to see how this works in detail, select View→Show Formula to display the calculations that HPjmeter uses.
Table 5-5 Key to Default Color Representation of Thread States in Histograms Lock Contention. Time wasted trying to acquire a Java lock while it was acquired by another thread. More precisely, it is the summary time measured from the moment the thread requests a Java lock until the lock is actually granted. CPU. Time spent running on any of the computer's processors. This includes Java code, the native code (if any), and the time spent in the virtual machine by this thread.
TIP: To best view the relative amount of starvation for a particular thread, highlight the thread by clicking on it once, or double-click to reveal the thread pie chart for that thread. Starvation is represented by the “empty” slice of the pie. NOTE: Be careful when analyzing the spectrum of a thread group. A thread group may contain several threads with completely different roles within the application. Mixing them up and analyzing just the summary may make little sense.
from the caller to the number of all calls to this method from all callers. It may be non-intuitive to traverse this graph if you get used to the call graphs representing time. This is because below a low number of calls—for example, 1—a very large number of further calls can be hidden. In cases when there is more than one caller of a method, the number of other callers is also given. Example of Node Color Display It is best to explain the meaning of the numbers in a tree in an example.
NOTE: Inclusive times include the time spent in called methods. Therefore, for recursive calls, the same time is accumulated many times, depending on the depth of the recursion. When this happens, a recursive method may show an inclusive time that is greater than the inclusive time shown for its parent method. TIP: To see the list of all callers of a given method, double-click on the method name. The line showing repaint is marked in red.
Searching the Trees The search functions work in trees basically the same way they work for other metrics, but there are a few special considerations for searching call graph trees. • Occasionally, not all nodes found in a search will be accessible (visible). You'll be notified about such situations in the status bar at the bottom of the window.
with the number of bytes that would be reclaimed. It is up to you to check if the presented objects are still needed by the application, and how the critical references should be removed.
6 Analyzing Garbage Collection Data HPjmeter allows you to process garbage collection (GC) data from Java virtual machines. Separating the GC data collection step from the analysis step has the following advantages: • The data analysis can be done at a different time and on a different platform than was used to run the application. For example, it can be done on a desktop system or on a laptop. • A non-interactive agent will often impose less overhead than an interactive one.
Table 6-1 Supported -Xverbosegc options for Java 1.5.0.04 (continued) filename writes output to the specified file. [n][h][d][u][t] n prevents appending the pid to the log filename. h appends the hostname to the log file name. u appends the username to the log file name. d appends the date to the log file name. t appends the time to the log file name. At every selected garbage collection, the following 20 fields are printed. These values hold true for Java 1.5.0.04. Other versions may differ.
Table 6-2 Fields Captured in GC Log Data When Using -Xverbosegc print options (continued) — — — — 3: Incremental collection will fail 4: Linear allocation will fail 5: Anticipated promotion 6: Incremental CMS garbage collection and m indicates the background CMS state when yielding, as follows: — — — — — — — — 0: Resetting 1: Idling 2: Initial marking 3: Marking 4: Final marking 5: Pre-cleaning 6: Sweeping 7.
Collecting Allocation Site Statistics for Viewing in HPjmeter Use the following command to capture allocation site statistics in a file for display in the HPjmeter garbage collection viewer. The GC viewer automatically opens when you open an Xverbosegc file from the HPjmeter console in one of the following ways: • Using -XX:+PrintAllocStatistics option to enable the Allocation Statistics function by entering the following: $ java ...
Let the collection generated by the JVM continue for the length of time that you think will be meaningful. 4. Stop the data collection interval by sending the same signal to the JVM: kill -PROF pid You can now open the saved file in the HPjmeter console and view the collected metrics. Time periods where no data is collected are shown in a light purple in all the GC graphic visualizers.
• • • Heap in use after the GC event Current maximum heap size Duration of the GC event To capture basic -Xloggc data, use the following command: $ java ... -Xloggc:filename More data are available if you start the JVM with the -XX:+PrintGCDetails or -XX:+PrintHeapAtGC options. For example, you can type: $ java ... -XX:+PrintGCDetails -Xloggc:filename Use these additional options to obtain more data on GC activity in the eden space, the survivor space, the old generation, and the permanent generation.
† Requires the additional option -XX:+PrintAllocStatistics. See Collecting Allocation Site Statistics for Viewing in HPjmeter (page 83) for an example of command syntax. * You can create a graphical view of these metrics using the User-Defined X-Y Axes visualizer [p. 167]. Other raw metrics are also available for graphing from this visualizer.
3. Two ways exist to create a combined set of comparable data: • In one viewer, click File→Compare , or • click the Compare button . The Compare window opens, and you can now see the other possible files to compare your first file against. 4. Click the file name that you want to compare against; or, if choosing more than one file, hold the Ctrl key while selecting each desired file. 5. When the files are highlighted, click the Compare button.
Table 6-5 Common Garbage Collection Types Recognized by HPjmeter Concurrent Mark-Sweep (CMS) A garbage collection performed in the old area of the heap as a background thread that does its work with few pauses in the application run. Parallel Scavenge Only objects from the young generation are collected using a multi-threaded garbage collector. Scavenge Objects from the young generation only are collected using a single thread.
Figure 6-2 Summary Panel Showing Garbage Collection Statistics Heap Capacity the heap. Initial, final, and peak sizes allocated for various organizational spaces in Peak Usage of Capacity The highest percentage of actual use by the application of the heap at its final capacity configuration. A rule of thumb for optimal utilization would be to have the eden space at 100% with the survivor and old spaces showing some reserve capacity, depending on the application.
preparation GC data collection is used during a monitoring session. They are not present when collecting GC data using the -Xverbosegc or -Xloggc options on the command line. Total Bytes Allocated The total amount of space created for new objects over the lifetime of the application. This number represents the total amount of memory the program would have consumed had no garbage collection been performed. It is an abstract measure of the total work done by the application. Number of GC Events program run.
Figure 6-3 Summary Panel Showing System Details for a Period of Xverbosegc Data Collection NOTE: The Number of Localities, Heap Pointer Mode, and UseNUMA fields only display if JDK/JRE 6.0.08 or later is being used. As with any field whose value is too long for its column width, if the JVM Arguments value shown at the bottom of the panel is extensive and runs beyond its column, you can mouse over the value to display a yellow pop-up box that shows all the JVM arguments.
Figure 6-4 System Details Tab with JVM Arguments Pop-up Box Related Topics • • • • 92 Using Specialized Garbage Collection Displays (page 157) Basic Garbage Collection Concepts (page 87) Tool Bar Buttons for Manipulating Garbage Collection Data (page 177) Changing Time Interval in GC Data Visualizers (page 181) Analyzing Garbage Collection Data
7 Using the Console The console main window displays when you start HPjmeter. Starting the Console You can run the HPjmeter console on HP-UX, Linux, and Microsoft Windows systems. Starting the Console On HP-UX There are two ways to start the console on HP-UX. • From the command line, type /opt/hpjmeter/bin/hpjmeter The console will be ready to display performance data.
Figure 7-1 The Main Console Window Data Representation The console main window pane displays session information in a tree format. Figure 7-2 Main Pane with Several File Types Loaded and an Open Session When monitoring a running application, double-click on a JVM entry to open a session. You can display multiple node agents and sessions in the data tree.
When viewing activity in the JMX server, double-click on the Summary to see the data collection options used for this session, or while monitoring, change management bean attributes and observe resulting changes in the data. To quickly access alert visualizers or the Alert Controller, double-click on the alert entries in the tree. Status messages appear briefly in this area. To view console memory usage, mouse over the bar to see current percentage use of memory allocated to the console.
Table 7-2 JVM Agent Connection Icons (continued) The JVM agent is not ready to open a session. Another console has already opened a session with it, the JVM agent is running in batch mode, or the JVM agent is incompatible. The JVM agent is unresponsive, that is, the node agent lost contact with it. The JVM might be down, or it might be overloaded. Verify that the JVM is running and wait until it becomes responsive again before opening a monitoring session with it.
Data from a session is saved by using the appropriate options on the command line when you start your application. • • • Collecting Profile Data (page 61) Setting Data Collection Preferences (page 36) Saving Monitoring Metrics Information (page 38) Console Tool Bar Buttons Select a previously saved monitor batch file, a snapshot file, or a profile data file to view. Open File After opening a saved file, you can review data using the menu metrics appropriate for the data file type.
it looked the last time. With this menu item disabled, the console looks in the current directory. This menu selection is enabled by default. ◦ Remember Main Window Location Click the menu item to toggle whether HPjmeter places the console window in the same location when you restart. ◦ Cascade Metric Windows Click the menu item to have HPjmeter open new metric windows position so you can see the windows beneath the newly opened window.
NOTE: Some metrics may appear grayed out when you select a menu category. These metrics may have been disabled in the session preferences or by specifying a JVM agent option. To re-enable these metrics, you may need to close the current session and start a new session, enabling the desired metrics in the Session Preferences window; or restart the application with different JVM options; or both. See Setting Monitoring Session Preferences (page 100).
Status Bar The Status Bar in the lower bottom and right corner of the console provides standard notifications including: • status and error messages • warning icon when an alert is present • memory use by the console (mousing over the horizontal bar reveals percent memory use of total allocated MB) Setting Monitoring Session Preferences The Session Preferences dialog box allows you to: • Specify which monitoring metrics to collect. • Enable or disable specific alerts.
1. 2. 3. 4. 5. Click the Metrics tab. Click the Enabled box to toggle the corresponding metric on or off. Make sure the measurements that you want are enabled. To modify filters or alerts, click the corresponding tab. Click OK to save the changes. CAUTION: TheThrown Exceptions with Stacktraces metric is unselected by default. Because collecting stack traces can impair your application's performance, enable this metric only when you need to get information about where most exceptions are thrown.
Figure 7-5 Enable or Disable Alerts Window Related Topics • Using the Alert Controller (page 105) to deactivate or reactivate enabled alerts Specifying Filters for Monitoring You can control the amount of information that HPjmeter provides during a live session by filtering out classes using the Filters tab of the Session Preferences dialog box. Filters may be inclusive or exclusive, but not both: • Inclusive filters include only the specified classes in a filter set.
2. Click Add Filter Set. The Filter Set Editor opens. 3. 4. Type the name of the filter set in the Filter Name field. Specify the list of packages for the filter in the Filter Set field. See Specifying Filters for Monitoring (page 102) for details. Specifying Filter Sets Filter sets are comma-separated lists of packages, classes, or both. You can use wildcards or periods at the end of the name only, but they are not necessary. For example, the following are permitted: • javax.swing.* • com.foo.bar.
Figure 7-6 Appearance of Alert Threshold in GC Duration Visualizer If you disable the alert in the Alert Controller, that alert and its threshold value are removed from all relevent visualizers. For example, disabling the GC Duration (Minor) alert will then remove it from the GC Duration visualizer: Figure 7-7 GC Duration (Minor) alert Removed from GC Duration Visualizer See “Responding to Alerts” for information on specific alert notifications.
The console displays an alert while the alert condition exists, and stops displaying when the alert condition no longer exists. A displayed alert may also stop displaying when HPjmeter recalculates the alert based on real-time information. The console displays a warning icon in the lower right corner of the window showing the current number of alerts detected. Using the Alert Controller To open the Alert Controller, double-click on an Alerts counter row in the data tree.
NOTE: If you disabled specific alerts when you set your session preferences, information for those alerts will not be available in the Alert Controller or the main console pane. To re-enable those alerts, you must enable the alert in the session preferences for that session. Figure 7-9 Alert Controller Window You can use the Alert Controller in these ways: Deactivate or reactivate enabled alerts. To deactivate or reactivate alerts once a session has started, use the check box next to the alert name.
panel for as long as the threshold is crossed, periodically updating to show the current value of the heap usage. New values for alerts are applied immediately when you select Apply. Viewing a Log of the Alert History From the Alert Controller, select the Alert Log button to see a history of alerts noted during the monitoring session. Figure 7-10 Example Alert Log Change the alert log capacity. From the Alert Log window, click the “Select Log Capacity” icon to open the log capacity window.
Table 7-6 E-mail Notification Fields and Options for Receiving Alerts Fields/Options Expected Attribute Values Description SMTP E-mail Server: Correctly constructed host name of a valid SMTP server that allows connection without authentication.
Figure 7-12 Abnormal Thread Termination Alert Visualizer You can use this information to debug the cause of the exception. For example, each row in the Abnormal Thread Termination window represents a thread in the Thread Histogram window. You can open the Thread Histogram metric, and notice that when a thread terminates, the associated bar in the Thread Histogram display is shorter than the others. Note that the thread histogram does not distinguish between threads that terminate normally or abnormally.
The window shows events in a hierarchical tree. • Select the View→Show Packages menu item to alternatively hide or show the Java package names to shorten the lines in the display. • Select the View→Show Stacktraces menu item to alternatively expand or collapse the thrown stack traces of all the exception nodes, or click on a specific node to expand or collapse its thrown stack trace only. The results are cumulative over the life time of the session.
• • • Viewing the Application Load (page 40) Checking for Long Garbage Collection Pauses (page 40) Current Live Heap Objects (page 129) Heap Usage Notification It can be useful to change the heap usage threshold in order to observe the heap structure at the selected usage point. You can also set this threshold to notify you when heap usage exceeds your maximum or minimum expectations for the application.
Figure 7-13 Java Collection Leak Locations Visualizer After some time, the alert icon may disappear if: • HPjmeter determines that the previously reported leak object is not a leak. • The leak-related collection object is removed by garbage collection. HPjmeter removes the alert icon when it removes all the leak nodes and no new memory leak is detected. NOTE: This alert is not available when dynamically attaching to a running Java application.
Figure 7-14 Array Leak Locations Visualizer After some time, the alert icon may disappear if: • HPjmeter determines that the previously reported leak object is not a leak. • The leak-related array is removed by garbage collection. HPjmeter removes the alert icon when it removes all the leak nodes and no new memory leak is detected. NOTE: This alert is not available when dynamically attaching to a running Java application.
3. 4. 5. Scroll down if needed to see the Process CPU Usage Alert. Check this box also to turn this alert on. Set the usage threshold to reflect the expected consumption and designate a duration, if desired. Click Apply to activate checking for this threshold. When the set threshold is met, an alert notice is posted on the console. Double-click on the console alert to display the Percent CPU Utilization visualizer.
300% of 1 CPU out of a total of 4 available CPU. By applying this value (300%) for a 5–minute duration in the Alert Controller, HPjmeter notifies you each time that value is exceeded for 5 minutes during the monitoring session. A notification is generated for display on the console and recorded in the alert log.
8 Using Visualizer Functions Visualizers behave somewhat differently depending on the type of data displayed. Visualizer Behavior When Monitoring Behavior or Analyzing Data Visualizers that open as a result of selecting metrics from the Monitor menu on the main console appear as individual, standalone windows. This behavior allows you to view a number of individual metrics at the same time.
Figure 8-2 The Garbage Collection Data Viewer Both standalone visualizers and tabbed viewers have tool bars and other controls for manipulating the data display. See Using Visualizer Tool Bars (page 176). Locating Information About a JVM and its Environment HPjmeter collects information about a running JVM and its environment and displays it in the JVM Summary visualizer. This feature is available by default when monitoring an application or when viewing a saved file from a monitoring session.
Monitor Memory and/or Heap Activity Menu (page 122) • Heap Monitor (page 122) • Garbage Collections (page 123) • GC Duration (page 124) • Percentage of Time Spent in Garbage Collection (page 125) • Unfinalized Objects (page 126) • Allocated Object Statistics by Class (page 127) • Allocating Method Statistics (page 128) • Current Live Heap Objects (page 129) Monitor Threads and/or Locks Menu (page 131) • Thread Histogram (page 131) • Lock Contention (page 133) Monitor JVM and/or System Activity Menu (page 13
Figure 8-3 Monitoring Metric: Java Methods HotSpots with Confidence Interval Graphically Displayed for Each Method Guidelines • • If this metric shows a large percentage of CPU time spent in just a few methods listed at the top, it indicates a performance problem or a method performance problem that might be improved. When the top entry represents a single-digit percentage, a performance problem is unlikely unless the entry describes a method that you did not expect to see.
NOTE: The data collection for Java Method HotSpots may be significantly delayed if one or more of these conditions exist: • The application runs on a single CPU system. • The application does not consume a lot of CPU. • The application consumes CPU exclusively in non-profiled methods. Restarting the application server or node agent will not improve data collection.
NOTE: Collecting the stack trace information could impair the performance of your application if the application throws a large number of exceptions during the session. To minimize the effect on your application, you can enable the Thrown Exceptions (page 120) metric, which does not collect stack traces, when you start your session.
Monitor Memory and/or Heap Activity Menu Heap Monitor Displays free and used memory sizes in the heap and garbage collection events over time. The used heap space includes live objects and dead objects that have not yet been collected as garbage. Specifically, this visualizer shows the heap in use by objects in the eden space and in the old and survivor generations, but does not include the permanent generation. (See Basic Garbage Collection Concepts (page 87) if you are unfamiliar with these terms.
• • When you select a high level of detail, 1 to 20 minutes, and the heap size does not go to the local maximum before a garbage collection happens, it could indicate excessive calls to System.gc(). See Identifying Excessive Calls to System.gc() (page 41). When you select coarse granularity, 1 to 24 hours, you may notice the overall change of behavior in heap size and garbage collection pattern. This can help with understanding the correlation between the application load and the pressure on the heap.
Guidelines • • • • • For a healthy heap, minor collections should dominate major garbage collections. If the number of minor collections is too small compared to the number of major garbage collections, the young generation of the heap may be too small. If the heap size shown by garbage collections converges towards the heap limit, the application has run out of memory, or soon will run out. If the old generation is too small, the application will run out of memory.
Details • For HP Java 1.5.0.12 and later or 6.0.01 or later, this visualizer distinguishes between major and minor garbage collections such as full GC and scavenge.
Figure 8-7 Monitoring Metric: Percentage of Time Spent in Garbage Collection Related Topics • • • • • Checking for Proper Heap Sizing (page 43) Reviewing the Percentage of Time Spent in Garbage Collection (page 41) Allocated Object Statistics by Class (page 127) Allocating Method Statistics (page 128) Identifying Excessive Object Allocation (page 44) Unfinalized Objects Shows a measure of the objects that have not been finalized at each garbage collection during the monitoring period.
Related Topics • • • • Unfinalized Queue Growth (page 115) Thread Histogram (page 131) Reading Profiling Histograms (page 73) Profiling with -Xeprof (page 61) Allocated Object Statistics by Class Shows object allocation statistics according to the object type allocated. Guidelines A typical large Java application allocates a lot of strings. This value can reach 25 percent of all allocations.
Figure 8-8 Monitoring Metric: Allocated Object Statistics by Class Related Topics • • • • • Identifying Excessive Object Allocation (page 44) JVM Agent Options (page 17) “Data Sampling Considerations” Mark an Item for Search (page 178) Find a Search Pattern (page 179) Allocating Method Statistics Shows the methods that allocate the most objects. This metric is useful when you choose to decrease heap pressure by modifying the application code.
Figure 8-9 Monitoring Metric: Allocating Method Statistics Related Topics • • • • • JVM Agent Options (page 17) Allocated Object Statistics by Class (page 127) “Data Sampling Considerations” Mark an Item for Search (page 178) Find a Search Pattern (page 179) Current Live Heap Objects Use this visualizer to obtain an immediate data summary of live objects in the heap each time that you click the Refresh Live Objects button.
Figure 8-10 Monitoring Metric: Current Live Heap Objects Table 8-1 Data Shown in Current Live Heap Objects Visualizer Column Heading Description Class Name of class to which object belongs % Heap Used Percent of allocated heap used Bytes Cumulative size occupied by the object (in bytes) +/- First Bytes The total change in the amount of bytes held for this class since the first snapshot was taken.
Figure 8-11 Copying Selected Current Live Heap Objects Data into Buffer To select all data for use in a spreadsheet, click Copy All to Buffer in the tool bar. Figure 8-12 Copying All Current Live Heap Objects Data into Buffer Click File →Save to capture all data as an ASCII text file that you can save onto your local machine. Related topics • Heap Monitor (page 122) Monitor Threads and/or Locks Menu Thread Histogram Displays thread states over time. Thread data arrives in time slices.
Lock Contention The thread is delayed while attempting to enter a Java monitor that is already acquired by another thread. Running All remaining cases. Guidelines • • • • • • Large amounts of red in Thread Histogram indicate heavy lock contention, which is usually a possible problem. On the other hand, large amounts of green indicate a potential processing capacity for the involved threads.
Lock Contention Provides lock contention statistics. The percentages for each method represent how much of the total lock contention observed occurred in that method. For example, if there was a single instance of lock contention during a run, that method would show 100 percent. Therefore, methods that show a high percentage of lock contention may not be a problem, unless you see a significant amount lock contention in your application.
Considerations When Comparing Lock Contention and Thread Histogram Metrics: Lock contention data is sampled less frequently than thread histogram data. When there is lock contention with a short lifespan, a small amount of lock contention might appear in the Thread Histogram, but not be shown in the Lock Contention percentages. This can happen when the contention occurs during a time when the Thread Histogram sample is being taken, but the lock contention sample is not.
Related Topics • • • Identifying Excessive Method Compilation (page 46) Method Compilation Frequency (page 135) Find a Search Pattern (page 179) Method Compilation Frequency Produces a graph that shows the compilation frequency. This is a companion to the existing Method Compilation Count. The Method Compilation Frequency metric provides a view of how much effort the JVM is spending on method compilation.
Figure 8-17 Monitoring Metric: Loaded Classes Related Topics • • Identifying Too Many Classes Loaded (page 47) Class Loaders (page 150) Percent (%) CPU Utilization Displays total system and process CPU consumption. Guidelines • • • 136 Percentages are displayed as number of CPUs x 100%. Excessive use of CPU resources (greater than 80% of the total number of CPUs) may indicate that the application load limit is close, even though the application may appear to be performing well.
Figure 8-18 Percent CPU Utilization for System and Processes Related Topics • • Process CPU Usage Alert (page 113) System CPU Usage Alert (page 114) Viewing Profiling or GC Data in HPjmeter To view metrics for saved data: • From the main console, click File→Open to open a file browser window and locate your saved file. You can also drag and drop the file onto the console. Selecting the file automatically opens a data viewer.
On the Summary tab, as with any field whose value is too long for its column width, if the JVM Arguments value shown at the bottom of the upper panel is extensive and runs beyond its column, you can mouse over the value to display a yellow pop-up box that shows all the JVM arguments. When you mouse away from it, the yellow box disappears.
The profile displays are categorized as follows under the Metrics menu: • • • • Profile Code and/or CPU Activity (page 140) Profile Memory and/or Heap Activity (page 144) Profile by Threads (page 152) (Threads/Locks submenu) Profile by Locks (page 153) (Threads/Locks submenu) These heuristic displays appear under the Estimate menu: • • • Inline Candidates (page 156) Exceptions Thrown (page 156) Memory Leaks (page 156) See also: • • • Profiling Applications (page 58) Using Visualizer Tool Bars (page 176
Menu Choices The menu bar can contain one or more of these choices, depending on the data type and visualizer you are viewing: • File — Contains selections for printing, saving, and comparing data, closing the window, or opening another window that is viewing the same data file. Opening another window is useful when you want to view more than one metric at the same time.
Double-click the method name to see a list of its callers with the cumulative number of calls from each caller. Related Topics • • • • • • Approaches to Analyzing Performance Data (page 68) Profiling with -Xeprof (page 61) Profiling with -agentlib:hprof (page 63) Data Sampling Considerations (page 188) Mark an Item for Search (page 178) Find a Search Pattern (page 179) Exclusive Method Times (CPU) Use this visualizer to see CPU time spent by methods found in the captured data.
Call Graph Tree with CPU The inclusive CPU time shown on the call graph. Related Topics • • • • • Using Call Graph Trees (page 75) Inclusive Versus Exclusive Time (page 69) Time Units (page 69) Mark an Item for Search (page 178) Find a Search Pattern (page 179) Call Graph Tree with Clock Time The inclusive clock time shown on the call graph.
Average Exclusive Method Clock Times* Exclusive Clock Method Time divided by the number of calls to the method. This is the average time spent within the method. Related Topics • • • • Inclusive Versus Exclusive Time (page 69) Time Units (page 69) Mark an Item for Search (page 178) Find a Search Pattern (page 179) Average Inclusive Method CPU Times* Inclusive CPU Method Time divided by the number of calls to the method. This gives the average CPU cost for a high-level operation.
Starvation Ratio* Shows the starvation (see above) divided by the Exclusive Method Clock Time for each method. Values close to one indicate that the method did not consume CPU, but nevertheless delayed execution of the program. Such methods are worthwhile to investigate to understand the reason of their behavior.
The estimation entirely misses the arrays, and may report inaccurate numbers. Some of the calls to constructors may remain unprofiled, or be not recognized as such by HPjmeter. It is also possible to err on the plus side, when HPjmeter does not properly recognize the class hierarchy. These metrics – Reference Graph Tree, Reference Sub-tree by Size, Residual Objects (Count), and Residual Objects (Bytes) – show the residual objects.
Unfinalized Objects The number of objects with pending finalization, arranged by their type and sorted by the number of objects of each type. These are all objects with the finalize() method available, but not yet invoked. Related Topics • • • • • Mark an Item for Search (page 178) Find a Search Pattern (page 179) Threads Histogram (page 152) Unfinalized Objects (page 126) Unfinalized Queue Growth (page 115) Reference Graph Tree Shows all live objects and references between them.
Figure 8-19 Reference Sub-Trees by Size Visualizer – Attributes table NOTE: The symbol // indicates a reference for which memory occupied has already been counted elsewhere in the tree, and therefore, it is not included in the memory held by the parent node of this particular reference.
• • selected object node shows an int type with a value of 1, and a boolean type with a value of true. For instances, classes, and arrays (except string and char arrays), the Value column shows the object id. For example, the Attributes table for the selected object node shows an array type MyElement3[ ] with an object id value of 74afb930. For string and char arrays, see “String and char arrays” (page 148).
Figure 8-21 Long String Pop-up Box For char arrays, the Value column shows the character value for each element of the array: Figure 8-22 Character array Related Topics • • Mark an Item for Search (page 178) Find a Search Pattern (page 179) Using Profile Displays 149
Class Loaders This visualizer displays the name and count of class loaders found operating in the application. Figure 8-23 Class Loaders Visualizer You can double click one of the listed class loaders to get a pop-up window displaying the class types that have been loaded by that class loader.
Figure 8-24 Class Types Pop-up Window Guidelines • • Class types reside in the Permanent Generation space, which is usually more limited in size than other areas of memory. Continuing incorporation of additional instances of a class loader can begin to fill this area, creating an out-of-memory error in the Perm Gen region of memory.
Residual Objects (Count) The number of residual objects presented by object types. Residual objects are those remaining on the heap while the heap dump occurred. Not all residual objects are live. Related Topics • • Mark an Item for Search (page 178) Find a Search Pattern (page 179) Residual Objects (Bytes) The number of bytes occupied by residual objects of a given type, presented by object types.
Thread state indicator The colored areas of each line or line segment show relative proportions of time that a thread has spent in a particular state. Double-click anywhere on the line to open the pie chart and see actual percentages for the states of the selected thread. Color change In this image, the default color for starvation has been changed from white to purple using the color chooser. Use the selections in the Sort menu to change the criteria by which the threads are ordered and presented.
Do not confuse lock delay with the time spent in the java.lang.Object.wait() method, or with the time spent by a given thread within a Java monitor. Lock metrics are produced only by the -Xeprof profiling option. Note that the lock metrics do not show individual locks, but only values accumulated over all locks used by a given method. Most methods, however, contain only one synchronization statement or none at all, so identifying the contended lock should be easy if the source code is available.
• • Mark an Item for Search (page 178) Find a Search Pattern (page 179) Lock Delay - Method Inclusive Accumulated lock delay, wasted in the method acquiring the lock(s) and all methods called from it and their descendants. Related Topics • • • Inclusive Versus Exclusive Time (page 69) Mark an Item for Search (page 178) Find a Search Pattern (page 179) Lock Contention Ratio by Method* Contested Lock Claims divided by All Lock Claims for each method.
Related Topics • • • • Inclusive Versus Exclusive Time (page 69) Time Units (page 69) Mark an Item for Search (page 178) Find a Search Pattern (page 179) Exclusive Class Lock Delay* The total of the Exclusive Method Lock Delay, summed over all methods of the class. To see the list of all contributing methods, double click on the class name.
Related Topics • • • • • • Mark an Item for Search (page 178) Find a Search Pattern (page 179) Using Heuristics to Locate Possible Hot Spots (page 78) How Memory Leak Detection Works (page 189) Java Collection Leak Locations Alert (page 111) (available in monitoring mode) Array Leak Locations Alert (page 112) (available in real-time mode) Using Specialized Garbage Collection Displays GC data displays are available immediately on opening a .
NOTE: Difference in Display of Garbage Collection in the Heap Monitor and the Heap Usage After GC Visualizers Monitoring visualizers like the Heap Monitor show only object data from the eden, survivor, and old generation spaces. Visualizers that display GC data, like the Heap Usage After GC visualizer, show object data from the permanent generation, as well as from the eden, survivor, and old generation spaces.
Figure 8-28 GC Metric: Duration Related Topics • • • • Using Visualizer Tool Bars (page 176) Changing Time Interval in GC Data Visualizers (page 181) Basic Garbage Collection Concepts (page 87) Identifying Excessive Calls to System.gc() (page 41) (available in monitoring mode) Cumulative Allocation This visualizer shows the cumulative amount of storage for new objects requested by the application over time, which can indicate the overall progress of the application.
Figure 8-29 GC Metric: Cumulative Allocation Related Topics • • • • Using Visualizer Tool Bars (page 176) Changing Time Interval in GC Data Visualizers (page 181) Basic Garbage Collection Concepts (page 87) Creation Rate (page 160) Creation Rate This panel shows the incremental object creation rate for the application as observed at each garbage collection event. The increment at each data point is the amount of elapsed time since the previous GC event.
Figure 8-30 GC Metric: Creation Rate Related Topics • • • Using Visualizer Tool Bars (page 176) Changing Time Interval in GC Data Visualizers (page 181) Basic Garbage Collection Concepts (page 87) Allocation Site Statistics This visualizer displays the sites (methods) in the JDK and user's Java code where objects have been created.
possible, the most active allocation sites (the sites most likely to cause GC performance problems) are reported. The allocation data can be displayed as either combined or separated time sets. Use the toolbar button combined (the default) and separated to toggle between them. When the button is toggled to combined, the clock button is disabled, and the background color is white.
Figure 8-32 GC Metric: Separated Sets of Allocation Site Statistics. The filter button opens a pop-up dialog box where you can provide various filters to narrow down the data. A Filter entry is also listed under the Edit menu. If the sets are combined, the columns and filter dialog box show as combined. With the filter dialog box still displayed, pressing the Toggle Allocation Data into Combined or Separated Sets button would then show the columns and filter dialog box as separated.
Figure 8-33 GC Metric: Allocation Site Statistics - Combined Sets, Filter Dialog Box 164 Using Visualizer Functions
Figure 8-34 GC Metric: Allocation Site Statistics - Separated Sets, Filter Dialog Box Filter Field Descriptions: • Time – Only visible for separated sets. Select the time set from the dropdown list, which contains a sorted list of all time sets for the GC file. If there is more than one time set, the value All is also listed, which if chosen will display all time sets for this GC file in the table.
How a Snapshot of Allocation Sites Statistics is Shown in GC Visualizers Figure 8-35 Presentation of Allocation Sites Statistics and Xverbosegc Data Collected During the Same Time Period 166 Using Visualizer Functions
Figure 8-36 Presentation of Allocation Sites Statistics and Zero Preparation GC Data Collection During the Same Time Period Related Topics • • • • • Collecting Allocation Site Statistics for Viewing in HPjmeter (page 83) Collecting GC Data with Zero Preparation (page 83) Using Visualizer Tool Bars (page 176) Basic Garbage Collection Concepts (page 87) Identifying the Site of Excessive Object Allocation (page 44) User-Defined X-Y Axes This visualizer is supplied to help you to experiment with your own com
Figure 8-37 User-Defined Comparisons of GC Metrics Related Topics • • • Using Visualizer Tool Bars (page 176) Changing Time Interval in GC Data Visualizers (page 181) Basic Garbage Collection Concepts (page 87) Multiple User-Defined This visualizer is supplied to help you experiment with your own combinations of data gathered by -Xverbosegc or -Xloggc, including Glance Data (page 170) that you might have collected. The Heap Usage After GC metric is shown by default.
Figure 8-38 Multiple User-Defined Metrics Click the toolbar button to open a metric configuration pop-up box that lets you select the metrics you want to display: Figure 8-39 Multiple User-Defined Metric Configuration Box Choose the Metric Selection Mode by clicking the Single or Multiple metric radio button. Then click the check boxes to select the desired metric or combination of metrics. When you check a metric, it is immediately added to the chart. Uncheck it, and it is removed from the chart.
When displaying multiple metrics, HPjmeter tries to share the default y-axis if possible. If not, it creates additional y-axes to the right of the graph, in units appropriate to the selected metrics. For example, Figure 8-38 (page 169) shows four metrics, but only three y-axes. The units for the Heap Usage After GC and Old After metrics are MBytes, so they share the same default MBytes y-axis to the left of the graph.
Figure 8-40 GC Metric: Glance Data You can display a different Glance metric or a combination of multiple metrics using the toolbar button.
Figure 8-41 Glance Data Metric Configuration Box Click the Single or Multiple metric radio button to choose the Metric Selection Mode. Then click the check boxes to select the desired metric or combination of metrics. When you check a metric, it is immediately added to the chart. Uncheck it, and it is removed from the chart. You can also view these metrics together with GC metrics and Glance system call metrics on the Multiple User-Defined tab.
Table 8-2 Metrics Shown in Glance Adviser Visualizer (continued) Glance Adviser Metric Name HP GlancePlus Dictionary Metric Name Run Queue Average GBL_RUN_QUEUE Average number of threads waiting in the run queue over the interval. Global Private Virtual Memory GBL_MEM_VIRT Total private virtual memory (in MBs unless otherwise specified) at the end of the interval. This is the sum of the virtual allocation of private data and stack regions for all processes.
Table 8-2 Metrics Shown in Glance Adviser Visualizer (continued) Glance Adviser Metric Name HP GlancePlus Dictionary Metric Name Data Region Virtual Memory PROC_REGION_VIRT_DATA Size (in KBs unless otherwise indicated) of the total virtual memory occupied by data regions of this process. This value is not affected by the reference count since all data regions are private.
Figure 8-42 GC Metric: Glance System Call Data You can display a different Glance system call metric or a combination of multiple metrics using the toolbar button.
Figure 8-43 Glance System Call Metric Configuration Box Click the Single or Multiple metric radio button to choose the Metric Selection Mode. Then click the check boxes to select the desired metric or combination of metrics. When you check a metric, it is immediately added to the chart. Uncheck it, and it is removed from the chart. You can also view these metrics together with GC metrics and Glance Adviser metrics on the Multiple User-Defined tab.
Bring Console to Front Bring console to the front of all other HPjmeter visualizers and viewers. Compare Data Compare data from one or more comparable data sets. For details, see Comparing Profiling Data Files (page 71) or Comparing Garbage Collection Data Files (page 86). Refresh Data Refresh the visualizer presentation. When viewing a live monitoring session, the refresh will be a new snapshot of the current state of the application.
Show Hours:Minutes Show data along the X-axis according to the hour and minutes at which each event occurred. Show Cumulative Seconds Show data along the X-axis according to the cumulative number of seconds that have passed since the recording started. Connecting Lines On Show GC events with black lines connecting each representative dot in order of occurrence. Connecting Lines Off Show GC events as unconnected dots .
items in other windows, or in displays showing other monitoring metrics. The marking mechanism is based on names, so it works across different data files provided they contain items with matching names. When you select an item in a display to mark (single-click the item with the left mouse button), click the Mark button [ ] in the tool bar, or Mark to Find in the Edit menu. Marking an item copies the name and/or other properties of the current item (highlighted by blue-green color) into the search pattern.
The elements of the search pattern used for matching the items depend on the actual kind of items being searched (types, methods, objects, for example) and are a subset of the following list: • Java package name • Java class name • Java type name (classes, primitive types, and arrays) • Java method name • Java object reference (ID) As a result of the search, all found items are highlighted.
Editing the Finder For each search field , you can either type in a different value, or you can click Clear to eliminate information from the search. You can broaden the search (search on the package name only) or narrow the search (search on the class name and/or method name only). Once the fields are the way that you want them, click Find All to start the search.
Figure 8-46 The View Menu Selections in the GC Viewer To drill in further, repeat this method of defining the area of view until you reach your desired resolution. Use the slider underneath the graph to view additional ranges in the data without changing the resolution interval. The following images show a selected area on a graph and the resulting display.
Figure 8-48 New Resolution After Selection Apply Selected Interval Across All Metrics When you have your desired resolution in one metric visualizer, click the button to apply the selected time interval across all metrics in the GC viewer. This resolution will apply until the reset button is clicked or a new resolution is chosen.
Figure 8-49 Resolution After Interval Applied to All Metrics Reset Interval to Default Settings Across All Metrics Click the button to reset all metrics to their original graphic display seen on first opening the GC viewer. Changing Time Interval in Monitoring Visualizers To change the time interval currently displayed, click the drop down in the top right of a graphical monitoring visualizer, and select the time interval that you want to see.
Figure 8-50 Time Interval Use in a Monitoring Visualizer Change Color Selection for Histogram Display The console provides you with the ability to change colors on thread histogram displays in order to better support variations in color vision capability and to help you selectively improve contrast for items you are particularly interested in. To access the color changer on these displays, double-click the color rectangle (in the legend) that you would like to change. A color chooser will open.
Figure 8-51 Using the Color Changer on Histogram Displays Related Topics • • 186 Thread Histogram (page 131) Profile by Threads (page 152) Using Visualizer Functions
9 Understanding How HPjmeter Works Here are details on how HPjmeter works to collect and display metrics. Performance Overhead on Running Applications HPjmeter is designed to minimize data collection overhead on deployed applications. The most significant change you will notice is slightly longer application server startup times. Application Server Startup Time HPjmeter uses load-time bytecode instrumentation to reduce overhead in deployment situations.
agent spends almost all its time blocked on socket and first-in first-out (fifo) waits. When the node agent is managing open sessions, overhead is extremely low. The physical memory footprint of the node agent is about 1 MB when idle and 1.5 MB when active. Console Overhead You should not run the console on a machine that is a managed node to minimize impact on your system. The console uses 256 MB of memory by default.
Until the algorithm can show the data with confidence, the graphical confidence interval is displayed as a gray bar showing only relative relationship. How Memory Leak Detection Works HPjmeter constantly monitors all open sessions and analyzes the heap size after each garbage collection. It tries to detect a longtime, increasing trend in the heap sizes.
Figure 9-2 Early Indicator of Possible Memory Leak Using Heap Monitor Display HPjmeter uses the long term linear upward trends in the heap size after garbage collections to calculate the presence of memory leaks. Short-term increases in heap size are normal. HPjmeter measures this rate in MB/hour. HPjmeter does not report a leak in situations in which the heap size increase is only temporary, or in situations in which the slope of the fitted line is too small to raise an alarm.
HPjmeter also uses data collected from the JVM to enrich the quality and kind of information presented in HPjmeter monitoring metrics and to expand the number of alerting notifications HPjmeter is able to provide via the Alert Controller.
10 Troubleshooting Here is information to help you find answers and possible solutions to problems you might see. Documentation and Support You can find printable PDF versions of the user's guide and the release notes in the installation directory within the doc subdirectory. Updates to documentation are made available at the HP Business Support Center at http://www.hp.com/go/hpux-java-docs.
To fix this problem, you can make either of these changes: — Reinstall a JRE in the expected location specified in the HPjmeter scripts. — Edit the HPjmeter scripts to specify the path to an existing JRE. The scripts containing the path to the JRE are in the location defined by JM_JAVA_HOME in the bin/ directory. These are the scripts: ◦ hpjmeter - Runs the HPjmeter GUI console. ◦ run_simple_jvmagent - Runs a simple Java program together with the HPjmeter JVM agent.
• On occasion, you can successfully connect to the JVM and open a session, but discover that no metrics are available for viewing (menu options are grayed out). This can occur if the application is experiencing a long GC at the time that you open the session. Once the GC completes, the console should begin receiving data and behave normally. • You can set a log level on the console, which allows you to collect varying amounts of information about the console and node agent.
Check to see what Java version you are using. If you are running Java 1.4.2_07, try setting SHLIB_PATH, rather than using LD_LIBRARY_PATH to define the shared library location. If this doesn't work, you may need to update to a newer JDK. • You can set a log level on the JVM agent, which lets you collect varying amounts of information about the console and JVM agent. For example: $ /opt/java1.
ability to do zero preparation profiling altogether to avoid interfering with your application. To disable zero preparation profiling, use -xeprof:off Unexpected Behavior in Visualizers For best results, update the HPjmeter node agent on your server(s) when you update the console. This allows you to take advantage of new metrics or improved support for newer JDKs/JREs.
A Quick References Connecting to the HPjmeter Node Agent For information on how to connect to the node agent, see Connect to the Node Agent from the Console (page 23). Table A-1 Fields Used When Creating a Connection to the HPjmeter Node Agent. Field Expected Input Purpose Host Name Fully qualified domain name Required. Designates the location of the server running the HPjmeter node agent or the location of the designated SSH host.
Table A-2 HPjmeter Features Available by JVM Version (continued) JVM HPjmeter Feature Command Line Option or Action Java 1.4x Java 5.x Java 6.x Start and stop verbose GC data collection with zero preparation — X (5.0.14) X (6.0.02) kill -PROF pid or kill -21 pid Obtain allocation site statistics and -Xverbosegc data together — X (5.0.14) X (6.0.02) -Xverbosegc and -XX:+PrintAllocStatistics * This HPjmeter switch is optional when running Java 6.0.03 or later.
Glossary active mode An HPjmeter JVM agent that is part of an open console session and is collecting data from the JVM and application and is sending it to a console. agent A module that represents HPjmeter in your environment. HPjmeter provides two types of agents, a node agent and a JVM agent. Batch mode An HPjmeter JVM agent that is installed, configured, and is collecting data, but is not connected to a console and is not part of a session.
hot spot A section of code, typically a method, with a higher-than-normal level of execution or access requiring high CPU usage. inclusive time The accumulated total time used by all invocations of the method and all methods that were called from it directly or indirectly. inlining Replacing a call to a function with an instance of the function; usually done at compile time. Java Management Extensions See JMX . JMX™ Java Management Extensions.
SSL Secure Sockets Layer stop-the-world time Duration of a full garbage collection where processing stops during the collection time. thrashing Excessive paging due to a disparity between physical memory actually available to the application and the specified maximum size for memory usage. thread spectrum When profiling, the distribution of thread lifetime over the possible thread states. time slice Time slice entries represent the life span of the current monitoring session for an application.
Index Symbols -agentlib effect on application performance, 17, 64 JVM agent options, 17 locating Java version, 17 specifying hprof agent for Java 1.5, 64 specifying JVM agent option for Java 1.
agent running, 95 Mark item for search, 177 open alert controller, 97, 178 open session preferences, 97 Print, 177 Refresh live objects, 177 Save to file, 177 Save to snapshot file, 97 session active, 96 session unavailable, 96 Time slice life span, 96 Apply selected interval to all metrics, 178 Close file or connection, 97 Display allocation site data in combined sets, 178 Display allocation site data in separated sets, 178 Filter allocation site statistics data, 178 Reset default interval for all metrics,
arc call graph tree, 75 array leak location, 112 live sizes, 145 automatic find immediately, 70 B batch JVM agent options, 19, 38 saving session data, 38 BEA lock contention, 45 benchmark creating for performance testing, 60 guidelines, 60 bytecode instrumentation filtering, 18 bytes total allocated, 90 C call count, 75 methods, 140 call graph, 59, 68 Tree menu, 140 call graph tree call count, 75 clock time, 75 CPU time, 75 cumulative calls, 75 interpreting data, 75 lock delay, 75, 154 method call count,
connection failed node agent not running, 37 console -log option, 194 bring to front, 177 command-line options, 194 filters, 36 guide, 99 launching from HP Systems Insight Manager, 93 main data pane, 94 main window, 93 memory use, 100 OutOfMemoryError, 193 performance, 193 performance overhead, 188 reports incorrect application name, 193 requirements, 14 session preferences, 100 starting, 23, 93 starting from command line on HP-UX, 93 starting on Linux, 93 starting on Microsoft Windows, 93 status bar, 100 u
thrown, 156 uncaught, 44 uncaught statistics, 109 excessive compilation set threshold for alert, 106 exclude=, 18 exclusive method clock time, 68 exclusive method CPU time, 68 exclusive time, 69 F features, 12 and JVM version, 197 file -Xeprof options, 62 compressed, 39 default location for allocation sites statistics data, 198 default location for Xverbosegc data, 198 directories, 15 open, 97 File menu, 97, 140 file name suffix .eprof, 65 .hpjmeter, 39 .hprof.txt, 65 .
cumulative duration, 50 duration, 84 GC events total count, 50 get started console guide, 23 monitoring application, 23 Glance data, 170 collecting, 83 using javaGlanceAdviser.
launching, 47 see thread list, 52 set deadlock detection, 52 set memory thresholds, 50 set notifications, 52 track live threads, 51 using, 47 view runtime data, 52 JVM Summary, 98, 117 version and HPjmeter features, 197 JVM agent -logging option, 195 active mode, 187 command-line options, 195 concepts, 13 data icons, 95 filters, 36 performance overhead, 187 JVM agent options, 17 group_private, 17 include=, exclude=, 18 logging, 19 monitor_batch, 38 noalloc, 20 noarrayleak, 20 noexception, 20 nohotspots, 20,
HotSpot JVM, 194 MBean attributes, 54, 55 Attributes tab in JMX viewer, 54 change attributes in JMX viewer, 54 enabling notifications for attributes, 56 filter list of, 53 list of, 53 Notification tab, 56 Operations tab, 55 relational information, 56 tab navigation in JMX viewer, 53 using operations functions in the JMX viewer, 55 measurement disabled for zero preparation GC data collection, 89 enabled for zero preparation GC data collection, 89 memory allocation statistics, 48 checking heap allocation, 43
naming monitoring data files, 39 naming profile data files, 65 object allocation by class, 44, 127 object allocation by method, 44, 128 OutOfMemory error, 110 percentage spent in garbage collection, 125 saving session data, 38 specifying, 100 system call data, 174 thread histogram, 131 thread histogram and abnormal thread termination, 44 thread histogram and lock contention, 45 thread histogram and short-lived threads, 44 thrown exceptions, 120 thrown exceptions with stack trace, 120 unfinalized objects, 12
program termination, 47 overhead performance, 187 profiling, 74, 187 owner_private, 20 P package filtering, 18 paging heap monitor, 40 parallel scavenge, 88 patches required, 14, 15 PATH configure, 16 path JRE and installation, 192 pattern locating in search, 179 pause in garbage collection heap monitor, 40 peak utilization heap, 89 percent of time spent in garbage collection , 125 performance console, 193 overhead, 187 tuning, 59 perm full GC, 88 permanent generation spaces, 82 pooling threads, 44 port co
saving from application session, 38 scalability assess for application, 74 Scale, 72 Scale Special, 72 scaling data examples, 72 scavenge, 81, 87, 88 during CMS, 88 non-parallel, 81 parallel, 81, 88 print data, 80 scope adjusting, 70 automatic find immediately, 70 changing in profiling viewer, 140 scripts hpjmeter, 193 run_simple_jvmagent, 32, 33, 35, 193 setup_jvmagent_environment, 193 search mark item for, 178 patterns, 179 SecureShell tunnel, 24 security, 21 node agents, 38 session cached, 96 data icons,
abnormal termination, 44, 108 activity in JMX viewer, 52 color values for in histogram, 74 CPU time, 74 current state in JMX viewer, 52 deadlock, 34, 115 deadlocked, 45 detecting deadlock, 52 excessive, 45 filter list in JMX viewer, 52 garbage collection, 74 GC, 81 group histogram, 153 group lifetime, 153 histogram, 131 input/output time, 74 JMX tab, 51 key to states, 73 lifetime, 152 lock contention, 45, 74, 131 native idle, 74 pooling, 44 profiling histogram, 152 profiling overhead, 74 running, 131 scope
threads, 74 wall clock time, 89 warranty, 2 WebLogic excluded packages, 18 lock contention, 45 WebSphere excluded packages, 18 window main, 93 X Xbootclasspath configure, 16 Xmx console memory, 193 Xverbosegc default location of saved data file, 198 Z zero preparation GC data collection starting, 83 stopping, 83 zero preparation profiling starting, 62 stopping, 62 zooming in on a data subset, 181 214 Index
