HP Caliper Ktrace Features Guide Release 5.2 March 2010 HP Part Number: 5969-7017 Published: March 2010 Edition: 3.
© Copyright 2010 Hewlett-Packard Development Company, L.P. Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor's standard commercial license. The information contained herein is subject to change without notice.
Table of Contents About This Document.....................................................................................................................7 1 HP Caliper Ktrace Feature Overview..........................................................................................11 What Is HP Caliper Ktrace?................................................................................................11 Ktrace Command Format................................................................................
A Sample ktracedump Output.......................................................................................................33 B Errors and Warning Messages...................................................................................................35 Index..........................................................................................................................................
List of Figures 1-1 Trace Buffers................................................................................................................
List of Tables 2-1 3-1 B-1 B-2 B-3 6 ktracer options.............................................................................................................20 ktracedump options..................................................................................................27 ktracer —a Errors....................................................................................................35 ktracer -a func Diagnostics ..............................................................................
About This Document This document describes how to use the HP Caliper Ktrace tracing features to analyze applications running on HP-UX Integrity servers. NOTE: For the latest version of this document, go to the HP Caliper Web site at the following URL and click on Documentation in the Product Information box: http://hp.com/go/caliper This document is sometimes updated after a release. The document publication date appears on the title page.
Ctrl+x A key sequence. A sequence such as Ctrl+x indicates that you must hold down the key labeled Ctrl while you press another key or mouse button. ENVIRONMENT VARIABLE The name of an environment variable, for example, PATH. [ERROR NAME] The name of an error, usually returned in the errno variable. File name The name of a file, a measurement, or an event set. Key The name of a keyboard key. Return and Enter both refer to the same key. Term The defined use of an important word or phrase.
HP Encourages Your Questions You can get help with using and understanding HP Caliper by sending email to the HP Caliper development team. Please send questions to caliper-help@cup.hp.com. You can also see the HP Caliper Technical Support web page.
1 HP Caliper Ktrace Feature Overview What Is HP Caliper Ktrace? HP-UX kernel tracing is a powerful, acclaimed, indispensable facility for debugging kernel problems and for understanding kernel behavior. It gives insight into kernel events and activity by capturing data across all CPUS about kernel function call sequence, parameters, and timing. The Ktrace features provide additional analysis tools for applications.
• • • • • Quickly identify the source of repeatable problems. Determine if error routines are being invoked. Determine if a function is being passed the proper parameters. Identify who failed to release a resource. Identify who shut off interrupts. You analyze performance issues: • Determine what an unresponsive CPU is doing. • Identify a process that is switched out during a critical time. • Determine if caller, callee sequences and timing have a path length problem.
Supported Hardware and Operating Systems The Ktrace features are supported on HP Integrity systems and HP Integrity Virtual Machines (HPVM) running HP-UX 11i v3 Update 3 or later. User Requirements and Security Restrictions ktracer requires superuser privilege. The security of ktracer depends upon the protection of superuser privilege. For systems using fine-grained privileges, PRIV_DEVOPS is required (see the privileges(5) manpage). Only one superuser at a time can run kracer on a system.
How does Kernel Tracing Work? You use ktracer to define trace points to be associated with functions being traced. A trace point is similar to a tag. During kernel tracing, the kernel creates a trace record each time it encounters a function with a trace point. The trace record contains information about the function call and is saved in a trace buffer in kernel memory. See the “Trace Buffers” section in this chapter for more information about the trace records and trace buffers.
Figure 1-1 Trace Buffers Trace Points A trace point is a place where the kernel will branch to tracing code, collect data for a trace record, then resume normal processing. ktracer -b (begin tracing) sets a trace point at each selected function and begins data collection. ktracer —h halts all data collection and removes all trace points, restoring original performance. You can customize a set of trace points dynamically for each test you make.
Specify a function to trace (or add a function to the function list): ktracer -a func • Trace for a shorter time by specifying how many seconds to trace: ktracer -g #seconds • Trace only a single process: ktracer -p pid Traces are generated each time an installed function is executed. If a function you are interested in is on the traced function list, but you see no traces for it, this means one of three things.
_swtch() which calls swtch_to_thread(). And function real_sleep() calls _swtch(). When pm_swtch.c is optimized, real_sleep() sometimes calls swtch_to_thread() directly, and in those cases the calls to _swtch() are missing. Performance The performance cost to run ktracer depends on the frequency of calls to traced functions. Performance also depends on the processor architecture and speed, cache hit rate and memory speed. A performance estimate is a cost of 0.
2 ktracer: Kernel Tracer This chapter summarizes the ktracer command and its options. See the ktracer(1M) manpage for detailed syntax information and other examples. Syntax The following options are for getting started with ktracer: ktracer [-R] [-L] [-b] [-B] [ -w workload] [ -g numsec] The following options are for changing the function list: ktracer [-a func].... [-e [mod:]pattern]... [-l lib]... [-m mod]... [-r func]... [-S func]...
Options Table 2–1 summarizes the command line options. Details are in the ktracer(1M) manpage. Options are executed in the order specified, meaning their order on the command line makes a difference. Table 2-1 ktracer options Option Description Getting Started —R Reasonable defaults: An easy way to get started tracing everything. This option does the following tasks: clear previous tracing state, allocate trace buffers, add all functions, insert trace points, and begin tracing.
Table 2-1 ktracer options (continued) Option Description -a func Add the specified function to the list of functions to trace. You can specify the function by name or by hexadecimal or decimal address. Multiple -a options can be specified on a command line. -e [modname:]func_pattern Add all functions from the module named modname (default vmunix) that match the egrep pattern func_pattern to the traced function list. See egrep(1) and regexp(5).
Table 2-1 ktracer options (continued) Option Description -U Uninsert. Remove all trace points. The -U uninsert option reverses the effect of -I (insert). -z Remove all functions from function list. Managing Memory for Traces - Each trace record shows as one line in the ktracedump output. Each CPU has a circular trace buffer. ktracer defaults to collecting 2048 traces per CPU. Use the ktracer -A option to increase or decrease the number of traces per CPU.
Table 2-1 ktracer options (continued) Option Description —p PID Trace only the specified PID (0 for all). Sample usage would be ktracer -Z -p 1234 -R to trace all the kernel function calls that PID 1234 makes. —D ktracedump-options Pass the ktracedump-options through to the ktracedump program. Two ktracer options, -g and -w, automatically invoke ktracedump. Because ktracer executes command line options in order, the —D option must be specified before the —g or —w option.
Example Troubleshooting a Performance Problem After upgrading an Operating System, the system's benchmark performance regressed 18%. The system configuration is the same as when the last benchmark was run with the exception of a different root disk. Additional symptoms are: • • • time (1M) indicates 627 seconds of real time were used vs 557 in the previous benchmark. 28 seconds more of user time and 3 seconds more in system time. sar -d indicates 100 times longer disk avwait and avqueue.
1. 2. 3. Capture the trace data again during the last phase of sas. Accurately identify long hold times for vx_recsmp_rangelock. Map lock arg0 into a vx_inode and get its path name. (This works only for names not pushed out of the DNLC.) 4. 5. 6. Send ktracer data and problem report to the vxfs owner. The support response stated that the performance degradation was due to the need for a larger disk queue size when the OS was upgraded. Run vxtunefs to increase the max_diskq from 1 MB to 2048 MB.
3 ktracedump: Kernel Trace Reporter This chapter summarizes the ktracedump command and its options. See the ktracedump(1M) manpage for detailed syntax information and other examples. ktracedump prints a formatted report of the traces that ktracer collected. ktracedump works on live systems and on crash dumps. Syntax ktracedump [ -D | -m ] [-H] [ -J col ] [ -j col ] [-S col] [-a] [-g] [-A] [-N] [-F] [-X ext] Usage In a dump directory: ktracedump -m > kd.
Table 3-1 ktracedump options (continued) Option Description H Display which fields (columns) can be printed. ktracedump-H displays a list of all printable fields, with column name, description, column number, and an indicator of whether the column will be shown by default. A sample -H output is shown here.
Table 3-1 ktracedump options (continued) Option Description J col%format Turn on the specified column for printing. For example,: -J AbsTime Column may be specified by name or number. AbsTime is column 12, so -J 12 would do the same thing as -J AbsTime. The -J option takes an optional printf format string after the column name or number. For example, "-J AbsSec%16.
to ktracedump. Save your ktracedump report to a file if you want to view it again later before restarting ktracer. Each CPU has its own interval timer. ktracer captures the interval timer value shortly after entering each traced function, adjusts it by tod_info.offset_correction, and reports it as AbsTime. You can modify how many traces the kernel collects by using ktracer -A numtraces.
SyMsk - PSR pp/ic/pk/dt/i = pQPDI when on, _ off The PSR is the Processor Status Register, described in detail in the Itanium System Architecture manual. The full 64 bit PSR value is captured in each trace record. To compact a lot of PSR information into just 5 characters of data in the SyMsk column, ktracedump has adapted the following method, which is subject to future change. If If If If If PSR.pp PSR.ic PSR.pk PSR.dt PSR.
ktracer -R (Turn on tracing of everything possible) run workload ktracer -h (Halt tracing) ktracedump -D > ktrc.out (Produce basic trace report) Report a trace listing on a crash dump: cd dump_directory ktracedump -m > ktrace.dump Trace only calls to read and write, capture 30000 calls per CPU, and produce a trace report that includes all data in each trace. ktracer -z -Z -A 30000 -a read -a write -B run workload ktracer -h (Halt tracing) ktracedump-D -A > ktrace.
A Sample ktracedump Output Appendix A shows an example of the content of a ktrace.out file. This output was produced by running: ktracer -w workload > ktrace.out ktracer: INFO: starting workload 'workload‘at Mon Jul 9 15:12:51 2007 ktracer: INFO: finished workload 'workload' at Mon Jul 9 15:12:51 2007 Executing: ktracedump –D ktracedump 3.4.
The largest ElUSec, 436.
B Errors and Warning Messages This appendix describes some errors and warning messages you might receive. The following errors occur if the ktracer —a command fails. Table B-1 ktracer —a Errors Message: EPERM: Must be superuser or have DEVOPS privilege. See the privileges(3) manpage. Message: ENOMEM: Not enough kernel memory. Message: ENOSYS: Attempted operation is not supported. Message: EINVAL: Invalid parameter(s).
The following errors occur if ktracedump fails. Table B-3 ktracedump Errors 36 Message: EPERM: Must be superuser or have DEVOPS privilege. See the privileges(3) manpage. Message: EINVAL: Invalid parameter(s). Message: ENOSPC: Not enough file system space to save output file.
Index A W Audience of User Guide, 7 Warning messages, 35 D Diagnostic messages, 35 Documentation resources, 8 E Error messages, 35 H HP Caliper ktracer, 19 HP Ktrace overview, 11 I Integrity Virtual Machine environment using HP Ktrace in, 13 K ktracedump, 27 ktracedump options, 29 ktracedump sample output, 33 ktracer, 19 ktracer options, 23 O options, ktracedump, 29 options, ktracer, 23 Organization of User Guide, 7 output, 33 Overview of HP Ktrace, 11 R Related documentation, 8 reports, 27, 29 S
