Debugging Core Files Using HP WDB HP Part Number: 5900-1573 Published: January 2011 Edition: 5
© Copyright 2011 Hewlett-Packard Development Company, L.P Legal Notices Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor's standard commercial license. The information contained herein is subject to change without notice.
Contents About This Document................................................................................................5 Intended Audience...................................................................................................5 Typographic Conventions...........................................................................................5 Related Information...................................................................................................6 Introduction....................
Java Corefile Debugging Support.............................................................................35 Support for mmapfile command...............................................................................36 Summary...............................................................................................................39 Examples Illustrating Core File Debugging.................................................................39 FAQ..........................................................
About This Document This document describes how to debug core files and analyze the process state of an application, using HP WDB. Intended Audience This document is intended for C, Fortran, and C++ programmers who use HP WDB to debug core files generated by HP C, HP aC++, and Fortran90 compilers. Readers of this document must be familiar with the basic commands that HP WDB supports.
Related Information The following table lists the documentation available for HP WDB. Document Location Debugging with GDB /opt/langtools/wdb/doc/gdb.pdf GDB Quick Reference Card /opt/langtools/wdb/doc/refcard.pdf Getting Started With HP WDB /opt/langtools/wdb/doc/html/wdb/C/GDBtutorial.html WDB GUI Online Help /opt/langtools/wdb/doc/index.
The file command is the simplest method to analyze the cause of a core dump at the HP-UX prompt. The file command displays the signal that triggered the core dump. For example: $ file core core: core file from 'a' - received SIGBUS This example illustrates that the program dumped a core after receiving the SIGBUS signal.
• SIGSEGV A SIGSEGV signal is sent to a program when a segmentation violation occurs. A segmentation violation occurs when a process attempts to access an address that is not in the currently allocated address space of the process. Example 2 illustrates how a SIGSEGV signal can cause a core dump. Example 2 SIGSEGV Causes a Core Dump $ cat a.c int main() { int *i, j; i = (int *)0x48000000; j = *i; return 0; } $ aCC a.c -o a $ .
Example 3 SIGABRT Causes a Core Dump $ cat gdb_throw_example.c #include void foo(int i) { throw i; } int main() { foo(10); // will not be caught } $ a.out aCC runtime: Uncaught exception of type "int". Abort(coredump) Core was generated by `a.out'. Program terminated with signal 6, Aborted. (gdb) bt #0 0x60000000c0349f50:0 in kill+0x30 () from /usr/lib/hpux32/libc.so.1 #1 0x60000000c0240e90:0 in raise+0x30 () from /usr/lib/hpux32/libc.so.
If you can reproduce the problem when running the program under WDB, it is easier to use a live debugging session in WDB to debug the program, instead of debugging the core file. However, the same debug information in the program can be used for core file debugging. Support for Invoking GDB Before a Program Aborts WDB also provides the -crashdebug option to monitor the program execution and invoke the debugger when the program execution is about to abort.
$ gdb a.out -c core or $ gdb -c core • (Or) At the gdb prompt: (gdb) core core (where a.out is the executable that dumped core.) If the executable path is not provided, the debugger selects the invocation path of the process that generated the core file. The invocation path information is stored in the core file. If the invocation path is a relative path, you must specify the executable to debug the core file.
The executable and the core file inherently carry information about the list of shared libraries that were loaded at the instant of core dump. However, this list of shared libraries is referenced by pathnames (the invoked path of the shared libraries on the system where the core dump occurred). If the shared libraries are located at a path that is different from the invoked path, you must provide WDB with the path for the shared libraries.
Table 2 Commonly Used Commands for Core File Debugging (continued) Debugging Feature Command Traversing the up stack down frame Description The up and the down commands enable you to traverse (up or down) the call chain in the stack. You can traverse up to a specific number of frames in the stack if is specified. The frame command enables you to traverse the stack frame to the specified frame number, .
Table 2 Commonly Used Commands for Core File Debugging (continued) Debugging Feature Navigating the source code Command Description list [- | Enables you to navigate the source code if it has been compiled with the -g option. | | <*address> ] When no arguments are specified, it lists ten lines after or around the previous listing. The list - command lists the ten lines before a previous ten-line listing.
Table 2 Commonly Used Commands for Core File Debugging (continued) Debugging Feature Prints the target that is currently under the debugger Command Description info files The info files and info target commands print the current target, including the names of the executable and core dump files currently in use by GDB, and the files from which the symbols were loaded. With HP WDB 6.
Table 2 Commonly Used Commands for Core File Debugging (continued) Debugging Feature Command Description Unpacking the unpackcore core file along with the associated shared libraries Unpacks the tar file that is generated by the packcore command so that the debugger can use the executable and shared libraries from this bundle, when debugging the core file on a different system from the one on which the core file was originally created.
Example 4 Viewing Symbol Information by Using the nm Command This example illustrates the use of the nm command to display the symbol information for the common linker and debugger symbols, Cerrno and Cselectdraw, on an HP 9000 system: $ nm -x Cscreen_selection.o |grep Cerrno Cerrno | |undef |data | $ nm -x Cscreen_selection.
Debugging Core Files Created by Stripped Binaries (When the Symbol Table is Available) You can debug a core file that is created by a stripped binary effectively, if the symbol table for the unstripped version of the program (before the program is stripped) is available. Alternately, you can also debug the core file that is created by a stripped program if the symbol table is available from another program, which functionally uses the same symbols, but has a different link order.
Limitations for Debugging Core Files Created by Binaries Compiled Without the -g Option The following limitations apply for core files that are created by binaries compiled without the -g option: • Argument information in the stack traces is not displayed. • Local variables and type information are not displayed. • Inline frame information is not displayed. The source line information is not displayed for core files that are created by PA-RISC binaries.
NOTE: • To avoid these limitations in debugging core files created by stripped binaries, you can use the original unstripped version of the executable, if it is available. For more information about debugging stripped binaries by using the symbol table from the unstripped version of the executable, see “Debugging Core Files Created by Stripped Binaries (When the Symbol Table is Available)” (page 18).
Example 5 Debugging Core Files Created by Optimized Code, Stripped Binaries, and Code Compiled Without the -g Option Sample Program 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 // a.c // Generates coredump with 3 deep stack trace. #include
Copyright 1986 - 2001 Free Software Foundation, Inc. Hewlett-Packard Wildebeest (based on GDB) is covered by the GNU General Public License. Type "show copying" to see the conditions to change it and/or distribute copies. Type "show warranty" for warranty/support. .. Core was generated by `a.out'. Program terminated with signal 11, Segmentation fault. SEGV_ACCERR - Invalid Permissions for object #0 inline generate_core_dump () at a.
$ aCC a.c $ /opt/langtools/bin/gdb ./a.out core HP gdb for HP Itanium (32 or 64 bit) and target HP-UX 11.2x. Copyright 1986 - 2001 Free Software Foundation, Inc. Hewlett-Packard Wildebeest (based on GDB) is covered by the GNU General Public License. Type "show copying" to see the conditions to change it and/or distribute copies. Type "show warranty" for warranty/support. .. Core was generated by `a.out'. Program terminated with signal 11, Segmentation fault.
$ aCC -g a.c $ strip a.out $ /opt/langtools/bin/gdb ./a.out core HP gdb for HP Itanium (32 or 64 bit) and target HP-UX 11.2x. Copyright 1986 - 2001 Free Software Foundation, Inc. Hewlett-Packard Wildebeest (based on GDB) is covered by the GNU General Public License. Type "show copying" to see the conditions to change it and/or distribute copies. Type "show warranty" for warranty/support. .. warning: Load module ./a.out has been stripped. Debugging information is not available. (no debugging symbols found)..
Debugging a Core File Created by a Forced Core Dump To debug a core file that is created by a forced core dump, complete the following steps: 1. To dump the core for a live process, enter the following command at the gdb prompt: (gdb) dumpcore For example: (gdb) run Starting program: sample Breakpoint 3, main () at sample.c:1010 b= foo(a); (gdb) dumpcore Dumping core to the core file core.24091 (gdb) 2.
Debugging a core file on a system other than the one on which it was originally produced is supported only under the following condition: The correct system and user shared libraries are copied with the executable and core file to the other system, and the location of the shared libraries is defined by setting GDB_SHLIB_PATH or GDB_SHLIB_ROOT before debugging the core file. For more information about these variables, see the Debugging with GDB manual available at: http://www.hp.
1. 2. 3. 4. 5. 6. Invoke WDB on the core file on the system where the core file was created. Enter the packcore command to package the .tar file with the core file, the relevant libraries, and the relevant binaries. Transfer the .tar file to the required system. Enter the unpackcore command to unpack the .tar file on this system. Start debugging the core file on this system.
(libaries32.so) or that of the emulated PA-RISC library must be displayed while executing stack-related commands, such as backtrace and frame. Following is the syntax for the set debug-aries command: set debug-aries [on|off] where: on Displays the libaries32.so frames along with other frames of the application. off Displays the PA-RISC library frames along with other frames of the application. By default, the emulator library frames are not displayed during core file debugging.
info func Displays PA-RISC functions also for the regular expressions specified, if applicable. info symbol Describes the symbol at location addr for PA-RISC symbols if the symbol has a global scope. info address Describes where the PA-RISC symbol is stored if the symbol has a global scope. print Prints the values of global variables defined in PA-RISC libraries.
NOTE: If an application crashes in the aries signal handler, the debug-aries option is turned on by default and only the libaries frames are displayed. Command to Search for a Pattern in the Memory Address Space HP WDB 6.0 and later versions provide the find command to search for a pattern in the given memory address range for both live and corefile debugging. Following is the syntax for the find command: find [/size-char] [/max-count] start-address, end-address, expr1 [, expr2 ...
Example 6 Sample Output for the find command $ cat example.c #include #include int main() { char *str; str = (char *) malloc (15); strcpy(str,"hihi-hikh"); return 0; } (gdb) find &str[0], &str[15], "hi" 0x400123e0 0x400123e2 0x400123e5 3 patterns found. (gdb) find/2 &str[0], &str[15], "hi" 0x400123e0 0x400123e2 2 patterns found. (gdb) find/2b &str[0], &str[15], 0x68 0x400123e0 0x400123e2 2 patterns found. (gdb) find/2b &str[0], +10, 0x68 0x400123e0 0x400123e2 2 patterns found.
where: • &a[0] Specifies the start address of the memory address range. &a[10] Specifies the end address of the memory address range. “el”, 'l' Specifies the pattern. Using the start address (start-address), length (+length) parameter, and a pattern (expr1) find &str[0], +11, "hihi" • &str[0] Specifies the starting address. +11 Specifies the length of the memory address range, starting from &str[0]. "hihi" Specifies the pattern (expr1).
NOTE: Following are different ways of representing the /size-char and /max-count parameters: /1b /b1 /b /1 /1 /b where: 1 Specifies that find must display 1 matching pattern. b Specifies that the size of the pattern is 8 bits. Displaying run time type information HP WDB enables you to view the run time type information for C++ polymorphic objects. The following command displays the run time information for C++ polymorphic object.
Avoiding Core File Corruption for Applications Running HP-UX 11i v1 and HP-UX 11i v2 To prevent overwriting of core files from different processes for applications running HP–UX 11i v1 or 11i v2 operating systems, you must set the kernel parameter core_addpid to 1. The core file is stored in a file name, in the current directory. To store core files in a specific filesystem, you must switch to the required directory (using the cd command) and then run the required application.
3.
Following are examples that illustrate the gdb command-line options for invoking gdb on a core file: 1. Invoke gdb on a core file generated when running a 32-bit Java application on an Integrity system with /opt/java1.4/bin/java: $ gdb /opt/java1.4/bin/IA64N/java core.java 2. Invoke gdb on a core file generated when running a 64-bit Java application on an Integrity system with /opt/java1.4/bin/java -d64: $ gdb /opt/java1.4/bin/IA64W/java core.java 3.
Example 7 mmapfile command usage in GDB 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 #include #include #include #include #include #include #include
$ aCC -Ae -o mmap_test -g mmap.c $ ./mmap_test p 7aadc000 q 7aadb000 r 7aada000 s 7aad9000 t c1a91000 u c4406000 ABORT instruction (core dumped) Now, debug the core file with gdb. $ gdb mmap_test core … Core was generated by `mmap_test'. Program terminated with signal 6, Aborted. #0 0x60000000c035d750:0 in kill+0x30 () from /usr/lib/hpux32/libc.so.1 (gdb) frame 3 #3 0x4001400:0 in main () at mmap.
(gdb) p t $9 = 0x60000000c1a91000
(gdb) mmapfile testfile 0x60000000c1a91000 0 0xe1 (gdb) p t $10 = 0x60000000c1a91000 "#include \n#include \nint main(){\n foo();\n bar();\n return 0;\n}\nint foo(){\n int * res = malloc(4);\n printf(\"*res %d\\n\", *res);\n return 1;\n}\nint bar(){\n printf(\"Hel"...On the contrary, the source line number information is available for core files created by Itanium-based binaries, irrespective of whether the core file is compiled with the -g option, or not.
1. Invoke WDB on the core file, as follows: $gdb example core HP gdb Copyright 1986 - 2001 Free Software Foundation, Inc. Hewlett-Packard Wildebeest )Wildebeest is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for Wildebeest. Type "show warranty" for details. ..(no debugging symbols found)... Core was generated by `example'.
(gdb) #4 0x2420 in function_abort () from /home/u492893/example/./example • You can also directly traverse the stack by entering the number of frames as an option in the up or down command, as follows: (gdb) up 4 #4 0x2420 in function_abort () from /home/u492893/example/./example (gdb) down 4 #0 0xc01082b8 in kill () from /usr/lib/libc.
not passed up the stack, the value of the parameters are not available when you debug the core file. You can analyze the following lines from the assembler dump to view information about the function parameters: 0x23fc : stw %r26,-0x64(%sr0,%sp) 0x2400 : stw %r25,-0x68(%sr0,%sp) These lines provide information about the location of the function parameters in the stack. This calling convention for the function parameters is defined by the PA-RISC runtime architecture.
7. To determine the value of the variables, you must analyze the contents of the required memory location. In this example, the value of arg1 is an integer, and hence this value is 32(0x20). The argument, arg0, is a pointer to a structure. To arrive at the value of arg0, the offsets for the variables in the structure must be determined manually.
(gdb) x/x 0x7f7e6688 0x7f7e6688: 0x00000011 ◦ To display the char* value at the last field in the structure, enter the following command at the gdb prompt: (gdb) x/x 0x7f7e6688+0x10 0x7f7e6698: 0x40001120 To display the char value at this address, enter the following command at the gdb prompt: (gdb) x/s 0x40001120 0x40001120 <__d_trap_fptr+260>: "The meaning" ◦ To display the address of the st_two structure, enter the following command at the gdb prompt: (gdb) x/x 0x7f7e6688+0x8 0x7f7e6690: 0x7f7e669c
The values of the variables in the structure st_two are determined by using these offsets, as follows: ◦ To examine the first word of the structure st_two, enter the following command at the gdb prompt: (gdb) x/x 0x7f7e669c 0x7f7e669c: 0x40001130 To display the string value at this address, enter the following command at the gdb prompt: gdb) x/s 0x40001130 0x40001130 <__d_trap_fptr+276>: "of life" ◦ To examine the second word of the structure st_two, enter the following command at the gdb prompt: (gdb)
• Method 2: In this method, you can use the gdb convenience variables to store and manipulate memory addresses. You can use the show conv command to view the current values of the convenience variables. The nomenclature of all convenience variables is such that they start with the $ symbol.
◦ To display the float value at $xtra+0x8, enter the following command at the gdb prompt: (gdb) x/f $xtra+0x8 0x7f7e66a4: 19.2099991 ◦ To display the string value pointed to by $xtra+0xC , enter the following command at the gdb prompt: (gdb) x/s *($xtra+0xC) 0x40001138 <__d_trap_fptr+284>: "is 42" Example 9 Debugging a Core File to View Information on a Global Variable in a C program In this example, the address, &global_vars, of global_vars is required for debugging.
(gdb) x/s *($glob+0x8) 0x0: Error accessing memory address 0x0: Invalid argument. This indicates that the variable is a null pointer. 5. To display the string value pointed to by $glob+0xC, enter the following command at the gdb prompt: (gdb) x/s *($glob+0xC) 0x7f7e62f9: "/opt/softbench/bin:/usr/bin:/opt/user/bin: /opt/ansic/bin:/usr/ccs/bin:/usr/contrib/bin:/opt/net/bin: /opt/fc/bin:/opt/fcms/bin:/opt/upgrade/bin:/opt/pd/bin:/usr/bin/X11: /usr/contrib/bin/X11:/o"... 6.
Example 10 Debugging a Core File Created by a Stripped Binary When the Symbol Table is Available Sample Program The program in this example has the following global structure, global_vars: struct gvals { char *program; +0x0 int arg_count; +0x4 char *first_arg; +0x8 char *path; +0xC int secret; +0x10 }; struct gvals global_vars; Sample Debugging Session This sample debugging session illustrates how to debug a core file that is created by the stripped binary of this program.
(gdb) set $glob=0x40001180 (gdb) x/s *($glob+0x0) 0x7f7e6000: "./example" (gdb) x/x $glob+0x4 0x40001184: 0x00000001 (gdb) x/s *($glob+0x8) 0x0: Error accessing memory address 0x0: Invalid argument. (gdb) x/s *($glob+0xc) 0x7f7e62f9: "/opt/softbench/bin:/usr/bin:/opt/butthead/bin:/opt/an sic/bin:/usr/ccs/bin:/usr/contrib/bin:/opt/nettladm/bin:/opt/fc/bi n:/opt/fcms/bin:/opt/upgrade/bin:/opt/pd/bin:/usr/bin/X11:/usr/con trib/bin/X11:/o"...
Example 11 Debugging of a Core File Created by a Stripped Binary When the Symbol Table is Available from Another Program In this example, three copies of a program (program a1, program a2, and program a3 ) are compiled and linked with a different order. For example: Program a1 is stripped. Program a2 is an unstripped copy of program a1. Program a3 is functionally the same as program a1. However, the code and the symbols are in a different link order.
#2 0xc01d00dc in abort_C () from /usr/lib/libc.2 #3 0xc01d0134 in abort () from /usr/lib/libc.2 #4 0x2498 in b () from /home/shane/test/./a1 #5 0x2430 in main () from /home/shane/test/./a1 (gdb) symbol a3 Reading symbols from a3...(no debugging symbols found)...done. (gdb) bt #0 0xc01f2740 in kill () from /usr/lib/libc.2 #1 0xc018fc94 in raise () from /usr/lib/libc.2 #2 0xc01d00dc in abort_C () from /usr/lib/libc.2 #3 0xc01d0134 in abort () from /usr/lib/libc.2 #4 0x2498 in a () from /home/shane/test/.
Example 12 Core File Debugging Session for a Stripped Binary When the Symbol Table is Available from Another Program This example is similar to Example 11 (page 52). This example illustrates the debugging of a stripped binary when the symbol table is available from another program that uses the same symbols. The programs, example.c and example2.c , have the same symbol table. Sample Program 1 $ cat example.c #include #include
function_a(&one, 32); } int function_a(struct st_one *a, int b) { function_b(b, a); } int function_b(int a, struct st_one *b) { function_abort(b, a); } int function_abort(struct st_one *a, int b) { a->three->b=99; abort(); } Sample Program 2 $ cat example2.c struct st_two { char *a; int b; float c; char *d; }; struct st_one { int one; char *two; struct st_two *three; int *four; char *five; }; main() { } Sample Debugging Session In this example,example.c is compiled and stripped. The program,example2.
$ aCC -g example2.c -o example2 $ aCC -g example.c -o example $ strip example $ ./example Abort(coredump) $ gdb example core HP gdb ... Core was generated by `example'. Program terminated with signal 6, Aborted. warning: The shared libraries were not privately mapped; setting a breakpoint in a shared library will not work until you rerun the program. (no debugging symbols found)...(no debugging symbols found)... (no debugging symbols found)...#0 0xc01082b8 in kill () from /usr/lib/libc.
You can use the print command to print the values of the structure struct st_one , as follows: (gdb) p *(struct st_one *)(*(0x7f7e67b0-100)) $1 = {one = 17, two = 0x40001140 "NOT!", three = 0x7f7e66ac, four = 0x7f7e6698, five = 0x40001120 "The meaning"} (gdb) x/x (0x7f7e67b0-100) 0x7f7e674c: 0x7f7e6698 (gdb) x/x (*(0x7f7e67b0-100)) 0x7f7e6698: 0x00000011 From previous disassembling we know that the pointer to the struct st_one was stored at -0x64 (-100) on from $sp in the function_abort routine.
However, the shared memory segments can be dumped into the core file if you set the following kernel symbols: • core_addshmem_read This kernel symbol controls if shared memory segments that are mapped read-only into a process are dumped in a core file.
If the core file is not truncated or corrupted, the output of the elfdump -o -S core is as follows: core: Type CoreVer CoreKern CoreComm CoreProc CoreLoad CoreMMF . . .
