MDeBug: An MDB Based Debugging Tool

We want to hear from you! Please send us your FEEDBACK.

The following tool may contain actual software programs in source code form. This source code is made available for developers to use as needed, pursuant to the terms and conditions of this license.

You can download MDeBug and its corresponding files here.

Introduction:

MDeBug is a tool that can be used to perform system crash dump analysis for the Solaris[tm] 8 Operating Environment (OE) and above, and application core analysis for the Solaris 9 OE and above.

The target audience for this tool is developers and system administrators who might be interested in gathering information from core files to debug a hang, a panic or an application crash.

Some of the salient features of this tool are that it:

a) has an option to do a basic or a more advanced analysis.

b) uses "mdb" (modular debugger) which is the latest kernel crash dump analysis utility.

c) can be run on a 64-bit OS to analyze a 32-bit crash dump file (please note that the reverse is not possible i.e. one cannot debug a 64-bit crash dump while booted off 32-bit OS).

d) uses only "mdb" as opposed to a mix of adb, crash [which is being EOL'd (end of life'd)] and direct commands run on the system on which the crash dump file was generated.

e) captures some very useful information like "prtconf", "modinfo", /etc/system entries directly from the crash dump files.

f) provides an optional "findleaks" macro output which can provide useful information to debug kernel memory leaks.

g) can perform an analysis on an application core file to print the thread stack and shared objects mapped.

MDeBug provides an option to run a basic or an advanced analysis for the kernel crash dump.

Basic analysis: Captures some of the basic information like time of crash, panic string, panic message buffer, CPU structures, process table etc.

Advanced analysis: Captures all the information captured with basic analysis in addition to some advanced information like /etc/system entries, prtconf, modinfo outputs, kernel memory allocation statistics, etc. It will also collect the threadlist macro output (in case of a system hang) and optionally detect kernel memory leaks (if kmem flags are set to an appropriate value).

Prerequisites:

1. MDB package:
   

   The following packages should be present on the system on which you want to use this tool:

   SUNWmdb - Modular Debugger
   SUNWmdbx - Modular Debugger (64-bit).

   You can verify this by typing the command:
   

   %pkginfo | grep -i mdb

2. OS version:
   

   For the best reults, you should run this tool on the same system on which the crash dump files or application core were collected.

   If this is not the case, the alternate system used must at least have the same OS version.

Instructions to Run:

1. Copy the "mdebug" shell script to the location of your crash dump files or your application core.
2. Run the script as:
   %sh ./mdebug
3. Choose the appropriate option in the startup menu.

Script Output:

The scripts output for both basic and advanced analysis is in the file by name:

outfile.`date |cut -f4 -d" "`

where `date |cut -f4 -d" "` gets replaced by the current time on the system.

In case of advanced crash dump analysis, the script creates two additional files:

threadlist.txt ==>  In case the script detects that the crash dump files are from a hung system (forced crash dump).

memleak.txt ===>  In case the script detects that "kmem_flags" is set to equal to or greater than "0xf", and if you answer "yes" to the question of whether you would want to run the "findleaks" macro.

For application core dump analysis:

outfile.appcore ===>  This file contains the ouput of an application core dump analysis.

All the files are generated in the same directory as the crash dump or core file location (current working directory).

Errors and Warnings:

All errors are expected to be due to the utility in use, namely "mdb". Hence, one needs to refer to the man pages of mdb(1) for more information on mdb.

MDeBug --  Tool.

outfile.basic --  Output from a sample basic crash dump analysis.

outfile.adv --  Output from an advanced crash dump analysis.

outfile.appcore --  Output from an application core dump analysis.

DOC ID #1121