Now we know how it started, some basic information and terminology and why symbols are important, it’s now time to capture our first dump. How? When? Using which tool? It depends… ?
From the second post of this series we already know the difference between a hang and a crash dump and depending on the problem we are troubleshooting we know which one we need. There are different ways to capture such a dump but the most used tool in CSS is adplus, and specifically in the IIS and Internet Development team (which I belong to, by the way ?) we also use DebugDiag. I personally prefer adplus, I like it to be run at the command line and be highly configurable through its .cfg files, but there are circumstances (especially unmanaged memory leaks and hanging web applications where the customer can’t be always monitoring the server to readily capture a manual hang dump with adplus) where I find DebugDiag quite useful. Let’s see the differences.
Adplus.vbs
Adplus is essentially a VBScript (quite large, more than 5.000 lines) file which helps you capture a dump and it accepts some arguments at the command line (or it’s also possible to use a .cfg file to fine control its behavior). The basic arguments you must always provide are the dump mode (-hang or -crash) and the target process(es) (-p <process ID> or -pn <process name>):
adplus -hang -pn w3wp.exe
Optionally we can pass multiple process names (or process IDs) simply repeating the -p(n) switch; we can also use the special switch -iis to dump all the IIS related processes (inetinfo.exe, dllhost.exe and w3wp.exe/aspnet_wp.exe). We can also chose to not dump the first chance exceptions (-NoDumpOnFirst) or chose to have a full dump also on first chance (-FullOnFirst) etc…
Create your .cfg
Adplus supports a xml-based configuration file to avoid retyping lengthy and detailed command lines; you can specify some options in the .cfg and pass others directly from the command line. All options have defaults, so none is mandatory. Here is the .cfg file format:
1: <ADPlus>
2: <!-- Comments -->
3: <Settings>
4: <!-- defining basic settings (run mode, quiet mode, etc.) -->
5: </Settings>
6: <PreCommands>
7: <!-- defines a set of commands to execute before the sxe and bp commands -->
8: </PreCommands>
9: <PostCommands>
10: <!-- defines a set of commands to execute after the sxe and bp commands -->
11: </PostCommands>
12: <Exceptions>
13: <!-- commands acting on the exception actions -->
14: </Exceptions>
15: <Breakpoints>
16: <!-- defining breakpoints -->
17: </Breakpoints>
18: <HangActions>
19: <!-- defining actions for hang mode -->
20: </HangActions>
21: <LinkConfig>
22: <!-allows to link to another config file -->
23: </ LinkConfig >
24: </ADPlus>
The Setting section allows you to define the main settings for adplus, and most of them are equivalent to the command line switches; it’s possible to include multiple instances of ProcessID, ProcessName, Spawn and Notify tags:
1: <Settings>
2: <RunMode> runmode </RunMode>
3: <Option> IIS </Option>
4: <Option> Quiet </Option>
5: <Option> NoTlist </Option>
6: <Option> NoTsCheck </Option>
7: <Option> NoFreeSpaceChecking </Option>
8: <OutputDir> OutputDirectoryName </OutputDir>
9: <ProcessID> PID </ProcessID>
10: <ProcessName> ProcessName </ProcessName>
11: <GenerateScript> ScriptName </GenerateScript>
12: <Spawn> Spawn Command </Spawn>
13: <Sympath> Symbol Path </Sympath>
14: <SympathPlus> Symbol Path to add </SympathPlus>
15: <Notify> { ComputerName | UserName } </Notify>
16: <Debugger> Debugger </ Debugger >
17: </Settings>
The Exceptions section allows to customize the actions taken when the debugger breaks on the specified exception; it’s also possible to define custom exceptions to monitor:
1: <Exceptions>
2: <!-- options act on all currently defined exceptions -->
3: <Option> FullDumpOnFirstChance </Option>
4: <Option> MiniDumpOnSecondChance </Option>
5: <Option> NoDumpOnFirstChance </Option>
6: <Option> NoDumpOnSecondChance </Option>
7:
8: <!-Defining new exceptions -->
9: <NewException>
10: <Code> ExceptionCode </Code>
11: <Name> ExceptionName </Name>
12: </NewException>
13:
14: <!-Configuring already defined exceptions -->
15: <Config>
16: <Code> { ExceptionCode | AllExceptions } </Code>
17: <Actions1> Actions </Actions1>
18: <CustomActions1> CustomActions </CustomActions1>
19: <Actions2> Actions </Actions2>
20: <CustomActions2> CustomActions </CustomActions2>
21: <ReturnAction1> { G | GN | GH | Q | QD } </ReturnAction1>
22: <ReturnAction2> { G | GN | GH | Q | QD } </ReturnAction2>
23: </Config>
24: </Exceptions>
ReturnAction1 and ReturnAction2 tags are used to define additional actions for first and second chance exceptions respectively; they work just like the debugger commands and are supported only in Windows XP and later.
The HangActions section allows you specify actions to take when running adplus in hang mode:
1: <HangActions>
2: <Option> MiniDump </Option>
3: <Option> FullDump </Option>
4: <Option> NoDump </Option>
5: <Option> Clear </Option>
6: <Actions> Actions </Actions>
7: <CustomActions> CustomActions </CustomActions>
8: </HangActions>
Here is a sample configuration file (taken from WinDbg help):
1: <ADPlus>
2: <Settings>
3: <RunMode> CRASH </RunMode>
4: <Option> IIS </Option>
5: <OutputDir> c:\MyDumps </OutputDir>
6: </Settings>
7:
8: <PreCommands>
9: <Cmd> .load sos\sos.dll </Cmd>
10: </PreCommands>
11:
12: <!-- defining and configuring exceptions -->
13: <Exceptions>
14: <!-- First we redefine all exceptions -->
15: <Config>
16: <Code>AllExceptions</Code>
17: <Actions1>Log</Actions1>
18: <Actions2>MiniDump;Log;EventLog</Actions2>
19: </Config>
20: <!-- Next we define a special behavior for the Access Violation exception -->
21: <Config>
22: <Code> av </Code>
23: <Actions1>Log</Actions1>
24: <Actions2>FullDump;Log;EventLog</Actions2>
25: </Config>
26: <!-- Adding a custom exception -->
27: <NewException>
28: <Code> 0x80010005 </Code>
29: <Name> Custom_Exception_05 </Name>
30: </NewException>
31: <!-- Configuring the custom exception -->
32: <Config>
33: <Code> 0x80010005 </Code>
34: <Actions1>Log;Stack</Actions1>
35: <Actions2>FullDump;Log;EventLog</Actions2>
36: <CustomActions2> !heap;!locks </CustomActions2>
37: </Config>
38: </Exceptions>
39:
40: <!-- defining breakpoints -->
41: <Breakpoints>
42: <NewBP>
43: <Address> MyModule!MyClass::MyMethod </Address>
44: <Actions> Log;Stack;MiniDump </Actions>
45: <ReturnAction> G </ReturnAction>
46: </NewBP>
47: </Breakpoints>
48: </ADPlus>
For further details see ADPlus Configuration Files section in WinDbg help.
DebugDiag
This tool was originally developed by the IIS Escalation Engineer team (I’m not sure if they are still the ones behind it now) and essentially allows the user to create a set of rules to capture hang and crash dumps of your processes, also allowing you to specify on which exception to capture the dump, how many dumps to capture for each session, on which events etc… Very flexible.
debugdiag automated analysis
There are three circumstances where I find DebugDiag really helpful and can ease my debugging life compared to adplus. First, it makes very easy to debug in a Terminal Session, it’s just a matter of adding a flag in the Preferences tab. Second, creating a hang rule it’s possible to monitor a specific URL and DebugDiag will automatically “ping” it to check if the site is up and running; if the server does not respond, DebugDiag will capture the dump you told it. This can be very useful in situations where you need to troubleshoot a site which hangs sporadically, maybe at night, and it’s not possible to pay someone to patiently sit in front of the server waiting to run adplus at the right moment… ? When you create this kind of rule you have to specify one (or more) URLs to monitor and then you have to tell DebugDiag which process or application pool to monitor; so even if you can configure the rule on your client to monitor www.yoursite.com, of course you can’t dump a process which is running on a different machine so you must install and configure DebugDiag on the affected web server.
For memory leaks and crashes you can quickly have an idea of what’s in the dump opening it within DebugDiag and running the appropriate script in the Advanced Analysis tab; you’ll have a simple but precise report about which exception caused the crash and in which thread, or a list of the components responsible for the higher allocation rates etc… This is not enough to have a complete analysis on the problem, but it’s for sure a good point to start; of course you can still open and analyze those dumps with WinDbg as usual.
WinDbg while live debugging
Live debugging will be the subject of a future post, but just in case you attached WinDbg to a process and you’re now wondering how to dump it…
.dump /ma <output file>
Dumping on a terminal session?
If you need to dump a process on a Windows NT or Windows 2000 machine, you cannot to so through a Terminal Server session since the debugger cannot attach to a process which runs in a different Window Station, and if you try you’ll likely get a message like the following:
Win32 error 5
“Access is denied.”
This is by design but there are a couple of solutions if you need, check You cannot debug through a Terminal Server session or “Running in crash mode remotely” topic in Windbg help.
DebugDiag makes it easier to overcome this limitation, simply configuring it to run in service mode through its property dialog
Conclusion
We have different tools for different tasks but also to fit specific habits of every developer: a command line script for geeks whom want to be in full control of their machine and a GUI based debugger for speedy people and final customers whom want to start troubleshooting their applications on their own. Which one to chose is mainly a matter of taste and feelings, the environment you’re troubleshooting (if you already have one debugger installed on the machine, it does not make much sense to install the other as they are equivalent in most circumstances) etc… Make your choice and go with your preferred debugger, but if I can give you an advice don’t forget to give the other one a try, sooner or later it might come to the rescue… ?
Carlo
Quote of the Day:
Men achieve a certain greatness unawares, when working to another aim.
–Ralph Waldo Emerson