There are circumstances where taking a dump is not possible or simply not convenient; imagine a situation where you’re hunting a nasty exception you don’t know where it comes from and you need to follow some complex steps to get there, or when you have dozens of that exception (let’s say an Access Violation) but you’re interested in a specific one you get only when on click on a specific button, or after you paginate long enough through a GridView (and of course you don’t want dozens of full dumps, 1 Gb each), or if you’re hunting for some security related problem and you need to inspect the security token your thread is using in a particular moment (something I did here) etc… For sure if you ask Doug he’ll have dozens of good answers to this question ?
Getting started
Let’s assume you already have correctly setup your Windbg workspace, symbols etc…, then you can go from File > Attach to a process… (or hit F6) to get the Attach to process dialog below:
Note that if you expand the tree for your target process you’ll see some information about the process’ session, this is useful if for example you have multiple instances of ASP.NET worker process which loads different versions of the runtime, or if you’re interested in a specific application pool (check this post) etc…
The Noninvasive flag is important. With noninvasive debugging, you do not have as many debugging actions, however you can minimize the debugger’s interference with the target application. In noninvasive debugging, the debugger does not actually attach to the target application, but it rather suspends all of the target’s threads and has access to the target’s memory, registers, and other such information. However, the debugger cannot control the target, so commands like “g” (Go, more on this later) do not work. If you try to execute commands that are not permitted during noninvasive debugging, you receive an error message that states “The debugger is not attached, so process execution cannot be monitored“.
When you end the debugging session, the debugger releases the target application and the application continues running. You should close the session by using q (Quit), .detach (Detach from Process), or WinDbg’s Debug > Detach Debuggee or Debug > Stop Debugging command. If you close the debugging session by closing the debugger window or by using the Exit command on the File menu in WinDbg, your target application typically stops responding. Noninvasive debugging is useful if you do not want to end the target application at the end of the session, and the target application is running on Microsoft Windows NT or Windows 2000 (these operating systems do not support detaching from a target that the debugger has actually attached to). Noninvasive debugging is also useful if the target application has stopped responding and cannot open the break thread that you need to attach.
Now we need a breakpoint
It’s important to note that when you attach the debugger to your process, you’ll actually suspend all of its threads with the result that the process will be “hung”, completely unable to serve any incoming request from your users; this is the reason why generally it’s not a good idea to attach to a production process, unless this is the only way left… Once attached, the debugger will stop on a thread waiting for your input. Depending on what you’re looking for, you might need to set a breakpoint somewhere in the application (like you can do in Visual Studio) and then let the application run until you hit the breakpoint.
The most simple command to use is:
sxe clr
The sx* commands in Windbg control the action that the debugger takes when an exception occurs in the application that is being debugged, or when certain events occur; with this command we’re telling the debugger to stop on every managed exception raised. Note that the code 0xE0434F4D means you’re dealing with a COM exception which is bubbled up to the CLR.
If you need to stop on a specific managed exception, then the SOS extension comes at hand with !StopOnException:
!StopOnException helps when you want the Windows Debugger to stop on a particular managed exception, say a System.OutOfMemoryException, but continue running if other exceptions are thrown. The command can be used in two ways:
- When you just want to stop on one particular CLR exception
At the debugger prompt, anytime after loading SOS, type:
!StopOnException -create System.OutOfMemoryException 1
The pseudo-register number (1) indicates that SOS can use register $t1 for
maintaining the breakpoint. The -create parameter allows SOS to go ahead
and set up the breakpoint as a first-chance exception. -create2 would set
it up as a 2nd-chance exception.
- When you need more complex logic for stopping on a CLR exception
!StopOnException can be used purely as a predicate in a larger expression.
If you type:
!StopOnException System.OutOfMemoryException 3
then register $t3 will be set to 1 if the last thrown exception on the
current thread is a System.OutOfMemoryException. Otherwise, $t3 will be set
to 0. Using the Windows Debugger scripting language, you could chain
such calls together to stop on various exception types. You’ll have to
manually create such predicates, for example:sxe -c “!soe System.OutOfMemoryException 3;
!soe -derived System.IOException 4;
.if(@$t3==1 || @$t4==1) { .echo ‘stop’ } .else {g}”The -derived option will cause StopOnException to set the pseudo-register to
1 even if the thrown exception type doesn’t exactly match the exception type
given, but merely derives from it. So, “-derived System.Exception” would catch
every exception in the System.Exception heirarchy.The pseudo-register number is optional. If you don’t pass a number, SOS will
use pseudo-register $t1.
Note that you can use !soe as a shortcut for !StopOnException.
If you want to test yourself, you can try this sample page:
<%@ Page Language="C#" %> <%@ Import Namespace="System.Data" %> <%@ Import Namespace="System.Data.SqlClient" %> <%@ Import Namespace="System.IO" %> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <script runat="server"> protected void FileNotFoundButton_Click(object sender, EventArgs e) { //The file does not exists StreamReader reader = new StreamReader(@"c:\temp\pippo.txt"); } protected void InvalidOperation_Click(object sender, EventArgs e) { SqlConnection conn = new SqlConnection("data source=fakeServer; database=fakeDb; integrated security=sspi"); SqlCommand cmd = new SqlCommand("select * from fakeTable", conn); //The connection should be opened!!! SqlDataReader dr = cmd.ExecuteReader(); conn.Close(); } protected void SqlException_Click(object senter, EventArgs e) { SqlConnection conn = new SqlConnection("data source=fakeServer; database=fakeDb; integrated security=sspi"); SqlCommand cmd = new SqlCommand("select * from fakeTable", conn); //We open the connection... but the database we're trying to connect to does not exists!!! conn.Open(); SqlDataReader dr = cmd.ExecuteReader(); conn.Close(); } </script> <html xmlns="http://www.w3.org/1999/xhtml"> <head runat="server"> <title>Exceptions</title> </head> <body> <form id="form1" runat="server"> <div> <asp:Button ID="FileNotFoundButton" runat="server" Text="File Not Found" OnClick="FileNotFoundButton_Click" /> <br /> <br /> <asp:Button ID="InvalidOperationButton" runat="server" Text="Invalid Operation" OnClick="InvalidOperation_Click" /> <br /> <br /> <asp:Button ID="SqlConnectionButton" runat="server" Text="Sql Exception" OnClick="SqlException_Click" /> </div> </form> </body> </html>
Let the fun begin
Run the page, attache Windbg to your worker process, assure you’re loading the correct SOS version (use .loadby sos mscorwks) and you’ll break into the process; now set the breakpoint, let’s say on first chance InvalidOperationExpcetion:
!soe -create System.InvalidOperationException g
Go back to your browser a click on the FileNotFound button, you’ll see something like the following:
0:015> g (814.119c): CLR exception - code e0434f4d (first chance) (814.119c): CLR exception - code e0434f4d (first chance) (814.119c): CLR exception - code e0434f4d (first chance) (814.119c): CLR exception - code e0434f4d (first chance)
As you can see Windbg tracks all the exceptions but does nothing special since we told it to break only on certain exceptions; see the e0434f4d code? That indicates an unmanaged exception. Why is it there? Well… sooner or later the CLR talks to the underlying OS (to simplify) which tries to open the file we asked for, but the file does not exist… so the OS throws an exception which is bubbled up to the CLR. The same happens for the SqlException but not with DivideByZeroException (this is a special exception I’ll cover later).
But if you click the right button… here we are:
'System.InvalidOperationException hit' First chance exceptions are reported before any exception handling. This exception may be expected and handled. eax=03f9d7dc ebx=e0434f4d ecx=00000000 edx=00000029 esi=03f9d868 edi=032c86c0 eip=7c812a5b esp=03f9d7d8 ebp=03f9d82c iopl=0 nv up ei pl nz na po nc cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202 KERNEL32!RaiseException+0x53: 7c812a5b 5e pop esi
Now we can have a look at the managed stack with !clrstack:
0:012> !clrstack
OS Thread Id: 0x12c4 (12)
ESP EIP
03f9d8b4 7c812a5b [HelperMethodFrame: 03f9d8b4]
03f9d958 653d7691 System.Data.SqlClient.SqlConnection.GetOpenConnection(System.String)
03f9d964 652f568a System.Data.SqlClient.SqlConnection.ValidateConnectionForExecute(System.String, System.Data.SqlClient.SqlCommand)
03f9d970 652f0eb4 System.Data.SqlClient.SqlCommand.ValidateCommand(System.String, Boolean)
03f9d9a0 652f039a System.Data.SqlClient.SqlCommand.RunExecuteReader(System.Data.CommandBehavior, System.Data.SqlClient.RunBehavior, Boolean, System.String, System.Data.Common.DbAsyncResult)
03f9d9e0 652f0331 System.Data.SqlClient.SqlCommand.RunExecuteReader(System.Data.CommandBehavior, System.Data.SqlClient.RunBehavior, Boolean, System.String)
03f9d9fc 652eee23 System.Data.SqlClient.SqlCommand.ExecuteReader(System.Data.CommandBehavior, System.String)
03f9da38 652eebe5 System.Data.SqlClient.SqlCommand.ExecuteReader()
03f9da6c 048b0954 ASP.default_aspx.InvalidOperation_Click(System.Object, System.EventArgs)
03f9da8c 6619004e System.Web.UI.WebControls.Button.OnClick(System.EventArgs)
03f9daa0 6619023c System.Web.UI.WebControls.Button.RaisePostBackEvent(System.String)
03f9dab4 661901b8 System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(System.String)
03f9dab8 6614b47c System.Web.UI.Page.RaisePostBackEvent(System.Web.UI.IPostBackEventHandler, System.String)
03f9dac0 6614b3d2 System.Web.UI.Page.RaisePostBackEvent(System.Collections.Specialized.NameValueCollection)
03f9dad0 6614e263 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)
03f9dccc 6614d8c3 System.Web.UI.Page.ProcessRequest(Boolean, Boolean)
03f9dd04 6614d80f System.Web.UI.Page.ProcessRequest()
03f9dd3c 6614d72f System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext)
03f9dd44 6614d6c2 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)
03f9dd58 048b0186 ASP.default_aspx.ProcessRequest(System.Web.HttpContext)
[...]
Have a look at the objects on this thread with !dumpstackobjects:
0:012> !dso OS Thread Id: 0x12c4 (12) ESP/REG Object Name 03f9d848 013ea8c4 System.InvalidOperationException 03f9d894 013ea8c4 System.InvalidOperationException 03f9d8a4 013dc4c0 System.String ExecuteReader 03f9d8a8 013d9c74 System.Data.ProviderBase.DbConnectionClosedNeverOpened 03f9d8bc 013ea738 System.Char[] 03f9d8d8 013ea8c4 System.InvalidOperationException 03f9d8e0 013d9c74 System.Data.ProviderBase.DbConnectionClosedNeverOpened 03f9d8e4 013ea68c System.Text.StringBuilder 03f9d908 013ea90c System.String System.InvalidOperationException 03f9d90c 013ea960 System.String System.InvalidOperationException: ExecuteReader requires an open and available Connection. The connection's current state is closed. 03f9d910 013ea7a8 System.String ExecuteReader requires an open and available Connection. The connection's current state is closed. 03f9d918 013ea7a8 System.String ExecuteReader requires an open and available Connection. The connection's current state is closed. 03f9d93c 013ea8c4 System.InvalidOperationException 03f9d958 013dbf7c System.Data.SqlClient.SqlCommand 03f9d95c 013dc4c0 System.String ExecuteReader 03f9d964 013dbf7c System.Data.SqlClient.SqlCommand 03f9d96c 013dbf7c System.Data.SqlClient.SqlCommand [...]
Here we already have some details about the exception, we can dump objects (!do <address>) etc… as if we’re analyzing a static dump file.
If you’re hunting a permission problem, the !token -n command is very handy, it dumps the security token of the current thread (which might change over time if you’re using impersonation):
0:003> !token -n Thread is not impersonating. Using process token... TS Session ID: 0 User: S-1-5-21-1721254763-462695806-1538882281-2553087 (User: EUROPE\carloc) Groups: 00 S-1-5-21-1721254763-462695806-1538882281-513 ConvertSidToFriendlyName on failed with error code 0x800006fd Attributes - Mandatory Default Enabled 01 S-1-1-0 (Well Known Group: localhost\Everyone) Attributes - Mandatory Default Enabled [...] Primary Group: S-1-5-21-1721254763-462695806-1538882281-513 ConvertSidToFriendlyName on failed with error code 0x800006fd Privs: 00 0x000000017 SeChangeNotifyPrivilege Attributes - Enabled Default 01 0x000000008 SeSecurityPrivilege Attributes - 02 0x000000011 SeBackupPrivilege Attributes - 03 0x000000012 SeRestorePrivilege Attributes - 04 0x00000000c SeSystemtimePrivilege Attributes - 05 0x000000013 SeShutdownPrivilege Attributes - 06 0x000000018 SeRemoteShutdownPrivilege Attributes - 07 0x000000009 SeTakeOwnershipPrivilege Attributes - 08 0x000000014 SeDebugPrivilege Attributes - Enabled 09 0x000000016 SeSystemEnvironmentPrivilege Attributes - 10 0x00000000b SeSystemProfilePrivilege Attributes - 11 0x00000000d SeProfileSingleProcessPrivilege Attributes - 12 0x00000000e SeIncreaseBasePriorityPrivilege Attributes - 13 0x00000000a SeLoadDriverPrivilege Attributes - Enabled 14 0x00000000f SeCreatePagefilePrivilege Attributes - 15 0x000000005 SeIncreaseQuotaPrivilege Attributes - 16 0x000000019 SeUndockPrivilege Attributes - Enabled 17 0x00000001c SeManageVolumePrivilege Attributes - 18 0x00000001d SeImpersonatePrivilege Attributes - Enabled Default 19 0x00000001e SeCreateGlobalPrivilege Attributes - Enabled Default Auth ID: 0:13927 Impersonation Level: Anonymous TokenType: Primary
If you just want to have a look at the exceptions thrown by your application without necessarily breaking in the debugger or taking a dump, Windbg can print some tracking information and let the application continue:
0:006> sxe -c "!pe;!clrstack;gc" clr 0:006> g (14c4.10a4): CLR exception - code e0434f4d (first chance) Exception object: 013d8a20 Exception type: System.IO.FileNotFoundException Message: Could not find file 'c:\temp\pippo.txt'. InnerException: <none> StackTrace (generated): <none> StackTraceString: <none> HResult: 80070002 OS Thread Id: 0x10a4 (3) ESP EIP 036dd6c8 7c812a5b [HelperMethodFrame: 036dd6c8] 036dd76c 7948d83d System.IO.__Error.WinIOError(Int32, System.String) 036dd798 79395557 System.IO.FileStream.Init(System.String, System.IO.FileMode, System.IO.FileAccess, Int32, Boolean, System.IO.FileShare, Int32, System.IO.FileOptions, SECURITY_ATTRIBUTES, System.String, Boolean) 036dd890 793983c8 System.IO.FileStream..ctor(System.String, System.IO.FileMode, System.IO.FileAccess, System.IO.FileShare, Int32, System.IO.FileOptions) 036dd8bc 793a3189 System.IO.StreamReader..ctor(System.String, System.Text.Encoding, Boolean, Int32) 036dd8e0 79497d71 System.IO.StreamReader..ctor(System.String) 036dd8f4 048b08ce ASP.default_aspx.FileNotFoundButton_Click(System.Object, System.EventArgs) 036dd90c 6619004e System.Web.UI.WebControls.Button.OnClick(System.EventArgs) 036dd920 6619023c System.Web.UI.WebControls.Button.RaisePostBackEvent(System.String) 036dd934 661901b8 System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(System.String) 036dd938 6614b47c System.Web.UI.Page.RaisePostBackEvent(System.Web.UI.IPostBackEventHandler, System.String) 036dd940 6614b3d2 System.Web.UI.Page.RaisePostBackEvent(System.Collections.Specialized.NameValueCollection) 036dd950 6614e263 System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean) 036ddb4c 6614d8c3 System.Web.UI.Page.ProcessRequest(Boolean, Boolean) 036ddb84 6614d80f System.Web.UI.Page.ProcessRequest() 036ddbbc 6614d72f System.Web.UI.Page.ProcessRequestWithNoAssert(System.Web.HttpContext) 036ddbc4 6614d6c2 System.Web.UI.Page.ProcessRequest(System.Web.HttpContext) 036ddbd8 048b0186 ASP.default_aspx.ProcessRequest(System.Web.HttpContext) 036ddbe4 65fe6bfb System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() 036ddc18 65fe3f51 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef) 036ddc58 65fe7733 System.Web.HttpApplication+ApplicationStepManager.ResumeSteps(System.Exception) 036ddca8 65fccbfe System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object) 036ddcc4 65fd19c5 System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest) 036ddcf8 65fd16b2 System.Web.HttpRuntime.ProcessRequestNoDemand(System.Web.HttpWorkerRequest) 036ddd04 6600545c System.Web.HttpRuntime.ProcessRequest(System.Web.HttpWorkerRequest) 036ddd14 033f34cd Microsoft.VisualStudio.WebHost.Request.Process() 036ddd28 033f31c2 Microsoft.VisualStudio.WebHost.Host.ProcessRequest(Microsoft.VisualStudio.WebHost.Connection) [...]
Another useful command to break on managed exception is !CheckCurrentException, but you’ll likely use it in a Windbg script or adplus configuration file:
!CheckCurrentException checks if the current exception is the one specified and stores 1 or 0 in the pseudo register supplied (1 = $t1 register). This is useful when scripting a config file to use with Adplus to get a dump only when a certain type of managed exception happens. The config file would look like:
<ADPlus>
<!– Configuring ADPlus to log all first chance exceptions –>
<!– Will still create full dump for any type of second chance exceptions –>
<Exceptions>
<Config>
<!– This is for the CLR exception –>
<Code> clr </Code>
<Actions1> Log </Actions1>
<CustomActions1> !cce System.Configuration.ConfigurationException 1; j ($t1 = 1) ‘.dump /ma /u c:\dumps\mydump.dmp’ ; ” </CustomActions1>
<ReturnAction1> gn </ReturnAction1>
<Actions2> Log;FullDump </Actions2>
</Config>
</Exceptions></ADPlus>
Remarks: This will dump on a System.Configuration.ConfigurationException only. It will
make a dump in the c:/dumps directory when that happens.
!bp instead sets a breakpoint on a managed function:
!Bp sets a breakpoint on the given function. It will break even if the function isn’t JIT’ed yet
!Bp <module name> <method name>
!Bp <module name>!<method name>
!sos.bp MyApp!MyFunction
And if you reach an interesting point you can always manually capture a dump with .dump /ma <dumpname>
What’s next?
Debugging live you can do almost everything you would do with a static dump file, but if you’re an hard core debugger you can experiment changing variable values, pointers, memory addresses contents and see the results directly in the application you’re attached to, a sort of “live memory programming”… for a small sample have a look at this posts from Roberto to cheat with Minesweeper ?. There is a rumor here in Italy of an Engineer whom while onsite for an urgent problem fixed it directly in memory and then had just to correct and deploy the new binary, but the customer was already happily using his application without further annoyances: true or legend? Well, I know this guy and I tend to thing this is a true story ?.
Another interesting and powerful feature of Windbg is the ability to see registers, source code, locals, disassembly etc… (similar to what we have in Visual Studio), this is also available with a static dump file but is especially useful when debugging live… but I guess that will be the subject of another post ?
Carlo
Quote of the day:
Whenever I hear anyone arguing for slavery, I feel a strong impulse to see it tried on him personally. – Abraham Lincoln