Does hot deployment really means “HOT”?

Every now and then we get calls from customer having troubles with ASP.NET deployment, usually in cluster/NLB environment but sometimes also in single servers; what they usually have in common is the complexity of the web application, maybe made of dozens (if not hundreds) of pages, controls etc… Typical questions are:

• I am deploying my new pages and dlls but then suddenly the application is restarted, and my user lose their work and have to login again
• After the deployment the application is slow at startup
• In my NLB I randomly get inconsistent behaviors and weird exceptions we never got before
• We are using a tool to synchronize files and folders, and during the sync for a few seconds we receive a lot of errors telling that some dlls are not available, but after some more seconds everything is running fine again

There are a few variations to the above, but you got the point. I think this might be due to a misunderstanding (of should I say the documentation is not clear enough?) about deployment capabilities of ASP.NET and what is recommended for complex/full loaded applications; the point is that ASP.NET supports hot deployment meaning that whenever you update a “core” component (the content of the /bin folder, for instance) the runtime finishes serving queued requests and then reload the application to reflect your changes. But this also means that a number of things are happening, (roughly) including recompiling the the new assemblies, re-creating the AppDomain to host the application, go through security checks and load the modules into the AppDomain.

You can easily imagine that for a complex application those steps might require some time during which we application is not able to respond to incoming user’s requests. If we then consider a cluster/NBL environment, where every request from your users might be served by a different server… if you’re in the middle of your deployment a user might be “bounced” from a server with the new dlls already installed and another with the old ones still in place, bad thinks will happen then… ?

A few days ago one of those cases came in, and the customer was reporting a deadlock problem in his w3wp.exe, with the following entry in his event log:

Event Type:     Warning 
Event Source:   W3SVC-WP 
Event Category: None 
Event ID:  2262 
Date:      18/10/2007 
Time:      15:46:10 
User:      N/A 
Computer:  <computername> 

Description: 
ISAPI 'c:\windows\microsoft.net\framework\v2.0.50727\aspnet_isapi.dll' reported itself as unhealthy for the following reason: 'Deadlock detected'. 




Event Type:     Warning 
Event Source:   W3SVC 
Event Category: None 
Event ID:  1013 
Date:      18/10/2007 
Time:      15:47:43 
User:      N/A 
Computer:  <computername> 
Description: 
A process serving application pool 'DefaultAppPool' exceeded time limits during shut down. The process id was '5780'.

We captured a dump as described in How to generate a dump file when ASP.NET deadlocks in IIS 6.0, and the first thing to note is that almost all the threads in the process were waiting on a stack exactly like the following:

[HelperMethodFrame: 06f9e968] System.Threading.Monitor.Enter(System.Object) 
System.Web.Compilation.CompilationLock.GetLock(Boolean ByRef) 
System.Web.Compilation.BuildManager.GetVPathBuildResultInternal(System.Web.VirtualPath, Boolean, Boolean, Boolean) 
System.Web.Compilation.BuildManager.GetVPathBuildResultWithNoAssert(System.Web.HttpContext, System.Web.VirtualPath, Boolean, Boolean, Boolean) 
System.Web.Compilation.BuildManager.GetVirtualPathObjectFactory(System.Web.VirtualPath, System.Web.HttpContext, Boolean, Boolean) 
System.Web.Compilation.BuildManager.CreateInstanceFromVirtualPath(System.Web.VirtualPath, System.Type, System.Web.HttpContext, Boolean, Boolean) 
System.Web.UI.PageHandlerFactory.GetHandlerHelper(System.Web.HttpContext, System.String, System.Web.VirtualPath, System.String) 
System.Web.UI.PageHandlerFactory.System.Web.IHttpHandlerFactory2.GetHandler(System.Web.HttpContext, System.String, System.Web.VirtualPath, System.String) 
System.Web.HttpApplication.MapHttpHandler(System.Web.HttpContext, System.String, System.Web.VirtualPath, System.String, Boolean) 
System.Web.HttpApplication+MapHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() 
System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef) 
System.Web.HttpApplication.ResumeSteps(System.Exception) 
System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object) 
System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest) 
System.Web.HttpRuntime.ProcessRequestNoDemand(System.Web.HttpWorkerRequest) 
System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr, Int32) 
[ContextTransitionFrame: 06f9ee00] 
[GCFrame: 06f9ee50] 
[ComMethodFrame: 06f9efa8]

As you can see we are in the middle of a massive compilation, most likely triggered by the application update the customer mentioned.

In the meantime there was one AppDomains which was shutting down because it reached its allowed compilation limit before recycling (again, this sound really like a consequence of the massive compilation going on in the meantime in other threads):

HttpRuntime 0x143fcc24: 
_shutDownStack:    at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo) 
   at System.Environment.get_StackTrace() 
   at System.Web.HttpRuntime.ShutdownAppDomain() 
   at System.Web.Hosting.HostingEnvironment.ShutdownThisAppDomainOnce() 
   at System.Web.Hosting.HostingEnvironment.InitiateShutdownWorkItemCallback(Object state) 
   at System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(Object state) 
   at System.Threading.ExecutionContext.runTryCode(Object userData) 
   at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData) 
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) 
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) 
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state) 

_shutDownMessage: Recompilation limit of 15 reached 
HostingEnvironment caused shutdown 
_shutdownInProgress: 1 
_requestQueue: 0x144255a0 
_appDomainAppPath: C:\Inetpub\wwwroot 
_appDomainAppId: /LM/W3SVC/235991876/Root

Imagine a situation where while the server is still receiving incoming request from the clients, the application’s files are being update and the runtime needs to go through the initialization steps: it’s a risky situation where it’s easy to find yourself stuck in the middle of something, likely with performance and locking problems and eventually with a hang or even worse a deadlock.

Despite the fact that ASP.NET supports “hot deployment” (i.e. you can override application files and the runtime will automatically load the new version at next request), but this is not recommended in “hot” production environments and in particular in a cluster/NBL configuration, where there is always the risk that a request is served first by the updated server, while the next one might be served by the second server which is not updated yet, causing unpredictable behaviors; in such situations it’s always a good idea to schedule your deployments in a “maintenance window” where you can temporarily take offline the application, update everything you need and put it back online when done (if you’re just updating application’s files should be a matter of a couple of minutes, just the time needed for the transfer and restart the services).

Depending on your needs you may also want to edit your web.config and increase the number of compilations allowed before recycling the AppDomain by setting a higher value for numRecompilesBeforeAppRestart (I suggest to not set it higher than 50)

<compilation 
    tempDirectory = "" 
    debug = "false" 
    strict = "false" 
    explicit = "true" 
    batch = "true" 
    urlLinePragmas = "false" 
    batchTimeout = "900"  
    maxBatchSize = "1000" 
    maxBatchGeneratedFileSize = "1000" 
    numRecompilesBeforeAppRestart = "15" 
    defaultLanguage = "vb" 
    assemblyPostProcessorType = "" 
/>

Carlo

Quote of the day:
First you’re an unknown, then you write one book and you move up to obscurity. – Martin Myers