Every now and then we got a call for an application which, randomly and without a specific pattern or apparent reason, shuts down all user’s sessions. Since ASP.NET 2.0 we can use Web Events to have more information on what is happening at the runtime level and this time the message we had in the event log was pretty clear: Event code: 1002 Event message: Application is shutting down. Reason: Configuration changed. Event time: Event time (UTC): Event ID: Event sequence: Event occurrence: 1 Event detail code: Application information: Application domain: /LM/W3SVC/1045621189/Root-2-1742334186915428 Trust level: Full Application Virtual Path: / Application Path: C:\Inetpub\wwwroot\ Machine name: <machinename> Process information: Process ID: 1234 Process name: w3wp.exe Account name: NT AUTHORITY\NETWORK SERVICE We used Process Monitor to track access to the relevant .config files which are usually the cause for this kind of problem, but with no luck. To make a long story short, we had a look at the events around the time when the session was lost and we noticed it usually followed some exception tracking logic the customer had used, so we had a look at the source code and found something interesting: the customer had some code to write and…
-
-
Fatal Execution Engine Error on x64 Framework
I got two FEEE cases in a row last week, both of them for the same cause so I thought to blog about it hoping to save some stress and headshakes to someone else. The affected platform is 64 bit Framework (32 bit works fine), either Web or Windows application: randomly the application crashes and if you check your event log you’ll find messages like the following: Event Type: Error Event Source: .NET Runtime Date: 12-02-2009 10:58:36 User: N/A Computer: <computername> Description: .NET Runtime version 2.0.50727.3082 – Fatal Execution Engine Error (000006427F8A5DC8) (80131506) The stack of the faulting thread looks like this: 0:025> kpL1000 Child-SP RetAddr Call Site 00000000`0643e350 00000642`78acb013 mscorwks!COMCryptography::_DecryptData+0x329 00000000`0643e590 00000642`801f4b87 mscorlib_ni!System.Security.Cryptography.CryptoAPITransform.TransformFinalBlock(<HRESULT 0x80004001>)+0x123 00000000`0643e620 00000642`801f25c7 CryptoSample!MyCryptoClass.Decrypt(<HRESULT 0x80004001>)+0xf7 00000000`0643e960 00000642`bc8e449b System_Web_ni!System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute(<HRESULT 0x80004001>)+0x257 00000000`0643ea10 00000642`bc8f2215 System_Web_ni!System.Web.HttpApplication.ExecuteStep(<HRESULT 0x80004001>)+0xab 00000000`0643eab0 00000642`bc8e3553 System_Web_ni!System.Web.HttpApplication+ApplicationStepManager.ResumeSteps(<HRESULT 0x80004001>)+0x1a5 00000000`0643eb60 00000642`bc8e7874 System_Web_ni!System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(<HRESULT 0x80004001>)+0xd3 00000000`0643ebc0 00000642`bc8e745c System_Web_ni!System.Web.HttpRuntime.ProcessRequestInternal(<HRESULT 0x80004001>)+0x1c4 00000000`0643ec50 00000642`bc8e608c System_Web_ni!System.Web.HttpRuntime.ProcessRequestNoDemand(<HRESULT 0x80004001>)+0x7c 00000000`0643ec90 00000642`7f602322 System_Web_ni!System.Web.Hosting.ISAPIRuntime.ProcessRequest(<HRESULT 0x80004001>)+0x18c 00000000`0643edc0 00000642`7f503bb3 mscorwks!CallDescrWorker+0x82 00000000`0643ee20 00000642`7f5251f8 mscorwks!CallDescrWorkerWithHandler+0xd3 00000000`0643eec0 00000642`7f525563 mscorwks!ForwardCallToManagedMethod+0x160 00000000`0643ef60 00000642`7f544738 mscorwks!COMToCLRWorkerBody+0x35b 00000000`0643f1c0 00000642`7f50c8ae mscorwks!COMToCLRWorkerDebuggerWrapper+0x50 00000000`0643f230 00000642`7f60249e mscorwks!COMToCLRWorker+0x366 00000000`0643f520 00000642`fff58293 mscorwks!GenericComCallStub+0x5e 00000000`0643f5d0 00000642`fff58633 webengine!HttpCompletion::ProcessRequestInManagedCode+0x2a3 00000000`0643fa80 00000642`fff9abf4 webengine!HttpCompletion::ProcessCompletion+0x63 00000000`0643fac0 00000642`7f48dc77 webengine!CorThreadPoolWorkitemCallback+0x24 00000000`0643faf0 00000642`7f4a289a mscorwks!UnManagedPerAppDomainTPCount::DispatchWorkItem+0x157 00000000`0643fb90 00000642`7f41f0ac mscorwks!ThreadpoolMgr::WorkerThreadStart+0x1ba 00000000`0643fc30 00000000`77d6b6da mscorwks!Thread::intermediateThreadProc+0x78 00000000`0643ff80 00000000`00000000 kernel32!BaseThreadStart+0x3a 0:025> !clrstac OS Thread Id: 0x1258 (25) Child-SP RetAddr Call Site…
-
minWorkerThreads and autoConfig
A couple of days ago I was helping a colleague of mine with some tests and I had the need to change the ThreadPool configuration; more precisely, I wanted to increase the minWorkerThreads value. Everything was working fine at first, because I was changing the value by code using the SetMinThreads method, but I also wanted to test it changing the values from my machine.config. First thing, to be on the safe side I set autoConfig=”false” (see why here) and set my <processModel> element as follows: <processModel autoConfig=”false” minWorkerThreads=”23” /> Why set it to 23? Simply because I did not want it to match any default. I run my test page, attached Windbg to the process and had a look at !threadpool output: something was definitely wrong, as it was telling that minWorkerThreads was set to 2 (the default value is 1 * CPU number = 2 for my machine): 0:029> !threadpool CPU utilization 48% Worker Thread: Total: 2 Running: 0 Idle: 2 MaxLimit: 40 MinLimit: 2 Work Request in Queue: 0 -------------------------------------- Number of Timers: 8 -------------------------------------- Completion Port Thread:Total: 1 Free: 1 MaxFree: 4 CurrentLimit: 0 MaxLimit: 40 MinLimit: 2 Now I see from here and with some…
-
HashTable insert failed. Load factor too high
I already encountered a few recurrences of this errors and every time it has been a torture to capture a correct set of data (especially memory dumps) to review because this usually happens sporadically and you could have to wait weeks if not months before getting it again. There are a couple of hotfix available for this error message (see http://support.microsoft.com/default.aspx?scid=kb;EN-US;831730 and http://support.microsoft.com/default.aspx?scid=kb;EN-US;893314) but you already have them if you’re on the latest Service Pack for the .NET Framework. Unless you’re using HashTable objects yourself… If that’s the case you might be running into a race condition which leads to the error, so you might want to consider to use the following code to synchronize the HashTable using SyncLock: Dim MyHashTable as new Hashtable() SyncLock MyHashtable.Syncroot MyHashtable.Add (myKey, myValue) End SyncLock As an alternative you can use the Hashtable.Synchronized() method to return a synchronized (thread-safe) wrapper for the HashTable object. Carlo Quote of the day: Everyone is a genius at least once a year. The real geniuses simply have their bright ideas closer together. – Georg Christoph Lichtenberg
-
autoConfig=”false” effects
My post on autoConfig triggered and interesting discussion with my colleague Cenk about which is the real effect of this setting; one of his customers asked what to do if they needed to adjust only one or two values and not all the ones discussed in Contention, poor performance, and deadlocks when you make Web service requests from ASP.NET applications, they wanted to be sure they could change only what they needed to fine tune and forget about the rest. The short answer is: you can change only what you need to and leave he rest as is, since we have default values hardcoded to improve performance so you’re safe (provided the values you’re changing make sense in the context, of course). If we have a look at System.Web.Configuration.ProcessModelSection with Windbg, we can see all the properties we have in <processModel> and _defaultValue corresponds to the defaults we have in machine.config.comments: 0:000> !do 06904868 Name: System.Web.Configuration.ProcessModelSection MethodTable: 6638dcb0 EEClass: 663a0604 Size: 64(0x40) bytes (C:\Windows\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll) Fields: MT Field Offset Type VT Attr Value Name 7910be50 400014d 30 System.Boolean 1 instance 0 _bDataToWrite 7910be50 400014e 31 System.Boolean 1 instance 0 _bModified 7910be50 400014f 32 System.Boolean 1 instance 1 _bReadOnly 7910be50 4000150 33…
-
“Invalid postback or callback argument” in ASP.NET
I saw this error twice recently, but as often happens for two completely different cases so here they are, hope it helps someone to same their time… Invalid postback or callback argument. Event validation is enabled using <pages enableEventValidation="true"/> in configuration or <%@ Page EnableEventValidation="true" %> in a page. For security purposes, this feature verifies that arguments to postback or callback events originate from the server control that originally rendered them. If the data is valid and expected, use the ClientScriptManager.RegisterForEventValidation method in order to register the postback or callback data for validation Nested forms The first problematic application was dynamically building page layout and manipulating the HTML stream sent to the client; in my specific case there was some manipulation carried on the client through Javascript, but I think the same may happen if for example the HTML stream is changed through an HttpHandler after the main ASP.NET processing has been completed. The page was rendered correctly within the browser, but a postback thrown the exception above and this was due to a malformed page structure, where we had nested <form> tags like in the following example: <body> <form id="form1" runat="server"> <div> <asp:Button ID="Button1" runat="server" Text="Button" /> <form></form> </div>…
-
MaxConnection does not have effect on ASP.NET 2.0
An interesting change which you might have missed in .NET 2.0 is about the autoConfig setting for the <processModel> element. The customer was running out of available connections while accessing a remote web service and was trying to apply what’s suggested in Contention, poor performance, and deadlocks when you make Web service requests from ASP.NET applications but apparently the change had not effect: a memory dump shown still 12 available connections even if he set MaxConnection=96. Interesting to note that reverting the application to .NET 1.1 the problem disappeared. The different behavior is determined by the autoConfig value, which affects only ASP.NET applications and not Console, WinForm etc… (while the <processModel> threading configuration in .NET 1.x affected all .NET apps running on the machine): Specifies whether to automatically configure the following settings to achieve optimal performance based on the machine configuration:The maxWorkerThreads attributeThe maxIoThreads attributeThe minFreeThreads attribute of the httpRuntime element.The minLocalRequestFreeThreads attribute of the httpRuntime elementThe maxConnection attribute of the <connectionManagement> Element (Network Settings) element. The values are set according to the KB article at http://support.microsoft.com/?id=821268. This attribute does not affect the .NET Framework client applications; only ASP.NET applications. The autoConfig attribute can be one of the following values. True: Indicates that ASP.NET automatically configures…
-
Web Garden, AppDomain, statics… when Sql Server is not enough to save your session
Things have gone a bit crazy lately, we’ve been under a huge workload and the time left for blogging was virtually non existent… but the good thing is that I’ve been working on a few interesting cases I hope I’ll have time to blog about, and this is the first one. The problem The application was running on a Windows 2003 cluster and the well known solution to avoid session problems in a multi server environment is to store user sessions out-of-process, either using a State Server or Sql Server, and in this case we were using Sql Server; this is also the right approach if you are using Web Garden for your application pool and this is the situation we were in (web garden + state on Sql Server). But some session-wide arrays the customer was using were suddenly empty as soon as the customer increased the number of worker processes above 1 for the application pool, which is exactly what is not supposed to happen with <sessionState mode=”SqlServer” … />. At the beginning I thought to a configuration problem on the affected servers but the usual quick check on the metabase shown that was not the case, so…
-
Version numbers in a compiled assembly
A while ago I got a call from a customer who wanted to understand what is the meaning of the version number’s in a compiled assembly and more than that, he wanted to understand how they are calculated. He expected that at least the build number were incremented every time a build is done, but that was not always happening… As I guess you know, what that customer was talking about are the Major Version, Minor Version, Build, Revision (see Assembly versioning). This version number is physically represented as a four-part string with the following format: <major version>.<minor version>.<build number>.<revision>. For example, version 1.5.1254.15632 indicates 1 as the major version, 5 as the minor version, 1254 as the build number, and 15632 as the revision number. The version number is stored in the assembly manifest along with other identity information, including the assembly name and public key, as well as information on relationships and identities of other assemblies connected with the application. Of course you can manually specify the assembly version you want to use by setting it in the AssemblyInfo.* file, but if you leave to Visual Studio the task to calculate it, here is what happens (as usual…
-
Code Access Security hosting control in IE
PROBLEM You have a Windows Forms component hosted in Internet Explorer, and you want to catch events raised by this control from client side scripting; to avoid security errors at runtime, the control must have “Allow calls to unmanaged assemblies” permission. if you do this, you’ll notice that this works only if you give this permission to the whole Zone or Site, but does not work if you give it just to the Assembly or the URL. REASON The reasoning behind the security exception is AppDomains. Before IE can load your assembly, it must create an AppDomain to load the assembly into. When it creates this AppDomain, it assigns all the evidence it knows without loading your assembly, the Site and Zone that it is loading from. Since the AppDomain itself does not get any evidence about the signature that your assembly has (it can’t since the assembly is not loaded yet), it will not match the code group that you created giving extra trust. Now when a security demand occurs, a stack walk begins. When your assembly is checked for correct permissions, it passes, and the stack walk continues until it gets to the AppDomain. Since the AppDomain doesn’t…