After 4 years as a Support Engineer for ASP.NET, when I get a new case I usually can make a decently reliable idea pretty soon (often from the problem description the customer gives us, if it has enough details) about the problem complexity, how long it will take to resolve etc…, but every now and then (luckily not too often) there are occasions, as the one I’m going to talk about, where a fairly common problem as a “File not found” issue turns into something able to leave you completely puzzled and after you have considered almost every possibility you are left without any clue or idea about how to continue with troubleshooting. And again most of the times in those situations the solution comes from a fresh pair of eyes (and a new, well skilled brain) which looking at the problem from a different perspective can spot that small clue which makes the rest appear as obvious as the easiest of the problems… ?
The problem
This time the problem description was well written: the customer developed a custom authentication mechanism for web applications, defined an authentication interface which can be implemented by other developers etc… The product exposes a “DoWorkProc” interface developed with Visual Studio 2003; they also developed an HttpModule “My Authentication” filter that allows customers to setup authentication by defining an assembly that implements an IAuth interface, and the customer shipped his product with a sample authentication assembly named MyAuthenticationSample.dll. My Authentication.dll and MyAuthenticationSample.dll assemblies are copied into the bin folder of the DoWorkProc application. When DoWorkProc is first referenced, the ASP.NET worker process binds the assemblies from the DoWorkProc /bin folder along with various system assemblies into the C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\Temporary ASP.NET Files folder. This all works properly and an intermediate binding assembly is created. Now the problem: if I create a new project under Visual Studio to implement the IAuth interface and I change the assembly name and project namespace to MyAuthenticationSample2 but making sure that everything else in the project is the same, the binding process fails with message “File or assembly name masDotNet, or one of its dependencies not found” and it highlights the “<add assembly=”*”/>” line in machine.config; it appears that not all assemblies get copied to the “Temporary ASP.NET Files” folder as well.
Troubleshooting
This smelt of a versioning problem, partially supported by the fact that using the compiled sample assembly resulted in the error above, while if the customer added the source project for the control to the new solution and compiled the them altogether, it worked…
The customer sent a repro I was able to run on my machine (it is always easier when we can repro, it’s much easier to debug and have a close look at what’s going wrong), and at first I actually found a versioning problem which fitted quite well with the error we got. The runtime was not able to find the appropriate version of a file it was looking for, hence the error: in the failing sample we were trying to load masDotNet.dll version 10.6.2719.37534, while the file version I had on disk (and the same loaded in the working sample) was version 10.6.0.5.
But… why does the error message mentions masDotNet.dll while to repro I had to use MyAuthenticationSample.dll and MyAuthenticationSample2.dll? ? Well, checking with the customer it turned out that masDotNet.dll is a mixed mode (C++ .NET) dll so we actually have two version numbers for that file: the version number in the PE header (what WinDBG shows, or what you can see from Windows Explorer if you look at the dll properties) and the version number in the assembly manifest have nothing to do with each other, it is the one in the manifest that is important for .NET assembly loading (binding), and at a more careful look that version was correct.
As often in this kind of cases I asked Doug to have a look at the problem with me and after some time spent working on the repro (basically to run again through the troubleshooting I did until that moment to double check) we confirmed in neither the non-working case nor the working case, neither MyAuthentication.dll nor MyAuthenticationSample.dll (nor MyAuthenticationSample2.dll in the non-working case) reference any mas* assembly in their manifest, neither directly nor indirectly. So why having differing versions of MyAuthentication.dll and MyAuthenticationSample.dll affects the version of masDotNet.dll that we look for? Very odd… ?
We then had a look at masDotNet.dll (which is managed C++) and we saw that from a native DLL dependency point of view it depends two third party native dlls we had in the /bin folder. Well, what are they doing in the BIN folder? ASP.NET’s BIN folder is for .NET assemblies, not native DLLs and the native OS loader will not find the native DLLs after they get shadow copied. So we did what should be done with native DLLs: put them in a location that was discoverable on the system PATH and removed them from the BIN folder.
I then checked and everything worked (also clearing out “temporary asp.net files” in between each test just to be sure), in both the previously working and non-working cases!
Conclusion
There would be some additional questions to answer, such as why under some circumstances the native dlls in /bin where not a problem, while in other cases it was and consistently failed; unfortunately we did not had enough time left to continue our investigations (the customer needed urgently a working solution and preferred to not spend more time on the theory behind the issue), but here is a good bet: of the two native third party dlls, one (let’s call it A.dll) has a static dependency on the other one (call it B.dll). In the working case, B.dll was loaded in memory before A.dll, so when it was its turn it was able to find all the needed dependencies already in memory. But in the failing case for some reason (maybe because of timing, but I’m not really sure) A.dll was loaded first and then load of B.dll failed because it was not discoverable on the normal loader DLL search path.
Bottom line: the /bin folder is a sensitive place, for the above and other ones (check here for another example), be careful what you store in there! ?
Carlo
Quote of the day:
The real problem is not whether machines think but whether men do. – B. F. Skinner