Investigating ASP.Net Memory Issues using WinDbg and SOS
by Jon Wojtowicz

You have just put the finishing touches on your latest application. Everything appears to be falling into place and then you get the bad news. The users are complaining that after they use the application for a short period of time it resets itself. Worse than that the users are complaining of out of memory exceptions and IIS needs to be reset to resolve the issue. These are all symptoms of ASP.Net memory issues, whether it be the loss of session state due to the aspnet_wp (w3wp) recycling or getting out of memory exceptions.


This is a brief overview of some of the most common issues that I have observed when analyzing ASP.Net application out of memory exceptions.
Background
The The ASP.Net worker process (aspnet_wp or w3wp) process hosts ASP.Net applications. This single process has up to 2GB of address space and is designed to recycle when the memory consumption gets to be excessive. By default the process will recycle when the memory consumption reaches 60% of the limit. This means that the worker process will recycle when the memory consumption gets to 1.2GB. My observations have indicated that the process will start develop issues when the memory consumption exceeds 600MB of memory. I have also found that there is no apparent pattern to when the process will have issues
Typically the process will recycle itself without any issues. This will only cause a loss of in memory session state. When under heavy load the ASP.Net applications will begin to throw out of memory exceptions rather than the process recycling. Another manifestation has been multiple instance of the ASP.Net worker process being in memory simultaneously with only a single instance active.
You might be asking yourself why you do not see this issue when you are performing your stress test. Most stress test environments only test a single application and do not perform load tests on the configuration running in production. This leads to an application that performs well in stress but cannot coexist with other applications due to memory consumption.
Obviously creating many large objects in your application can cause memory issues. If you review your code and see that you are declaring arrays of items that have 1 million members or more than you might want to reconsider how your applications works. This is seldom the case as the issues tend to be a little more subtle than this situation.
Microsoft has a very good article on analyzing .Net memory issues that can be found at http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnbda/html/dbgch02.asp
Tools
Any good investigator needs a set of tools to examine the evidence. Here are my tools.
  • WinDbg (Windows Debugger) from Microsoft. This tool allows you to analyze the memory contents of your application. This can be downloaded from Microsoft, http://www.microsoft.com/whdc/devtools/debugging/default.mspx
  • SOS CLR debugger extensions. This is used in conjunction with WinDbg to look inside of the managed memory. The latest version of SOS is included with the WinDbg download. There have been many good articles written of the basics of using SOS including this MSDN article, http://msdn.microsoft.com/msdnmag/issues/03/06/Bugslayer/
  • ADPlus which is included in the Microsoft debugging tools. This script allows for capturing memory dumps when a process hangs or crashes. For capturing crash dumps the following command line is used:

    ADPlus.vbs crash pn aspnet_wp.exe

    This attaches the CDB debugger to the process with the specified name. If you are running on Window 2003 where there may multiple instance of the w3_wp.exe processes, you can specify the process id (PID) by using the p switch followed by the process id. If you wish to create an immediate dump you can open Task Manager and end the process. This will simulate a process crash.
Setting Up WinDbg
Before WinDbg can be used debugging symbols must be set up. The easiest way to accomplish this is to create a server variable called _NT_SYMBOLS_ and set the appropriate path. Setting the path is described at http://www.microsoft.com/whdc/devtools/debugging/debugstart.mspx.
Obtaining the Memory Dump
Since this is related to problems on production servers the first step is to obtain the memory dump. The reason for the dump is simple, I'm not aware of any shop that allows interactive debugging on production servers. The dump is also obtained so that the memory can be analyzed without undue pressure, other than that of trying to resolve the issue.
The dump needs to be taken when the memory consumption is excessive. What is excessive memory consumption? Typically when the process memory grows beyond 600MB it should be considered high. Ideally the dump should be taken when the out of memory exceptions are thrown as this represents the best picture of the state of the process.
Once the memory dump is obtained it can be loaded into WinDbg. This is accomplished through the File -> Open Crash Dump menu item. After loading the crash dump we need to be able to look into the managed heap. For this the SOS (Son of Strike) CLR debugger extensions can be loaded using the following command:
.load clr10\sos.dll
For this demonstration I've created two simple web applications, badapp, which demonstrates the bad things that can happen and goodapp which shows the differences when an application is following the best practices. Each application contains 3 aspx pages. The dump for these two simple applications ranges from about 60-100MB. An actual production memory dump may easily exceed 1GB and can take a long time to analyze.
Common Reason for High Memory Consumption
This will be a description of some common issues of out of memory exceptions and how they appear within the managed heap.
Application Set Up For Debugging
Debugging is necessary for development of your application. By default when you create your web application in .Net you will see the following in the web.config:
<compilation

debug="true" />
This setting has several effects on the dynamic compilation of the web pages. It injects debugging code into the created assemblies. It also disables batch compilation. What this means is that every page will be compiled into its own assembly. This causes memory issues since every assembly is loaded on demand when a page is accessed. This leads to unusable gaps in the memory and severe fragmentation. The somewhat random loading and resulting fragmentation can make it difficult for the system to allocate more memory when needed.
Viewing information about the assemblies loaded in an app domain can be accomplished by using the following command in WinDbg. The stat switch provides us statistics about each app domain.
!DumpDomain stat
The result for this simple demonstration should look similar to the following. An actual production dump may have 30 or more app domains.
Domain Num Assemblies Size Assemblies Name
0x793e9d58 1 2,138,112 System Domain
0x793eb278 10 9,479,168 Shared Domain
0x00159ab8 2 2,482,176 DefaultDomain
0x0018bb68 14 7,380,992 /LM/w3svc/1/root/BadApp-1-127813414066093750
0x032326b0 10 7,312,384 /LM/w3svc/1/root/GoodApp-2-127813414228125000
The first thing that should be noted is the fact that badapp has 14 assemblies loaded while goodapp has only 10. Again the code for these applications is very similar and simple so we should not expect any differences from the dynamic compilation. We can then pick a particular domain and dump out the assembly information by using DumpDomain with the domain address.
!DumpDomain 0018bb68
To look at the assemblies loaded in an app domain we use the use the following command in WinDbg:
Output for the baddapp domain (abbreviated):

Domain: 0x18bb68
LowFrequencyHeap: 0x0018bbcc
HighFrequencyHeap: 0x0018bc24
StubHeap: 0x0018bc7c
Name: /LM/w3svc/1/root/BadApp-1-127813414066093750
Assembly: 0x0018a248 [System.Web, Version=1.0.5000.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]
ClassLoader: 0x0017e5e8
Module Name
0x001c9b80 c:\windows\assembly\gac\system.web\1.0.5000.0__b03f5f7f11d50a3a\system.web.dll

Assembly: 0x001a9088 [System, Version=1.0.5000.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]
ClassLoader: 0x0018abf8
Module Name
0x0018c8c8 c:\windows\assembly\gac\system\1.0.5000.0__b77a5c561934e089\system.dll



Assembly: 0x0022b990 [qa-y3re_]
ClassLoader: 0x001e50a8
Module Name
0x0014c520 c:\windows\microsoft.net\framework\v1.1.4322\temporary asp.net files\badapp\141137f0\f42dfee7\qa-y3re_.dll

Assembly: 0x0022ded0 [BadApp]
ClassLoader: 0x00234468
Module Name
0x03230c98 c:\windows\microsoft.net\framework\v1.1.4322\temporary asp.net files\badapp\141137f0\f42dfee7\assembly\dl2\fbba12dd\2d93ab03_fd04c601\badapp.dll



Assembly: 0x03257b88 [b8624ntz]
ClassLoader: 0x032453c8
Module Name
0x001ddb60 c:\windows\microsoft.net\framework\v1.1.4322\temporary asp.net files\badapp\141137f0\f42dfee7\b8624ntz.dll

Assembly: 0x0327b0e8 [1lixxuqx]
ClassLoader: 0x03277d60
Module Name
0x001de148 c:\windows\microsoft.net\framework\v1.1.4322\temporary asp.net files\badapp\141137f0\f42dfee7\1lixxuqx.dll

Assembly: 0x0327e498 [gvdfsokf]
ClassLoader: 0x03277dd8
Module Name
0x03274d48 c:\windows\microsoft.net\framework\v1.1.4322\temporary asp.net files\badapp\141137f0\f42dfee7\gvdfsokf.dll
For the sake of brevity the framework assemblies have been removed from the listing. What is displayed are the dynamic assemblies created from the dynamic compilation. They can be easily identified by the odd looking assembly names. These assemblies represent the three pages in the application along with the global.asax page. As a comparison the assemblies for goodApp can be dumped.
Output for the goodapp domain (abbreviated):

Domain: 0x32326b0
LowFrequencyHeap: 0x03232714
HighFrequencyHeap: 0x0323276c
StubHeap: 0x032327c4
Name: /LM/w3svc/1/root/GoodApp-2-127813414228125000



Assembly: 0x03281ff0 [GoodApp]
ClassLoader: 0x03270908
Module Name
0x032c88b0 c:\windows\microsoft.net\framework\v1.1.4322\temporary asp.net files\goodapp\822c42b8\360972c7\assembly\dl2\ce1e7872\33328a03_fd04c601\goodapp.dll



Assembly: 0x032d2cc8 [5xhp4krf]
ClassLoader: 0x03270980
Module Name
0x032d2d90 c:\windows\microsoft.net\framework\v1.1.4322\temporary asp.net files\goodapp\822c42b8\360972c7\5xhp4krf.dll
The goodapp domain only has one dynamic assembly. We have saved the additional overhead and space of having multiple small assemblies loaded. This was accomplished by changing the configuration to not compile in debug mode.
<compilation

debug="false" />
The global.asax was also removed to save an additional assembly. If the events and methods of the Global class are not used by your application, remove it. The application will run fine without it. Also removing any unused event handlers and methods can result in some savings. One peculiarity is that the global.asax always compiles into its own assembly regardless of the debug settings.
Only putting release builds and setting the compilation to not build in debug mode are very important to the application. This can not only affect the memory issue but has a drastic effect on performance. If nothing else, verify that everything is set to release on the production server.
Installing .Net 1.1 SP1 will also help with the fragmentation caused by many assemblies loading. One of the fixes in this service pack changes the way in which memory is allocated. This can help when experiencing memory issues.
Other items that create dynamic assemblies are regular expressions, serialization and XML schemas. While I advocate using these items, they are mentioned as possible suspects when trying to examine the source of dynamic assemblies.
String Concatenation
By using the + operator (& in VB.Net) to concatenate string can cause many intermediate strings to be created and released. To demonstrate this badapp uses string concatenation in the following code:
string s = "test";
string z = string.Empty;
int count = int.Parse(TextBox1.Text);
for(int i=0; i < count; ++i)
{
z += s;
}
Goodapp uses a StringBuilder to perform the same. Running this for a count of 1000 we can see the effects. The badapp consumes a considerable amount of cpu and appears to thrash the memory as it tries to allocate the memory for the string concatenation. The goodapp using the StringBuilder shows no signs of this. At 100000 cycles running badapp appears to hang the system with the allocations and garbage collection. The goodapp with the StringBuilder appears to run nearly as fast as running 1000 cycles. A dump of the managed heap can be obtained to see the number of objects in memory. Using the stat switch provides a summary of the objects found on the heap. Forgetting to use the switch will dump each object on a separate line. There will be one heap per processor on a server.
!DumpHeap stat

Results:

total 47,485 objects
Statistics:
MT Count TotalSize Class Name
0x79bf6b0c 1 12 System.IO.TextReader/NullTextReader
0x79bf48ac 1 12 System.Runtime.Remoting.Messaging.IllogicalCallContext
0x79becb5c 1 12 System.Runtime.Remoting.Messaging.CallContextSecurityData
0x79be4478 1 12 System.Runtime.Remoting.Proxies.ProxyAttribute
0x79bce950 1 12 System.Resources.FastResourceComparer
0x79bbe6b8 1 12 System.Empty
0x00be209c 6,921 206,376 System.Object[]
0x00be292c 69 329,332 System.Byte[]
0x0015ebe0 30 2,079,656 Free
0x79b925c8 16,592 2,123,684 System.String
It is not unusual for a heavily utilized server to have over 1 million strings in memory. The only true way to find sting concatenation is by a code review. Analyzing the memory dump can help in determining if concatenation is a possible issue. It is important to remember that every string literal on an aspx page gets converted to a Literal control with the Text property assigned to the static string. This can create a tremendous amount of strings on a large site even without concatenation.
Large Object Allocation
Single allocation objects larger than 85kB are placed in what is known as the large object heap. The large object heap is never deallocated nor is it compacted like the normal heap. Allocation of large objects can cause this memory to fragment and grow.
Running the badapp with an allocation of a 1000000 byte array can demonstrate what happens. Running !DumpHeap after the allocation results in the following:

0x00be292c 94 1,240,680 System.Byte[]
0x0015ebe0 14 3,318,488 Free
Total 42,882 objects, Total size: 6,428,692

Fragmented blocks larger than 0.5MB:
Addr Size Followed by
0x010fc8c0 3.1MB large free object followed by 0x0141c98c System.Int32
The large block allocation is displayed after the heap object information. This shows that there is a 3.1MB block followed by an integer. This type of allocation can cause severe memory fragmentation.
Another type of large object issue is an object with a large and complicated object graph. This requires the GC heap allocate space for all the small sub objects and places extreme pressure on the GC. It also causes performance issues when the GC runs to collect all the objects.
The most common objects of this type have been DataSets, SessionStateModules and the HttpSessionState. We can dump an objects state by using the !DumpObject command with the address of the object we want to inspect. To find a particular object in memory the following steps can be used. In this case the code created a DataSet with a single empty table.
!DumpHeap stat can be used to provide the objects Method Table (MT) address in memory.
total 135,526 objects
Statistics:
MT Count TotalSize Class Name
0x79bfcc10 1 12 System.Runtime.Remoting.Activation.ConstructionLevelActivator
0x79bfc98c 1 12 System.Runtime.Remoting.Activation.ActivationListener
0x79bb4cdc 4 80 System.Globalization.CompareInfo
0x03a1213c 2 80 System.Data.DataRelationCollection/DataTableRelationCollection
0x0371b984 1 80 System.Data.DataSet
0x037184ec 2 80 System.Web.UI.LosWriter
0x031599b0 14 224 System.Web.HttpContextWrapper
0x0371cae8 1 232 System.Data.DataTable
0x79bce3b4 12 240 System.Reflection.CustomAttribute/CAData
Another method of finding the method table is to use the !Name2EE command. This takes the assembly name and class name and provides runtime information. This is usually easier to use than searching through the heap object summaries.
0:000> !Name2EE System.Data.dll System.Data.DataSet
Searching modules...

Module: 0332ab58 (system.data.dll)
MethodTable: 0x0371b984
EEClass: 0x039f0f04
Name: System.Data.DataSet
The DataSets method table is at 0x0371b984. Using the !DumpHeap command with the mt switch and the method table address provides a list of the addresses of objects of that type in memory.
0:000> !DumpHeap -mt 0x0371b984
Address MT Size Gen
0x0159777c 0x0371b984 80 0 System.Data.DataSet
total 1 objects
Now the individual object can be examined using the !DumpObj command to dump the contents. Running this on the DataSet results in the following data.
0:000> !DumpObj 0x0159777c
Name: System.Data.DataSet
MethodTable 0x0371b984
EEClass 0x039f0f04
Size 80(0x50) bytes
GC Generation: 0
mdToken: 0x0200003b (c:\windows\assembly\gac\system.data\1.0.5000.0__b77a5c561934e089\system.data.dll)
FieldDesc*: 0x0371afb4
MT Field Offset Type Attr Value Name
0x0371ad3c 0x4000582 0x4 CLASS instance 0x00000000 site
0x0371ad3c 0x4000583 0x8 CLASS instance 0x00000000 events
0x0371ad3c 0x4000581 0 CLASS shared static EventDisposed
>> Domain:Value 0x00159ce0:NotInit 0x001d6808:0x01597830 <<
0x0371b984 0x40003d3 0xc CLASS instance 0x00000000 defaultViewManager
0x0371b984 0x40003d4 0x10 CLASS instance 0x01598000 tableCollection
0x0371b984 0x40003d5 0x14 CLASS instance 0x00000000 relationCollection
0x0371b984 0x40003d6 0x18 CLASS instance 0x00000000 extendedProperties
0x0371b984 0x40003d7 0x1c CLASS instance 0x015977dc dataSetName
0x0371b984 0x40003d8 0x20 CLASS instance 0x00f011f4 _datasetPrefix
0x0371b984 0x40003d9 0x24 CLASS instance 0x00f011f4 namespaceURI
0x0371b984 0x40003da 0x40 System.Boolean instance 0 caseSensitive
0x0371b984 0x40003db 0x28 CLASS instance 0x01597804 culture
0x0371b984 0x40003dc 0x41 System.Boolean instance 1 enforceConstraints
0x0371b984 0x40003dd 0x42 System.Boolean instance 0 fInReadXml
0x0371b984 0x40003de 0x43 System.Boolean instance 0 fInLoadDiffgram
0x0371b984 0x40003df 0x44 System.Boolean instance 0 fTopLevelTable
0x0371b984 0x40003e0 0x45 System.Boolean instance 0 fInitInProgress
0x0371b984 0x40003e1 0x46 System.Boolean instance 1 fEnableCascading
0x0371b984 0x40003e2 0x47 System.Boolean instance 0 fIsSchemaLoading
0x0371b984 0x40003e3 0x2c CLASS instance 0x00000000 rowDiffId
0x0371b984 0x40003e4 0x48 System.Boolean instance 0 fBoundToDocument
0x0371b984 0x40003e5 0x30 CLASS instance 0x00000000 onPropertyChangingDelegate
0x0371b984 0x40003e6 0x34 CLASS instance 0x00000000 onMergeFailed
0x0371b984 0x40003e7 0x38 CLASS instance 0x00000000 onDataRowCreated
0x0371b984 0x40003e8 0x3c CLASS instance 0x00000000 onClearFunctionCalled
0x0371b984 0x40003e9 0 CLASS shared static zeroTables
>> Domain:Value 0x00159ce0:NotInit 0x001d6808:0x015977cc <<
We see the size of the DataSet as being 80 bytes. This seams rather small and can be misleading. Looking at the object dump we can add up all the state fields and we should get 80 bytes. This does not include the child the child objects. In order to obtain the actual size of an entire object graph the !ObjSize command with the objects address is used. This command walks the entire object graph to obtain the objects size. Running this command on the address of the DataSet results in the following.
0:000> !ObjSize 0x0159777c
sizeof(0x159777c) = 2,272 ( 0x8e0) bytes (System.Data.DataSet)
We see the actual size of the DataSet is 2,272 bytes. If we were interested we could dump out the size of every sub object as well. As a side note, a DataTable should have nearly the same size as the containing DataSet. This is due to the DataTable maintaining a reference to the owning DataSet. The moral is if you are storing a DataTable in session state, make a copy of it to release the containing DataSet.
This approach can be used for retrieving any object of interest. Objects that have a large memory footprint and are used extensively can be problematic. Using DataSets for temporary data storage, placing them in session state and creating in memory file streams are some common culprits.
Summary
Using WinDbg and SOS are helpful in analyzing application issues. These are powerful tools at your disposal for performing a post mortem on your web application. They can be a little intimidating to start with but Ive found them to be real eye openers and have provided a lot of insight to the inner workings of .Net.

I've included for download an extract of the help files from the SOS.dll. This is a handy reference on the commands available. Have fun and start debugging.

Download the attachment that accompanies this article

Jon Wojtowicz is a C# MVP and a Systems Analyst at a large insurance company in Chattanooga, TN where he currently provides developer support and internal training. He has worked as a consultant working with Microsoft Technologies. This includes ASP, COM, VB6 and .Net, both C# and VB.Net since Beta 1. He has been an MCSD since 1999 and an MCT since 2000. Prior to getting a degree in Computer science he worked as a process engineer focusing on process automation, programmable controllers and equipment installations. In his spare time he likes woodworking and gardening.
Article Discussion: