| You have just put the finishing touches on
your latest application. Everything appears to be falling into place and then
you get the bad news. The users are complaining that after they use the
application for a short period of time it resets itself. Worse than that the
users are complaining of out of memory exceptions and IIS needs to be reset to
resolve the issue. These are all symptoms of ASP.Net memory issues, whether it
be the loss of session state due to the aspnet_wp (w3wp) recycling or getting
out of memory exceptions.
|
| This is a brief overview of some of the most
common issues that I have observed when analyzing ASP.Net application out of
memory exceptions. |
| Background |
| The The ASP.Net worker process (aspnet_wp or
w3wp) process hosts ASP.Net applications. This single process has up to 2GB of
address space and is designed to recycle when the memory consumption gets to be
excessive. By default the process will recycle when the memory consumption
reaches 60% of the limit. This means that the worker process will recycle when
the memory consumption gets to 1.2GB. My observations have indicated that the
process will start develop issues when the memory consumption exceeds 600MB of
memory. I have also found that there is no apparent pattern to when the process
will have issues |
| Typically the process will recycle itself
without any issues. This will only cause a loss of in memory session state.
When under heavy load the ASP.Net applications will begin to throw out of
memory exceptions rather than the process recycling. Another manifestation has
been multiple instance of the ASP.Net worker process being in memory
simultaneously with only a single instance active. |
| You might be asking yourself why you do not
see this issue when you are performing your stress test. Most stress test
environments only test a single application and do not perform load tests on
the configuration running in production. This leads to an application that
performs well in stress but cannot coexist with other applications due to
memory consumption. |
| Obviously creating many large objects in your
application can cause memory issues. If you review your code and see that you
are declaring arrays of items that have 1 million members or more than you
might want to reconsider how your applications works. This is seldom the case
as the issues tend to be a little more subtle than this situation. |
|
| Tools
|
| Any good investigator needs a set of tools to
examine the evidence. Here are my tools. |
-
-
SOS CLR debugger extensions. This is used in conjunction with WinDbg to look
inside of the managed memory. The latest version of SOS is included with the
WinDbg download. There have been many good articles written of the basics of
using SOS including this MSDN article,
http://msdn.microsoft.com/msdnmag/issues/03/06/Bugslayer/
-
ADPlus which is included in the Microsoft debugging tools. This script allows
for capturing memory dumps when a process hangs or crashes. For capturing crash
dumps the following command line is used:
ADPlus.vbs crash pn aspnet_wp.exe
This attaches the CDB debugger to the process with the specified name. If you
are running on Window 2003 where there may multiple instance of the w3_wp.exe
processes, you can specify the process id (PID) by using the p switch followed
by the process id. If you wish to create an immediate dump you can open Task
Manager and end the process. This will simulate a process crash.
|
| Setting Up WinDbg
|
|
| Obtaining the Memory Dump
|
| Since this is related to problems on
production servers the first step is to obtain the memory dump. The reason for
the dump is simple, I'm not aware of any shop that allows interactive debugging
on production servers. The dump is also obtained so that the memory can be
analyzed without undue pressure, other than that of trying to resolve the
issue. |
| The dump needs to be taken when the memory
consumption is excessive. What is excessive memory consumption? Typically when
the process memory grows beyond 600MB it should be considered high. Ideally the
dump should be taken when the out of memory exceptions are thrown as this
represents the best picture of the state of the process. |
| Once the memory dump is obtained it can be
loaded into WinDbg. This is accomplished through the File -> Open Crash
Dump menu item. After loading the crash dump we need to be able to look into
the managed heap. For this the SOS (Son of Strike) CLR debugger extensions can
be loaded using the following command: |
| .load clr10\sos.dll |
| For this demonstration I've created two simple
web applications, badapp, which demonstrates the bad things that can happen and
goodapp which shows the differences when an application is following the best
practices. Each application contains 3 aspx pages. The dump for these two
simple applications ranges from about 60-100MB. An actual production memory
dump may easily exceed 1GB and can take a long time to analyze. |
| Common Reason for High Memory Consumption
|
| This will be a description of some common
issues of out of memory exceptions and how they appear within the managed heap. |
| Application Set Up For Debugging
|
| Debugging is necessary for development of your
application. By default when you create your web application in .Net you will
see the following in the web.config: |
<compilation
debug="true" />
|
| This setting has several effects on the
dynamic compilation of the web pages. It injects debugging code into the
created assemblies. It also disables batch compilation. What this means is that
every page will be compiled into its own assembly. This causes memory issues
since every assembly is loaded on demand when a page is accessed. This leads to
unusable gaps in the memory and severe fragmentation. The somewhat random
loading and resulting fragmentation can make it difficult for the system to
allocate more memory when needed. |
| Viewing information about the assemblies
loaded in an app domain can be accomplished by using the following command in
WinDbg. The stat switch provides us statistics about each app domain. |
| !DumpDomain stat |
| The result for this simple demonstration
should look similar to the following. An actual production dump may have 30 or
more app domains. |
| Domain |
Num Assemblies |
Size Assemblies |
Name |
| 0x793e9d58 |
1 |
2,138,112 |
System Domain |
| 0x793eb278 |
10 |
9,479,168 |
Shared Domain |
| 0x00159ab8 |
2 |
2,482,176 |
DefaultDomain |
| 0x0018bb68 |
14 |
7,380,992 |
/LM/w3svc/1/root/BadApp-1-127813414066093750 |
| 0x032326b0 |
10 |
7,312,384 |
/LM/w3svc/1/root/GoodApp-2-127813414228125000 |
|
| The first thing that should be noted is the
fact that badapp has 14 assemblies loaded while goodapp has only 10. Again the
code for these applications is very similar and simple so we should not expect
any differences from the dynamic compilation. We can then pick a particular
domain and dump out the assembly information by using DumpDomain with the
domain address. |
| !DumpDomain 0018bb68 |
| To look at the assemblies loaded in an app
domain we use the use the following command in WinDbg: |
Output for the baddapp domain (abbreviated):
Domain: 0x18bb68
LowFrequencyHeap: 0x0018bbcc
HighFrequencyHeap: 0x0018bc24
StubHeap: 0x0018bc7c
Name: /LM/w3svc/1/root/BadApp-1-127813414066093750
Assembly: 0x0018a248 [System.Web, Version=1.0.5000.0, Culture=neutral,
PublicKeyToken=b03f5f7f11d50a3a]
ClassLoader: 0x0017e5e8
Module Name
0x001c9b80
c:\windows\assembly\gac\system.web\1.0.5000.0__b03f5f7f11d50a3a\system.web.dll
Assembly: 0x001a9088 [System, Version=1.0.5000.0, Culture=neutral,
PublicKeyToken=b77a5c561934e089]
ClassLoader: 0x0018abf8
Module Name
0x0018c8c8
c:\windows\assembly\gac\system\1.0.5000.0__b77a5c561934e089\system.dll
Assembly: 0x0022b990 [qa-y3re_]
ClassLoader: 0x001e50a8
Module Name
0x0014c520 c:\windows\microsoft.net\framework\v1.1.4322\temporary asp.net
files\badapp\141137f0\f42dfee7\qa-y3re_.dll
Assembly: 0x0022ded0 [BadApp]
ClassLoader: 0x00234468
Module Name
0x03230c98 c:\windows\microsoft.net\framework\v1.1.4322\temporary asp.net
files\badapp\141137f0\f42dfee7\assembly\dl2\fbba12dd\2d93ab03_fd04c601\badapp.dll
Assembly: 0x03257b88 [b8624ntz]
ClassLoader: 0x032453c8
Module Name
0x001ddb60 c:\windows\microsoft.net\framework\v1.1.4322\temporary asp.net
files\badapp\141137f0\f42dfee7\b8624ntz.dll
Assembly: 0x0327b0e8 [1lixxuqx]
ClassLoader: 0x03277d60
Module Name
0x001de148 c:\windows\microsoft.net\framework\v1.1.4322\temporary asp.net
files\badapp\141137f0\f42dfee7\1lixxuqx.dll
Assembly: 0x0327e498 [gvdfsokf]
ClassLoader: 0x03277dd8
Module Name
0x03274d48 c:\windows\microsoft.net\framework\v1.1.4322\temporary asp.net
files\badapp\141137f0\f42dfee7\gvdfsokf.dll
|
| For the sake of brevity the framework
assemblies have been removed from the listing. What is displayed are the
dynamic assemblies created from the dynamic compilation. They can be easily
identified by the odd looking assembly names. These assemblies represent the
three pages in the application along with the global.asax page. As a comparison
the assemblies for goodApp can be dumped. |
Output for the goodapp domain (abbreviated):
Domain: 0x32326b0
LowFrequencyHeap: 0x03232714
HighFrequencyHeap: 0x0323276c
StubHeap: 0x032327c4
Name: /LM/w3svc/1/root/GoodApp-2-127813414228125000
Assembly: 0x03281ff0 [GoodApp]
ClassLoader: 0x03270908
Module Name
0x032c88b0 c:\windows\microsoft.net\framework\v1.1.4322\temporary asp.net
files\goodapp\822c42b8\360972c7\assembly\dl2\ce1e7872\33328a03_fd04c601\goodapp.dll
Assembly: 0x032d2cc8 [5xhp4krf]
ClassLoader: 0x03270980
Module Name
0x032d2d90 c:\windows\microsoft.net\framework\v1.1.4322\temporary asp.net
files\goodapp\822c42b8\360972c7\5xhp4krf.dll
|
| The goodapp domain only has one dynamic
assembly. We have saved the additional overhead and space of having multiple
small assemblies loaded. This was accomplished by changing the configuration to
not compile in debug mode. |
<compilation
debug="false" />
|
| The global.asax was also removed to save an
additional assembly. If the events and methods of the Global class are not used
by your application, remove it. The application will run fine without it. Also
removing any unused event handlers and methods can result in some savings. One
peculiarity is that the global.asax always compiles into its own assembly
regardless of the debug settings. |
| Only putting release builds and setting the
compilation to not build in debug mode are very important to the application.
This can not only affect the memory issue but has a drastic effect on
performance. If nothing else, verify that everything is set to release on the
production server. |
| Installing .Net 1.1 SP1 will also help with
the fragmentation caused by many assemblies loading. One of the fixes in this
service pack changes the way in which memory is allocated. This can help when
experiencing memory issues. |
| Other items that create dynamic assemblies are
regular expressions, serialization and XML schemas. While I advocate using
these items, they are mentioned as possible suspects when trying to examine the
source of dynamic assemblies. |
| String Concatenation
|
| By using the + operator (& in VB.Net) to
concatenate string can cause many intermediate strings to be created and
released. To demonstrate this badapp uses string concatenation in the following
code: |
string s = "test";
string z = string.Empty;
int count = int.Parse(TextBox1.Text);
for(int i=0; i < count; ++i)
{
z += s;
}
|
| Goodapp uses a StringBuilder to perform the
same. Running this for a count of 1000 we can see the effects. The badapp
consumes a considerable amount of cpu and appears to thrash the memory as it
tries to allocate the memory for the string concatenation. The goodapp using
the StringBuilder shows no signs of this. At 100000 cycles running badapp
appears to hang the system with the allocations and garbage collection. The
goodapp with the StringBuilder appears to run nearly as fast as running 1000
cycles. A dump of the managed heap can be obtained to see the number of objects
in memory. Using the stat switch provides a summary of the objects found on
the heap. Forgetting to use the switch will dump each object on a separate
line. There will be one heap per processor on a server. |
!DumpHeap stat
Results:
total 47,485 objects
Statistics:
|
MT |
Count |
TotalSize |
Class Name |
| 0x79bf6b0c |
1
|
12 |
System.IO.TextReader/NullTextReader |
| 0x79bf48ac |
1
|
12 |
System.Runtime.Remoting.Messaging.IllogicalCallContext |
| 0x79becb5c |
1
|
12 |
System.Runtime.Remoting.Messaging.CallContextSecurityData |
| 0x79be4478 |
1
|
12 |
System.Runtime.Remoting.Proxies.ProxyAttribute |
| 0x79bce950 |
1
|
12 |
System.Resources.FastResourceComparer |
| 0x79bbe6b8
|
1
|
12 |
System.Empty |
|
| 0x00be209c
|
6,921
|
206,376 |
System.Object[] |
| 0x00be292c
|
69
|
329,332 |
System.Byte[] |
| 0x0015ebe0
|
30
|
2,079,656 |
Free |
| 0x79b925c8
|
16,592
|
2,123,684 |
System.String |
|
| It is not unusual for a heavily utilized
server to have over 1 million strings in memory. The only true way to find
sting concatenation is by a code review. Analyzing the memory dump can help in
determining if concatenation is a possible issue. It is important to remember
that every string literal on an aspx page gets converted to a Literal control
with the Text property assigned to the static string. This can create a
tremendous amount of strings on a large site even without concatenation. |
| Large Object Allocation
|
| Single allocation objects larger than 85kB are
placed in what is known as the large object heap. The large object heap is
never deallocated nor is it compacted like the normal heap. Allocation of large
objects can cause this memory to fragment and grow.
|
| Running the badapp with an allocation of a
1000000 byte array can demonstrate what happens. Running !DumpHeap after the
allocation results in the following: |
| 0x00be292c |
94 |
1,240,680 |
System.Byte[] |
| 0x0015ebe0 |
14 |
3,318,488 |
Free |
Total 42,882 objects, Total size: 6,428,692
Fragmented blocks larger than 0.5MB:
|
Addr |
Size |
Followed by |
| 0x010fc8c0 |
3.1MB |
large free object followed by 0x0141c98c System.Int32 |
|
| The large block allocation is displayed after
the heap object information. This shows that there is a 3.1MB block followed by
an integer. This type of allocation can cause severe memory fragmentation. |
| Another type of large object issue is an
object with a large and complicated object graph. This requires the GC heap
allocate space for all the small sub objects and places extreme pressure on the
GC. It also causes performance issues when the GC runs to collect all the
objects. |
| The most common objects of this type have been
DataSets, SessionStateModules and the HttpSessionState. We can dump an objects
state by using the !DumpObject command with the address of the object we want
to inspect. To find a particular object in memory the following steps can be
used. In this case the code created a DataSet with a single empty table. |
| !DumpHeap stat can be used to provide the
objects Method Table (MT) address in memory. |
total 135,526 objects
Statistics:
|
MT |
Count |
TotalSize |
Class Name |
| 0x79bfcc10 |
1
|
12 |
System.Runtime.Remoting.Activation.ConstructionLevelActivator |
| 0x79bfc98c
|
1
|
12 |
System.Runtime.Remoting.Activation.ActivationListener |
|
| 0x79bb4cdc
|
4
|
80 |
System.Globalization.CompareInfo |
| 0x03a1213c
|
2
|
80 |
System.Data.DataRelationCollection/DataTableRelationCollection |
| 0x0371b984
|
1
|
80 |
System.Data.DataSet |
| 0x037184ec
|
2
|
80 |
System.Web.UI.LosWriter |
|
| 0x031599b0
|
14
|
224 |
System.Web.HttpContextWrapper |
| 0x0371cae8
|
1
|
232 |
System.Data.DataTable |
| 0x79bce3b4
|
12
|
240 |
System.Reflection.CustomAttribute/CAData |
|
| Another method of finding the method table is
to use the !Name2EE command. This takes the assembly name and class name and
provides runtime information. This is usually easier to use than searching
through the heap object summaries. |
0:000> !Name2EE System.Data.dll
System.Data.DataSet
Searching modules...
Module: 0332ab58 (system.data.dll)
MethodTable: 0x0371b984
EEClass: 0x039f0f04
Name: System.Data.DataSet
|
| The DataSets method table is at 0x0371b984.
Using the !DumpHeap command with the mt switch and the method table address
provides a list of the addresses of objects of that type in memory. |
0:000> !DumpHeap -mt 0x0371b984
|
Address
|
MT |
Size |
Gen |
|
|
0x0159777c |
0x0371b984 |
80
|
0
|
System.Data.DataSet
|
total 1 objects
|
| Now the individual object can be examined
using the !DumpObj command to dump the contents. Running this on the DataSet
results in the following data. |
0:000> !DumpObj 0x0159777c
Name: System.Data.DataSet
MethodTable 0x0371b984
EEClass 0x039f0f04
Size 80(0x50) bytes
GC Generation: 0
mdToken: 0x0200003b
(c:\windows\assembly\gac\system.data\1.0.5000.0__b77a5c561934e089\system.data.dll)
FieldDesc*: 0x0371afb4
|
MT |
Field |
Offset |
Type |
Attr |
Value Name |
| 0x0371ad3c |
0x4000582 |
0x4 |
CLASS |
instance |
0x00000000 |
site |
| 0x0371ad3c |
0x4000583 |
0x8 |
CLASS |
instance |
0x00000000 |
events |
| 0x0371ad3c |
0x4000581 |
0 |
CLASS |
shared |
static |
EventDisposed |
|
>> Domain:Value 0x00159ce0:NotInit 0x001d6808:0x01597830 << |
| 0x0371b984 |
0x40003d3 |
0xc
|
CLASS |
instance |
0x00000000 |
defaultViewManager |
| 0x0371b984 |
0x40003d4 |
0x10 |
CLASS |
instance |
0x01598000 |
tableCollection |
| 0x0371b984 |
0x40003d5 |
0x14 |
CLASS |
instance |
0x00000000 |
relationCollection |
| 0x0371b984 |
0x40003d6 |
0x18 |
CLASS |
instance |
0x00000000 |
extendedProperties |
| 0x0371b984 |
0x40003d7 |
0x1c |
CLASS |
instance |
0x015977dc |
dataSetName |
| 0x0371b984 |
0x40003d8 |
0x20 |
CLASS |
instance |
0x00f011f4 |
_datasetPrefix |
| 0x0371b984 |
0x40003d9 |
0x24 |
CLASS |
instance |
0x00f011f4 |
namespaceURI |
| 0x0371b984 |
0x40003da |
0x40 |
System.Boolean |
instance |
0 |
caseSensitive |
| 0x0371b984 |
0x40003db |
0x28 |
CLASS |
instance |
0x01597804 |
culture |
| 0x0371b984 |
0x40003dc |
0x41 |
System.Boolean |
instance |
1 |
enforceConstraints |
| 0x0371b984 |
0x40003dd |
0x42 |
System.Boolean |
instance |
0 |
fInReadXml |
| 0x0371b984 |
0x40003de |
0x43 |
System.Boolean |
instance |
0 |
fInLoadDiffgram |
| 0x0371b984 |
0x40003df |
0x44 |
System.Boolean |
instance |
0 |
fTopLevelTable |
| 0x0371b984 |
0x40003e0 |
0x45 |
System.Boolean |
instance |
0 |
fInitInProgress |
| 0x0371b984 |
0x40003e1 |
0x46 |
System.Boolean |
instance |
1 |
fEnableCascading |
| 0x0371b984 |
0x40003e2 |
0x47 |
System.Boolean |
instance |
0 |
fIsSchemaLoading |
| 0x0371b984 |
0x40003e3 |
0x2c |
CLASS |
instance |
0x00000000 |
rowDiffId |
| 0x0371b984 |
0x40003e4 |
0x48 |
System.Boolean |
instance |
0 |
fBoundToDocument |
| 0x0371b984 |
0x40003e5 |
0x30 |
CLASS |
instance |
0x00000000 |
onPropertyChangingDelegate |
| 0x0371b984 |
0x40003e6 |
0x34 |
CLASS |
instance |
0x00000000 |
onMergeFailed |
| 0x0371b984 |
0x40003e7 |
0x38 |
CLASS |
instance |
0x00000000 |
onDataRowCreated |
| 0x0371b984 |
0x40003e8 |
0x3c |
CLASS |
instance |
0x00000000 |
onClearFunctionCalled |
| 0x0371b984 |
0x40003e9 |
0 |
CLASS |
shared |
static |
zeroTables |
>> Domain:Value 0x00159ce0:NotInit 0x001d6808:0x015977cc <<
|
| We see the size of the DataSet as being 80
bytes. This seams rather small and can be misleading. Looking at the object
dump we can add up all the state fields and we should get 80 bytes. This does
not include the child the child objects. In order to obtain the actual size of
an entire object graph the !ObjSize command with the objects address is used.
This command walks the entire object graph to obtain the objects size. Running
this command on the address of the DataSet results in the following. |
0:000> !ObjSize 0x0159777c
sizeof(0x159777c) = 2,272 ( 0x8e0) bytes (System.Data.DataSet)
|
| We see the actual size of the DataSet is 2,272
bytes. If we were interested we could dump out the size of every sub object as
well. As a side note, a DataTable should have nearly the same size as the
containing DataSet. This is due to the DataTable maintaining a reference to the
owning DataSet. The moral is if you are storing a DataTable in session state,
make a copy of it to release the containing DataSet. |
| This approach can be used for retrieving any
object of interest. Objects that have a large memory footprint and are used
extensively can be problematic. Using DataSets for temporary data storage,
placing them in session state and creating in memory file streams are some
common culprits. |
| Summary |
| Using WinDbg and SOS are helpful in analyzing
application issues. These are powerful tools at your disposal for performing a
post mortem on your web application. They can be a little intimidating to start
with but Ive found them to be real eye openers and have provided a lot of
insight to the inner workings of .Net. |
I've included for download an extract of the
help files from the SOS.dll. This is a handy reference on the commands
available. Have fun and start debugging.
Download the attachment that accompanies this article |