Collecting a snapshot frees otherwise held memory

Use this forum for questions on how to use .NET Memory Profiler and how to analyse memory usage.
Post Reply
PGF

Collecting a snapshot frees otherwise held memory

Post by PGF » Tue Aug 23, 2005 2:55 am

I'm trialling MemProfiler hoping to find a major memory allocation problem. It looks like a wonderful tool but so far I haven't been able to use it to solve my problem - because the problem disappears under it.

Some pointers would be useful !

The problem occurs in a process that collects data from a DBMS. There are hundreds of queries being issued and the process runs for about 1 minute. There are no errors and all of the data is returned as expected. For testing the results are being discarded. None of the result sets are large.

During the routine, memory in use climbs from about 740Mb to 1440Mb so it is holding 700Mb. This is not expected as all of the objects and connections are I think properly disposed of after use. Worse still the memory is not freed at the end of the routine.

Sounds like a perfect application for MemProfiler.

I start the application under MemProfiler. When it is ready to go I take Snapshot 1 and everthing is fine. I cause the routine in question to execute. Memory in use climbs as before.

Then I take Snapshot 2. Immediately the extra memory is freed so comparing Snapshot 1 and 2 is pointless.

I understand that MemProfiler causes a GC before taking the snapshot. The weird thing is that doing System.GC.Collect() at intervals during the routine and even in an outer routine that calls it does not cause the memory to free up.

Even weirder, if I run the routine again in the same session, the memory build-up does not occur. It does everything that it should in exactly the same way with no memory utilisation peak like before.

This is perfectly repeatable.

Another observation is that when run outside of MemProfiler the application will hold the extra memory for quite a long time but then may release it with no apparent cause. Or the memory may be released when some other application is minimised. Putting in logging and stepping through the routine may also cause the problem to not occur or at least not manifest because possibly the memory is being reclaimed as it proceeds.

I don't think this is a traditional memory leak because GC can return the meory if it wants to. So what could be stopping it from doing so automatically and even when it is explicitly called in the routine in question ?

What should I look at next ? Can MemProfiler help ?

Andreas Suurkuusk
Posts: 1029
Joined: Wed Mar 02, 2005 7:53 pm

Post by Andreas Suurkuusk » Tue Aug 23, 2005 9:20 pm

The behaviour you're describing seems a bit odd. Calling GC.Collect should work similar to collecting a heap snapshot, but when collecting a heap snapshot, more than one GC is actually performed.

Below are some questions that might help you get closer to the solution of your problem:

Did you try to use the following sequence?

Code: Select all

GC.Collect(); 
GC.WaitForPendingFinalizers(); 
GC.Collect(); 
This sequence should clean up as much memory as possible.

How much managed memory usage are you seeing? 700 MB of data seems very much, and I doubt that the profiler works very well if you have 700 MB of managed instances.

Did you use the dispose tracker to make sure thay you dispose your instances correctly? If you have instances that wrap unmanaged resources, then it is very important that you dispose them correctly.

Are you using large instances? Large instances use a separate heap which does not get compacted. If you have a lot of large instances then the memory may become very fragmented, and the runtime may fail to release the memory.

Have you looked at the real-time graph? Is the total instances count much higher than the live bytes count?
Best regards,

Andreas Suurkuusk
SciTech Software AB

PGF

Post by PGF » Thu Aug 25, 2005 11:46 pm

Adding GC.WaitForPendingFinalizers() to my forced garbage collection leads to the memory being released. So that at least is a work-around. Now it exhibits a series of little peaks rather than the previous mountain. I can determine how big the peaks are by how frequently I force GC.

Thank you for that suggestion.

As to the real cause I'll keep looking and follow-up on your other ideas.

By the way, can you get MemProfiler to collect a snapshot without perfroming GC ? That would seem to be the quickest way to find out what sort of objects are building up in such cases.

Thank you again.

PGF

Andreas Suurkuusk
Posts: 1029
Joined: Wed Mar 02, 2005 7:53 pm

Post by Andreas Suurkuusk » Sun Aug 28, 2005 6:34 pm

It is possible to collect a heap snapshot by only performing a gen #0 GC.
You do this by enabling "Heap utilization tracking" in the Settings form. When "Heap utilization tracking" is enabled, you will have access to a new command: "Collect Gen #0 Heap Snapshot".

After collecting a heap snapshot using this command, you will also be able to see unreachable instances, which may be useful when you try to analyse your problem.
Best regards,

Andreas Suurkuusk
SciTech Software AB

Post Reply

Who is online

Users browsing this forum: No registered users and 24 guests