Value types memory summary

[Eamon]

Hi, is there any way to see the total memory consumed by value types?

i.e. I'd like to see that my snapshot had x decimal values consuming x*16 bytes

[Andreas Suurkuusk]

No, there's no functionality to present the number of bytes used by value types within other classes. The profiler will only present information about boxed value type instances.

It would be fairly easy to retrieve this information from a snapshot, but I don't know how it would be presented in a good way, and I'm not sure how valuable this information is.

If this information is important to you, I can help write a small helper console application that retrieves this information using the external profiler API. Is this something you would be interested in?

[Eamon]

Hi Andreas, yes this would be helpful to me.

FYI I'll outline the reasons I am after this information.

I have a fairly large application where the memory foot print is often ~2GB. A significant proportion of this memory is used by types containing many BCL value types that are significantly more precise than required. e.g. DateTimeOffset and decimal.

For the vast majority of decimal values, I could substitute a custom value type that was 4 bytes instead of 16 that would hold the full range of required values, reducing the memory requirement to 25% of the original. Similarly for DateTimeOffset.

I am trying to understand the impact of the overhead from the use of these types where a smaller custom value type would suffice. I know that I can calculate it manually per class, but I have many classes overall and in this case I know that the vast majority of DateTimeOffset and decimal values will be candidates for smaller types.

All I really need at this stage is a total count of the instances and bytes consumed for all value types (or at least in this case DateTimeOffset and decimal).

regards,
Eamon

[Andreas Suurkuusk]

OK, now I understand this requirement better. I will write an example program that extracts this information from a session file. I will post it as a reply to this topic, but it may be a few days before I get the time to write the program.

[Andreas Suurkuusk]

Now I have written an example that extracts value type usage information from a session. It was a little bit more complicated than I expected. You will find the sample code below. To compile the code, you need to add a reference to the "SciTech.NetMemProfiler.Core" assembly, which you will find in the directory "C:\Program Files\SciTech\NetMemProfiler5\Assemblies".

I hope you find this sample useful and if you have any questions, you can reply to this post.

Code: Select all

using SciTech.Metadata;
using SciTech.Metadata.Managed;
using SciTech.Profiler;
using SciTech.Profiler.Memory;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Reflection;

internal class Program
{
    private static void Main(string[] args)
    {
        ProfilerApplicationCore application = new ProfilerApplicationCore();

        // Load the session..
        using (ProfilerSession session = application.LoadSession(@"<path to session>"))
        {
            // Retrieve the available snapshots.
            SnapshotHeader[] snapshots = session.SessionFile.GetSnapshotHeaders();
            var process = session.GetProfiledProcesses().FirstOrDefault();

            if (snapshots.Length > 0 && process != null)
            {
                // Name of the types for which field memory information is wanted.
                string[] fieldTypeNames = {
                    "System.Decimal,mscorlib",
                    "System.TimeSpan,mscorlib",
                    "System.DateTime,mscorlib" };

                using (ProfilerComparison comparison = application.CreateComparison())
                {
                    // Select snapshots. Currently only the last one is selected, and no comparison snapshot is used.
                    ProcessSnapshotHeader selectedSnapshot = snapshots[snapshots.Length - 1].Processes.FirstOrDefault(s => s.ProcessId == process.Id);
                    ProcessSnapshotHeader comparisonSnapshot = ProcessSnapshotHeader.Empty;// snapshots[snapshots.Length - 2].Processes.FirstOrDefault(s => s.ProcessId == process.Id);

                    // A comparison needs to be made to allow instance information to 
                    // be extracted from the session.
                    comparison.CompareSnapshots(selectedSnapshot, comparisonSnapshot);

                    // Create the extractor that is used to extract field information
                    var extractor = new FieldUsageExtractor(fieldTypeNames, comparison);

                    // Extract and write the field information.
                    var fieldsInformation = extractor.GetValueTypeFieldInformation();
                    FieldUsageExtractor.WriteFieldInformation(fieldsInformation);
                }
            }

        }

    }

    /// <summary>
    /// Helper class that extracts information about fields of selected types.
    /// </summary>
    public class FieldUsageExtractor
    {
        private readonly ProfilerComparison comparison;

        private readonly ProfiledProcess process;

        /// <summary>
        /// Cache used when extracting information about type fields.
        /// </summary>
        private readonly Dictionary<RuntimeType, List<TypeFieldInformation>> typeFieldsCache = new Dictionary<RuntimeType, List<TypeFieldInformation>>();

        private bool fieldInformationExtracted;

        /// <summary>
        /// Dictionary for holding the results of the field information extraction. The keys
        /// in the dictionary are the field types for which usage information is wanted.
        /// </summary>
        private Dictionary<ManagedType, List<TypeFieldInformation>> valueTypes = new Dictionary<ManagedType, List<TypeFieldInformation>>();

        public FieldUsageExtractor(IReadOnlyList<string> valueTypeNames, ProfilerComparison comparison)
        {
            this.comparison = comparison;
            this.process = comparison.PrimaryProcess;

            foreach (var valueTypeName in valueTypeNames)
            {
                ManagedType valueType = this.process.GetTypeNameCache().GetMatchingType(valueTypeName, false);
                if (valueType != null)
                {
                    this.valueTypes.Add(valueType, new List<TypeFieldInformation>());
                }
            }
        }

        public static void WriteFieldInformation(Dictionary<ManagedType, List<TypeFieldInformation>> fieldsDictionary)
        {
            foreach (var pair in fieldsDictionary)
            {
                var fieldType = pair.Key;
                var fieldsList = pair.Value;
                int totalCount = fieldsList.Sum(f => f.Count);

                Console.WriteLine($"{GetShortTypeName(fieldType)} (total instances: {totalCount})");
                foreach (var field in fieldsList)
                {
                    string fieldName = !string.IsNullOrEmpty(field.Name) ? field.Name : " (instance)";
                    Console.WriteLine($"  {GetShortTypeName(field.DeclaringType.ManagedType)}{fieldName}: {field.Count}");
                }
            }
        }

        /// <summary>
        /// Extracts field usage information based on the types provided to the constructor.
        /// </summary>
        /// <returns></returns>
        public Dictionary<ManagedType, List<TypeFieldInformation>> GetValueTypeFieldInformation()
        {
            if (!this.fieldInformationExtracted)
            {
                foreach (var instanceId in this.comparison.EnumInstances(null, true))
                {
                    TypeInstance instance = this.comparison.GetTypeInstance(instanceId);
                    var rtType = instance.RuntimeType;

                    if (rtType.IsArray)
                    {
                        var valueTypeFields = this.ExtractValueTypeFields(rtType);

                        int nComponents = instance.GetArrayUsage()?.ComponentsCount ?? 0;
                        foreach (var valueTypeField in valueTypeFields)
                        {
                            valueTypeField.Count += nComponents;
                        }
                    }
                    else
                    {
                        var valueTypeFields = this.ExtractValueTypeFields(rtType);

                        foreach (var valueTypeField in valueTypeFields)
                        {
                            valueTypeField.Count++;
                        }
                    }
                }

                this.fieldInformationExtracted = true;
            }

            return this.valueTypes;
        }

        private static string GetShortTypeName(ManagedType managedType)
        {
            return managedType.ToString(SyntaxLanguage.CSharp,
                TypeNameOptions.IncludeNamespaceAndName
                | TypeNameOptions.UseBuiltInTypeNames
                | TypeNameOptions.UseBuiltInInternalTypeNames
                | TypeNameOptions.IncludeGenericArguments);
        }

        /// <summary>
        /// Helper method for ExtractValueTypeFields. Appends relevant fields to the provided fieldsList
        /// </summary>
        /// <param name="rtType"></param>
        /// <param name="outerType"></param>
        /// <param name="baseFieldName"></param>
        /// <param name="fieldsList"></param>
        private void AppendValueTypeFields(RuntimeType rtType, RuntimeType outerType, string baseFieldName, List<TypeFieldInformation> fieldsList)
        {
            foreach (var field in rtType.EnumFields(BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic))
            {
                var fieldType = field.FieldType;
                if (fieldType != null)
                {
                    var managedFieldType = fieldType.ManagedType;
                    string fullFieldName;

                    if (!string.IsNullOrEmpty(baseFieldName))
                    {
                        fullFieldName = $"{baseFieldName}.{field.Name}";
                    }
                    else
                    {
                        fullFieldName = $".{field.Name}";
                    }

                    if (this.valueTypes.TryGetValue(managedFieldType, out var wantedTypeList))
                    {
                        var valueTypeField = new TypeFieldInformation(managedFieldType, outerType, fullFieldName);
                        wantedTypeList.Add(valueTypeField);
                        fieldsList.Add(valueTypeField);
                    }
                    else if (fieldType.IsValueType && !fieldType.IsPrimitive)
                    {
                        this.AppendValueTypeFields(fieldType, outerType, fullFieldName, fieldsList);
                    }
                }
            }
        }

        /// <summary>
        /// Extracts fields information for the specified type. Caches the result in <see cref="typeFieldsCache"/>.
        /// </summary>
        /// <param name="rtType"></param>
        /// <returns></returns>
        private IReadOnlyList<TypeFieldInformation> ExtractValueTypeFields(RuntimeType rtType)
        {
            if (!this.typeFieldsCache.TryGetValue(rtType, out var fieldsList))
            {
                fieldsList = new List<TypeFieldInformation>();

                if (rtType.IsArray)
                {
                    // Array types are handles separately. If they 
                    // are arrays of value types, they may contain fields that we're interested in.
                    if (rtType.ElementType != null && rtType.ElementType.IsValueType)
                    {
                        var elementTypeFields = this.ExtractValueTypeFields(rtType.ElementType);
                        foreach (var f in elementTypeFields)
                        {
                            var arrayFieldInformation = new TypeFieldInformation(f.FieldType, rtType, f.Name);
                            fieldsList.Add(arrayFieldInformation);

                            var arrayTypeList = this.valueTypes[arrayFieldInformation.FieldType];
                            arrayTypeList.Add(arrayFieldInformation);
                        }
                    }
                }
                else if (this.valueTypes.TryGetValue(rtType.ManagedType, out var directTypeList))
                {
                    var directTypeField = new TypeFieldInformation(rtType.ManagedType, rtType, "");
                    fieldsList.Add(directTypeField);
                    directTypeList.Add(directTypeField);
                }
                else
                {
                    this.AppendValueTypeFields(rtType, rtType, "", fieldsList);
                }

                this.typeFieldsCache.Add(rtType, fieldsList);
            }

            return fieldsList;

        }
    }

    public class TypeFieldInformation
    {
        public TypeFieldInformation(ManagedType fieldType, RuntimeType declaringType, string fullFieldName)
        {
            this.FieldType = fieldType;
            this.DeclaringType = declaringType;
            this.Name = fullFieldName;
        }

        public int Count { get; set; }

        public RuntimeType DeclaringType { get; }

        public ManagedType FieldType { get; }

        public string Name { get; }
    }
}

[Eamon]

Thanks Andreas, it is very much appreciated. I will try to find time later today to test it.

regards,
Eamon