Memory Analysis

realiser · December 29, 2011, 8:49am

My handheld RFID data terminal project is getting rather big - up to the point that I am now running out of ROM space. I first noticed this while trying to deploy… It just gives a “hardware error” and reports the bytes used. This is pretty annoying because it doesn’t tell me how much I have available and where the problem is… You can only see memory utilisation when deployment is successful.

I crossed the line when I implemented SystemUpdate - which took 48k - which means my app needs to be 100kb or smaller to fit on a Panda II, which it no longer does.

After cutting a lot to get it to deploy, it now looks like this:

Total: (13196 RAM - 123444 ROM - 58116 METADATA)

AssemblyRef = 172 bytes ( 43 elements)

TypeRef = 1192 bytes ( 298 elements)

FieldRef = 80 bytes ( 20 elements)

MethodRef = 1792 bytes ( 448 elements)

TypeDef = 2904 bytes ( 363 elements)

FieldDef = 1184 bytes ( 585 elements)

MethodDef = 3832 bytes ( 1911 elements)

DebuggingInfo = 1940 bytes

Attributes = 48 bytes ( 6 elements)

TypeSpec = 32 bytes ( 8 elements)

Resources Files = 96 bytes ( 4 elements)

Resources = 312 bytes ( 39 elements)

Resources Data = 1184 bytes

Strings = 17494 bytes

Signatures = 7424 bytes

ByteCode = 44305 bytes

In my quest to shave off unwanted bytes, I developed an Excel spreadsheet and macro to parse the output from the Output window in Visual Studio and graph what’s going on for me. See attached image for an example. It is easy to see where the memory goes now:

Realiser 46836
GHI 4064
Microsoft 72544

Now I have some questions…

Is there a better way to profile the app to see what is using memory and how much?

Is there a way to hook into the debugger to get real stats and not rely on the debug output?

The numbers don’t add up… The total calculates correct for ROM, but is a few hundred bytes out for RAM in the Deploy output. I also throught the detail items (e.g. AssemblyRef, TypeRef etc.) counts up to METADATA but it doesn’t add up either. Can someone explain how the numbers are made up please?

When it failed to deploy, the Deploy window reported 103292 bytes. It finally worked when I shaved it down to 90208. This makes sense because of the 100k/48k split - but what I don’t understand is why it now reports 123444 ROM after deployment.

So, Device Deployment window now says: "Deploying assemblies for a total size of 90208 bytes"
and Debug window says “Total: (13196 RAM - 123444 ROM - 58116 METADATA)”

Any assistance is greatly appreciated.

Gus_Issa · December 29, 2011, 9:05am

I 100% agree with you and this should be brought up by everyone to Microsoft directly. GHI can’t add such feature as it is related to the VS plug in, not the device. http://netmf.codeplex.com/

ianlee74 · December 29, 2011, 12:17pm

@ realiser Add your suggestion to Microsoft and post a link here and we’ll all chime in

jasdev · December 29, 2011, 1:03pm

@ realiser This is really useful information. Any chance of publishing your Excel spreadsheets and macros?

realiser · January 3, 2012, 1:53pm

Just an update on this…

I spent some time optimizing my project and it is showing a 160% improvement overall so far using some basic techniques.

Have a read on my blog: [url]http://rudieshepherd.wordpress.com/2012/01/03/putting-netmf-on-a-diet[/url]

@ jasdev I’ll publish and explain the spreadsheet in the next instalment.

jasdev · January 3, 2012, 6:57pm

@ realiser
Thanks. I read your blog entry with great interest. I will need to go through a similar excersize on my Panda II project. It currently uses about 150K of ROM space, which I believe means I have about (180K - 150K) 30K bytes left for more application code. It’s probably enough, but I would still like to trim it down as much as possible.

It seems that the most significant change you made was to eliminate unused code. The other changes didn’t result in a big reduction in ROM size. In my application I also have long debug strings, and was hoping that eliminating them would help, but it doesn’t look like it will

realiser · January 4, 2012, 2:09pm

Part 2 published here…
[url]http://rudieshepherd.wordpress.com/2012/01/04/putting-netmf-on-a-diet-part-2/[/url]

I’m about ready to give up. I don’t know what else to try - so any comments and suggestions would be most welcome.

ransomhall · January 4, 2012, 3:18pm

I’ve never seen such a detailed breakdown of what’s actually “in there”. Both articles are great! Thanks for taking the time to do this.

jasdev · January 4, 2012, 5:58pm

I agree. I’ve learned a lot from reading your blog posts.

jasdev · January 10, 2012, 1:48am

@ realiser
I don’t remember if you tried this, but I was able to shave off 2140 bytes by enabling the Project Build “Optimize Code” setting.

realiser · January 10, 2012, 2:24am

Yes, I did try it and got back some bytes.

I’m stuffed again… I added a little bit of profiling code and now USB doesn’t fit anymore. It’s quite the juggling act!

With this profiler lib I implemented conditional compiler attributes and it works nice.

E.g.


        /// <summary>
        /// Clear the scope of the current trace and reset the root.
        /// </summary>
        [Conditional("TINYCLR_TRACE")]
        public static void Reset()
        {
            _CurrentProfile = null;
            _CurrentProfileCount = 0;
        }

See the [Conditional(“TINYCLR_TRACE”)] attribute? I can now drop a bunch of code in my project by switching the compiler switch on and off. I’ll probably do this for other code that I may need but don’t at the moment. At least it is a clean way to switch things in and out as an alternative to #if statements.

I wish I had switches for things like this on a Panda II:
INCLUDE_BITMAP
EXCLUDE_OUTPUTCOMPARE

so I could give up one to have the other

Seeing that I’ve pretty much come to the end of what I can do to optimize my code (other than going RLP), I’m now looking into dynamically loading certain libraries from SD card at runtime as I need them and even doing some culling on the MSIL before deployment. It sure is an interesting learning journey!

ianlee74 · January 10, 2012, 10:35am

Sounds like you’re ready to upgrade to Hydra

realiser · January 10, 2012, 11:29am

Been considering that but now its a matter of pride. This is not a big job for a micro… I know I can do this on an Arduino if I tried.

My PCB is also laid out for the Panda and there’s gone lots of hours into getting to this point.

Hydra is twice the price and it isn’t suited to be a module on a bigger board. My project is aimed at very low cost in production and that’s why I chose USBIZI.

Live and learn I guess.

ianlee74 · January 10, 2012, 11:44am

I have a hunch you’ll have better options in the near future.

Cosmin · January 16, 2012, 7:54am

@ realiser
Being in the same situation as you,I reduced my firmware size with more then 12KB , using
a free and good obfuscator (Eazfuscator.net)

Architect · January 16, 2012, 8:22am

@ Cosmin

That sounds interesting. More details please.

Cosmin · January 16, 2012, 9:40am

The firmware size depends on the length of names of functions,methods,variables etc;shortest name shortest output!
Every obfuscator on the market could renames all of these with one or two characters.
There are some obfuscators that are directly supporting .netmf,but are not free,so I choose Eazfuscator.NET.
With this program installed follow this steps.
1.Add the file bellow to the project.This is specific to Eazfuscator.NET and will set how the code will be obfuscated.
Eazfuscator.NET docs provide more details about this.


// Definition of custom attributes for declarative obfuscation.


using System;
using System.Runtime.InteropServices;
using System.Reflection;

[assembly: Obfuscation(Feature = "string encryption", Exclude = true)]
[assembly: Obfuscation(Feature = "rename symbol names with printable characters", Exclude = false)]


namespace System.Reflection
{
    /// <summary>
    /// Instructs obfuscation tools to take the specified actions for an assembly, type, or member.
    /// </summary>
    [ComVisible(true), AttributeUsage(AttributeTargets.Delegate | AttributeTargets.Parameter | AttributeTargets.Interface | AttributeTargets.Event | AttributeTargets.Field | AttributeTargets.Property | AttributeTargets.Method | AttributeTargets.Enum | AttributeTargets.Struct | AttributeTargets.Class | AttributeTargets.Assembly, AllowMultiple = true, Inherited = false)]
    sealed class ObfuscationAttribute : Attribute
    {
        /// <summary>
        /// Initializes a new instance of the <see cref="ObfuscationAttribute"/> class.
        /// </summary>
        public ObfuscationAttribute()
        {
            this.m_applyToMembers = true;
            this.m_exclude = true;
            this.m_feature = "all";
            this.m_stripAfterObfuscation = true;
        }

        bool m_applyToMembers;

        /// <summary>
        /// Gets or sets a <see cref="System.Boolean"/> value indicating whether the attribute of a type is to apply to the members of the type.
        /// </summary>
        /// <value>
        /// <c>true</c> if the attribute is to apply to the members of the type; otherwise, <c>false</c>. The default is <c>true</c>.
        /// </value>
        public bool ApplyToMembers
        {
            get { return m_applyToMembers; }
            set { m_applyToMembers = value; }
        }

        bool m_exclude;

        /// <summary>
        /// Gets or sets a <see cref="System.Boolean"/> value indicating whether the obfuscation tool should exclude the type or member from obfuscation.
        /// </summary>
        /// <value>
        /// <c>true</c> if the type or member to which this attribute is applied should be excluded from obfuscation; otherwise, <c>false</c>.
        /// The default is <c>true</c>.
        /// </value>
        public bool Exclude
        {
            get { return m_exclude; }
            set { m_exclude = value; }
        }

        string m_feature;

        /// <summary>
        /// Gets or sets a string value that is recognized by the obfuscation tool, and which specifies processing options.
        /// </summary>
        /// <value>
        /// A string value that is recognized by the obfuscation tool, and which specifies processing options. The default is "all".
        /// </value>
        public string Feature
        {
            get { return m_feature; }
            set { m_feature = value; }
        }

        bool m_stripAfterObfuscation;

        /// <summary>
        /// Gets or sets a <see cref="System.Boolean"/> value indicating whether the obfuscation tool should remove the attribute after processing.
        /// </summary>
        /// <value>
        /// <c>true</c> if the obfuscation tool should remove the attribute after processing; otherwise, <c>false</c>.
        /// The default value for this property is <c>true</c>.
        /// </value>
        public bool StripAfterObfuscation
        {
            get { return m_stripAfterObfuscation; }
            set { m_stripAfterObfuscation = value; }
        }
    }
}

2.Edit the post build event to run Eazfuscator.NET


if /I "$(ConfigurationName)" == "Release" Eazfuscator.NET.exe "$(TargetPath)" --msbuild-project-path "$(ProjectPath)" --msbuild-project-configuration "$(ConfigurationName)" --msbuild-project-platform "$(PlatformName)" --msbuild-solution-path "$(SolutionPath)" -n --newline-flush -v 3.1  -s

3.The next step is to run netmf MetaDataProcessor.exe against the obfuscated assembly,by adding to the postbuild event this command


if /I "$(ConfigurationName)" == "Release"  "C:\Program Files\Microsoft .NET Micro Framework\v4.1\Tools\MetaDataProcessor.exe" -verbose   -loadHints GHIElectronics.NETMF.Hardware "C:\Program Files\GHI Electronics\GHI NETMF v4.1 

SDK\Assemblies\GHIElectronics.NETMF.Hardware.dll" -loadHints GHIElectronics.NETMF.System "C:\Program Files\GHI Electronics\GHI NETMF v4.1 SDK\Assemblies\GHIElectronics.NETMF.System.dll" -loadHints GHIElectronics.NETMF.SystemUpdate "C:\Program 

Files\GHI Electronics\GHI NETMF v4.1 SDK\Assemblies\GHIElectronics.NETMF.SystemUpdate.dll" -loadHints Microsoft.SPOT.Hardware "C:\Program Files\Microsoft .NET Micro Framework\v4.1\Assemblies\le\Microsoft.SPOT.Hardware.dll" -loadHints 
Microsoft.SPOT.Hardware.SerialPort "C:\Program Files\Microsoft .NET Micro Framework\v4.1\Assemblies\le\Microsoft.SPOT.Hardware.SerialPort.dll" -loadHints Microsoft.SPOT.Native "C:\Program Files\Microsoft .NET Micro Framework\v4.1
\Assemblies\le\Microsoft.SPOT.Native.dll" -loadHints mscorlib "C:\Program Files\Microsoft .NET Micro Framework\v4.1\Assemblies\le\mscorlib.dll" -endian le -parse $(TargetDir)$(TargetFileName)  -minimize -compile $(TargetDir)le\$(TargetName).pe

This command must be changed according to the assemblies used in project. I have no idea how to reuse the existing project settings .
That’s all.
The new generated firmware couldn’t be debuged,so is a good idea to keep an unobfuscated project configuration.

Architect · January 16, 2012, 9:52am

That is a very good tip. Would be nice to have it on Wiki.
Thank you!

realiser · January 16, 2012, 10:49am

That is an awesome tip! I’ll definately try it when I get a moment of free (memory) time.

JacquesBlom · February 11, 2012, 12:11am

Thanks realiser!!!

I started running into deployment issues on my navigation button “prototype” app and figured out it was size related. This code read which of 5 buttons (up, down, left, right and select) is pressed and navigates through various “screens” on my 128x64 LCD. These “screens” are typically config screens.

I recalled seeing this thread of yours and read your blogs. I downloaded the Excel macro and found what was nailing me very hard.

I am also using the InternalFlashStorageEx class to read and write few setting in the Panda II’s flash memory. The method GetHashtableFromXMLMemoryStream uses the XmlTextReader and therefore I referenced System.XML.Legacy which was the main culprit.

“Deploying assemblies for a total size” before removing System.XML.Legacy was on 147104 but afterwards it was on 91672 !!! That is a huge difference of 55432 !!!

So I guess I’ll be re-writing the GetHashtableFromXMLMemoryStream method without using any XML classes.