FezDomino : Memory leaking using the HttpListener class?

Nicolas3 · March 16, 2011, 4:49pm

Hello !

Thanks to the new SDK, I built a simple HTTP server to serve pages on my Domino from a SD card. Using the HttpListener class is pretty straight forward. Or I thought so…

After serving (very well) a few pages, my application was crashing with an exception “Out of Memory”.

After a few investigations, I tend to feel that the problem is with a HttpListener class itself. Here is a very simple example to show the problem:


HttpListener Listener = new HttpListener("http", 80);
Listener.Start();
HttpListenerContext ctx;
string s;
byte[] b;
int i;

while (true)
	{
	ctx = Listener.GetContext();
	Debug.Print("Serving a page...");
	s = "<html><body>";
	for (i = 0; i < 500; i++) s += "Hello!<br>"; // a 5KB dumb page
	s += "</body></html>";
	b = Encoding.UTF8.GetBytes(s);
	ctx.Response.ContentLength64 = b.Length;
	ctx.Response.ContentType = "text/html";
	ctx.Response.OutputStream.Write(b, 0, b.Length);
	ctx.Response.OutputStream.Close();           
	s = null;
	b = null;
	Debug.GC(true);
	Debug.Print("Done with the page.");    
	}

Thanks to the Debug.GC(true), we see at the end of each page served that the free memory decreased. Should it ?

After serving first page:
GC: 2msec 24420 bytes used, 39960 bytes available
Type 0F (STRING ): 924 bytes
Type 11 (CLASS ): 3168 bytes
Type 12 (VALUETYPE ): 552 bytes
Type 13 (SZARRAY ): 1956 bytes
Type 15 (FREEBLOCK ): 39960 bytes
Type 16 (CACHEDBLOCK ): 60 bytes
Type 17 (ASSEMBLY ): 14328 bytes
Type 18 (WEAKCLASS ): 48 bytes
Type 19 (REFLECTION ): 24 bytes
Type 1B (DELEGATE_HEAD ): 288 bytes
Type 1D (OBJECT_TO_EVENT ): 96 bytes
Type 1E (BINARY_BLOB_HEAD ): 156 bytes
Type 1F (THREAD ): 384 bytes
Type 20 (SUBTHREAD ): 48 bytes
Type 21 (STACK_FRAME ): 480 bytes
Type 22 (TIMER_HEAD ): 72 bytes
Type 27 (FINALIZER_HEAD ): 96 bytes
Type 31 (IO_PORT ): 108 bytes
Type 34 (APPDOMAIN_HEAD ): 72 bytes
Type 36 (APPDOMAIN_ASSEMBLY ): 1560 bytes

After serving second page:
GC: 2msec 27684 bytes used, 36696 bytes available
Type 0F (STRING ): 1812 bytes
Type 11 (CLASS ): 4632 bytes
Type 12 (VALUETYPE ): 552 bytes
Type 13 (SZARRAY ): 2880 bytes
Type 15 (FREEBLOCK ): 36696 bytes
Type 17 (ASSEMBLY ): 14328 bytes
Type 18 (WEAKCLASS ): 48 bytes
Type 19 (REFLECTION ): 24 bytes
Type 1B (DELEGATE_HEAD ): 288 bytes
Type 1D (OBJECT_TO_EVENT ): 96 bytes
Type 1E (BINARY_BLOB_HEAD ): 156 bytes
Type 1F (THREAD ): 384 bytes
Type 20 (SUBTHREAD ): 48 bytes
Type 21 (STACK_FRAME ): 480 bytes
Type 22 (TIMER_HEAD ): 72 bytes
Type 27 (FINALIZER_HEAD ): 144 bytes
Type 31 (IO_PORT ): 108 bytes
Type 34 (APPDOMAIN_HEAD ): 72 bytes
Type 36 (APPDOMAIN_ASSEMBLY ): 1560 bytes

After serving third page:
GC: 3msec 29508 bytes used, 34872 bytes available
Type 0F (STRING ): 1920 bytes
Type 11 (CLASS ): 5508 bytes
Type 12 (VALUETYPE ): 552 bytes
Type 13 (SZARRAY ): 3672 bytes
Type 15 (FREEBLOCK ): 34872 bytes
Type 17 (ASSEMBLY ): 14328 bytes
Type 18 (WEAKCLASS ): 48 bytes
Type 19 (REFLECTION ): 24 bytes
Type 1B (DELEGATE_HEAD ): 288 bytes
Type 1D (OBJECT_TO_EVENT ): 96 bytes
Type 1E (BINARY_BLOB_HEAD ): 156 bytes
Type 1F (THREAD ): 384 bytes
Type 20 (SUBTHREAD ): 48 bytes
Type 21 (STACK_FRAME ): 480 bytes
Type 22 (TIMER_HEAD ): 72 bytes
Type 27 (FINALIZER_HEAD ): 192 bytes
Type 31 (IO_PORT ): 108 bytes
Type 34 (APPDOMAIN_HEAD ): 72 bytes
Type 36 (APPDOMAIN_ASSEMBLY ): 1560 bytes

etc, etc etc.
We see after each page that the values “CLASS” and “SZARRAY” keep increasing.

Just to make sure that was not something else, and because I am fairly new with C# (but not with programming… 20+ years experience), I built another simple test, doing almost the same, just not using the class:


Debug.GC(true);
string s;
byte[] b;
int i;

while (true)
	{
	Debug.Print("Serving a page...");
	s = "<html><body>";
	for (i = 0; i < 500; i++) s += "Hello!<br>"; // a 5KB dumb page
	s += "</body></html>";    
        b = Encoding.UTF8.GetBytes(s);
	s = null;
	b = null;
	Debug.GC(true);
	Debug.Print("Done with the page.");
	Thread.Sleep(10000);      
	}

Then, of course, we get the same amount of free memory after each run.

So : What am I doing wrong ?

Thank you in advance !
Nicolas

Architect · March 16, 2011, 5:02pm

close response object and context object after you are done with them.

WouterH · March 16, 2011, 5:06pm

See my code at http://www.fezzer.com/project/243/webserver-extension-for-fez-cobra/

Get the response property only once. And close it at the end.

Nicolas3 · March 16, 2011, 5:34pm

Let me know if I did wrong: I changed my code to the following:


HttpListener Listener = new HttpListener("http", 80);
Listener.Start();
HttpListenerContext ctx;
HttpListenerResponse Response;
string s;
byte[] b;
int i;

while (true)
	{
	ctx = Listener.GetContext();
	Debug.Print("Serving a page...");
	s = "<html><body>";
	for (i = 0; i < 500; i++) s += "Hello!<br>"; // a 5KB dumb page
	s += "</body></html>";
	b = Encoding.UTF8.GetBytes(s);
	Response= ctx.Response;
	Response.ContentLength64 = b.Length;
	Response.ContentType = "text/html";
	Response.OutputStream.Write(b, 0, b.Length);
	Response.OutputStream.Close();
	try { Response.Close(); }
	catch { }
	try { ctx.Close(); }
	catch { }
	Response = null;
        ctx=null;
	s = null;
	b = null;
	Debug.GC(true);
	Debug.Print("Done with the page.");
	}

Both Response.Close() and ctx.Close() are Throwing me ‘System.NullReferenceException’ exceptions.

And that does not clear the problem : After each page served, the memory available decreases.

I also did a test with the same code but with regular sockets (no httplistener class) and that works fine without leaking : meaning the W5100 stack is not faulty.

Any idea ?

Architect · March 16, 2011, 5:48pm

I will try your code tonight and see if I have same issue.

WouterH · March 16, 2011, 5:57pm

Can you try to create your page data ‘b’-array only once, so outside the while loop. Note that concatenating strings eats up memory.

Nicolas3 · March 16, 2011, 6:08pm

I did better : I removed the whole 5K load. And it leaks the same amount of memory between 2 pages (about 1,5k) : so that does not link to the page volume.


HttpListener Listener = new HttpListener("http", 80);
Listener.Start();
HttpListenerContext ctx;
HttpListenerResponse Response;
string s;
byte[] b;
int i;
s = "<html><body>Hello, World !</body></html>";
b = Encoding.UTF8.GetBytes(s);

while (true)
	{
	ctx = Listener.GetContext();
	Debug.Print("Serving a page...");           
	Response= ctx.Response;
	Response.ContentLength64 = b.Length;
	Response.ContentType = "text/html";
	Response.OutputStream.Write(b, 0, b.Length);
	Response.OutputStream.Close();
	try { Response.Close(); }
	catch { }
	try { ctx.Close(); }
	catch { }
	Response = null;
	ctx = null;
	Debug.GC(true);
	Debug.Print("Done with the page.");
	}

WouterH · March 16, 2011, 6:25pm

Can it be you’re creating 2 stream objects by accessing OutputStream twice? The documentation isn’t clear about that. Try this:


using (Stream stream = Response.OutputStream)
   stream.Write(b, 0, b.Length);

Nicolas3 · March 16, 2011, 7:29pm

It works !

However, it appears not useful to take out objects from their primary containers, as by anyway -as far as I understand, but again, I’m new to C#- you don’t do a copy of the object but you take a reference to it. So inside or outside should give the same result ? However, a reference in C# might be quite different from the C pointers I am comparing them to !

To make it work, I had to remove closing the OutputStream, just closing the Response and then the Context object (I initialy found it on a netmf example page somewhere, guess it was a wrong pick). That still does not make full sens to me, as when a parent object is closed, it should dispose automaticaly of its children, or anyway GC should do it. So there might still a little glitch somewhere, but that is no big deal.

For information, here is the code that works:


while (true)
	{
	ctx = Listener.GetContext();
	Debug.Print("Serving a page...");           
	ctx.Response.ContentLength64 = b.Length;
	ctx.Response.ContentType = "text/html";
	ctx.Response.OutputStream.Write(b, 0, b.Length);
	ctx.Response.Close(); 
	ctx.Close(); 
	Debug.GC(true);
	Debug.Print("Done with the page.");
	}

Thank you very much to you both for your very kind help !

User_12 · March 16, 2011, 8:33pm

I would like to take this chance to thank Architect for the amazing job he did in porting the HTTP library to get it working with our W5100 socket libraries. You are the man. 8)

WouterH · March 17, 2011, 2:08am

The documentation of the OutputStream property says: Gets ‘a’ stream… That ‘a’ is enough for me to asume that you get a new stream each time you read the property. Otherwise they would have typed ‘the’.

Also all examples out there read them out only once into a local variable (which is assigned on the stack btw, so doesn’t trigger gc, don’t be afraid of it).

Glad you got it working.

Architect · March 17, 2011, 9:48am

@ Joe Thanks! ;D

@ Nicolas I am glad it is working for you now!

User_12 · March 17, 2011, 2:31pm

I will try to post form http example code for W5100 support on fezzer soon. Probably this will easy the start-up

Gus_Issa · March 18, 2011, 10:57pm

Joe, there is already a great example from Wouter http://code.tinyclr.com/project/243/webserver-extension-for-fez-cobra/

Did Wouter also add a netbios class?

Nicolas3 · March 19, 2011, 10:23am

Yes, it’s here : http://code.tinyclr.com/project/244/name-service---give-your-fez-a-real-network-name/