Hi, I’m writing an application with an HTML forms interface, running on a Panda II with the expansion shield.
I build up my http response by appending to a string, with the html elements as needed.
I do a Debug.gc(true) after each string append to keep the memory from fragmenting, but I’m still running out of memory in this section of code:
Debug.GC(true);
myresponse += "<p>Winter on hour is " + PP.WinterOnHour.ToString() +
". Change Winter on hour to: <input type=\"text\" name=\"newwinteronhour\" />" +
"</p>";
Debug.GC(true);
myresponse += "<input type=\"submit\" value=\"Submit\" />" +
"</form>";
Debug.GC(true);
myresponse += "</body></html>"; <====== out of memory error here.
Debug.GC(true);
byte[] HTML = Encoding.UTF8.GetBytes(myresponse);
response.ContentType = "text/html";
response.OutputStream.Write(HTML, 0, HTML.Length);
response.Close();
Debug.GC(true);
The error returned is:
GC: 4msec 57948 bytes used, 6432 bytes available
Type 0F (STRING ): 3324 bytes
Type 11 (CLASS ): 10428 bytes
Type 12 (VALUETYPE ): 3192 bytes
Type 13 (SZARRAY ): 9744 bytes
Type 15 (FREEBLOCK ): 6432 bytes
Type 17 (ASSEMBLY ): 19068 bytes
Type 18 (WEAKCLASS ): 48 bytes
Type 19 (REFLECTION ): 48 bytes
Type 1B (DELEGATE_HEAD ): 612 bytes
Type 1D (OBJECT_TO_EVENT ): 336 bytes
Type 1E (BINARY_BLOB_HEAD ): 2076 bytes
Type 1F (THREAD ): 2688 bytes
Type 20 (SUBTHREAD ): 288 bytes
Type 21 (STACK_FRAME ): 2292 bytes
Type 22 (TIMER_HEAD ): 72 bytes
Type 27 (FINALIZER_HEAD ): 480 bytes
Type 31 (IO_PORT ): 324 bytes
Type 34 (APPDOMAIN_HEAD ): 72 bytes
Type 36 (APPDOMAIN_ASSEMBLY ): 2856 bytes
Failed allocation for 85 blocks, 1020 bytes
#### Exception System.OutOfMemoryException - CLR_E_OUT_OF_MEMORY (7) ####
#### Message:
#### System.String::Concat [IP: 0000] ####
#### PoolController.Program::httpserverthread [IP: 03c7] ####
A first chance exception of type 'System.OutOfMemoryException' occurred in mscorlib.dll
If I’m reading this correctly, I’m trying to allocate 1020 bytes out of the 6432 available, and failing.
Can anyone point me in the right direction as to why this is failing?
There was another thread here about long responses in HTML and fragmenting it - that’s the only way you’re going to get around this I’d reckon.
The way you’ve explained the error you’re seeing seems right to me - the issue is actually that you can’t find an unfragmented area of memory to allocate your required space.
In fact, you probably have a problem way before the allocation you see throwing you an exception.
6K is already very low memory. You have memory lleaking before that point. Your memory is probaby fragmented, and even if you have 6k available, you probably don’ t have 1020 bytes available in a row. GC should prevent that, however it is far to be perfect ! You expect a single Debug.GC(true) to sort that out, however sometimes I found out that it takes minutes / hours before it is done.
I have been fighting during weeks with the httplistener/Wiz5100 class to make it stable !
This library is very tricky to use, not to say buggy, depending on how you close the Response and after that, you need to close the Context object as well, cleanly managing the exceptions…
I realise I’m already low on memory, this app uses the Spiral libraries and the http listener class, and between them they eat memory like hot fries!!! Also the Spiral libraries are “closed” so I can’t see if there are any memory leaks there, or where I could use them more efficiently.
I force a GC after every append to the http reply string, but it’s not seeming to be enough.
The netconfig stuff looks good, and its tempting to add it to my app, but then I definitely wouldn’t have enough memory.
@ Ross I’ll do some tests on Spiral to investigae if it’s leaking any memory. You can also save some RAM by including only the Spiral libraries you use.
Skewworks.Spiral- Main DLL; required
Skewworks.Spiral.Emulation- Emulator ONLY
Skewworks.Spiral.Input - Used for Modal Prompts
Skewworks.Spiral.Image- Used for steaming images from uSD
now its got even more confusing.
I have moved from “String” to use the StringBuilder class from the code repository here. I’m hoping that by pre-allocating the string space, I will avoid fragmenting the memory with lots of string += as I build up the html response string.
I have defined the stringbuilder as a 4k array of chars, which I allocate as private static (global) right at the start of program. And this is what I see:
Total: (16172 RAM - 173928 ROM - 74746 METADATA)
AssemblyRef = 184 bytes ( 46 elements)
TypeRef = 1420 bytes ( 355 elements)
FieldRef = 68 bytes ( 17 elements)
MethodRef = 1988 bytes ( 497 elements)
TypeDef = 3736 bytes ( 467 elements)
FieldDef = 1776 bytes ( 880 elements)
MethodDef = 4824 bytes ( 2407 elements)
DebuggingInfo = 2436 bytes
Attributes = 48 bytes ( 6 elements)
TypeSpec = 32 bytes ( 8 elements)
Resources Files = 120 bytes ( 5 elements)
Resources = 360 bytes ( 45 elements)
Resources Data = 4418 bytes
Strings = 28515 bytes
Signatures = 10402 bytes
ByteCode = 63709 bytes
The debugging target runtime is loading the application assemblies and starting execution.
Ready.
'Microsoft.SPOT.Debugger.CorDebug.dll' (Managed): Loaded 'C:\Program Files (x86)\Microsoft .NET Micro Framework\v4.1\Assemblies\le\mscorlib.dll', Symbols loaded.
'Microsoft.SPOT.Debugger.CorDebug.dll' (Managed): Loaded 'C:\Program Files (x86)\Microsoft .NET Micro Framework\v4.1\Assemblies\le\Microsoft.SPOT.Native.dll', Symbols loaded.
'Microsoft.SPOT.Debugger.CorDebug.dll' (Managed): Loaded 'C:\Program Files (x86)\Microsoft .NET Micro Framework\v4.1\Assemblies\le\Microsoft.SPOT.Hardware.dll', Symbols loaded.
'Microsoft.SPOT.Debugger.CorDebug.dll' (Managed): Loaded 'C:\Program Files (x86)\Microsoft .NET Micro Framework\v4.1\Assemblies\le\Microsoft.SPOT.IO.dll', Symbols loaded.
'Microsoft.SPOT.Debugger.CorDebug.dll' (Managed): Loaded 'C:\Program Files (x86)\Microsoft .NET Micro Framework\v4.1\Assemblies\le\Microsoft.SPOT.Net.dll', Symbols loaded.
'Microsoft.SPOT.Debugger.CorDebug.dll' (Managed): Loaded 'C:\Program Files (x86)\Microsoft .NET Micro Framework\v4.1\Assemblies\le\Microsoft.SPOT.Graphics.dll', Symbols loaded.
'Microsoft.SPOT.Debugger.CorDebug.dll' (Managed): Loaded 'C:\Program Files (x86)\GHI Electronics\GHI NETMF v4.1 SDK\Assemblies\le\GHIElectronics.NETMF.System.dll'
'Microsoft.SPOT.Debugger.CorDebug.dll' (Managed): Loaded 'C:\Program Files (x86)\GHI Electronics\GHI NETMF v4.1 SDK\Assemblies\le\GHIElectronics.NETMF.Hardware.dll'
'Microsoft.SPOT.Debugger.CorDebug.dll' (Managed): Loaded 'C:\Program Files (x86)\GHI Electronics\GHI NETMF v4.1 SDK\Assemblies\le\GHIElectronics.NETMF.W5100.dll'
'Microsoft.SPOT.Debugger.CorDebug.dll' (Managed): Loaded 'C:\Program Files (x86)\GHI Electronics\GHI NETMF v4.1 SDK\Assemblies\le\GHIElectronics.NETMF.W5100.Dhcp.dll'
'Microsoft.SPOT.Debugger.CorDebug.dll' (Managed): Loaded 'C:\Program Files (x86)\Microsoft .NET Micro Framework\v4.1\Assemblies\le\System.dll', Symbols loaded.
'Microsoft.SPOT.Debugger.CorDebug.dll' (Managed): Loaded 'C:\Program Files (x86)\Skewworks\Spiral\Assemblies\le\Skewworks.Spiral.dll'
'Microsoft.SPOT.Debugger.CorDebug.dll' (Managed): Loaded 'C:\Program Files (x86)\GHI Electronics\GHI NETMF v4.1 SDK\Assemblies\le\GHIElectronics.NETMF.W5100.Http.dll'
'Microsoft.SPOT.Debugger.CorDebug.dll' (Managed): Loaded 'C:\Users\ross\Documents\Visual Studio 2010\Projects\graphicsdemo\graphicsdemo\bin\Debug\le\PoolController.exe', Symbols loaded.
'Microsoft.SPOT.Debugger.CorDebug.dll' (Managed): Loaded 'C:\Program Files (x86)\GHI Electronics\GHI NETMF v4.1 SDK\Assemblies\le\GHIElectronics.NETMF.IO.dll'
'Microsoft.SPOT.Debugger.CorDebug.dll' (Managed): Loaded 'C:\Program Files (x86)\GHI Electronics\GHI NETMF v4.1 SDK\Assemblies\le\FEZPanda_II_GHIElectronics.NETMF.FEZ.dll'
Display: 48
GC: 2msec 36612 bytes used, 27768 bytes available
Type 0F (STRING ): 96 bytes
Type 11 (CLASS ): 2304 bytes
Type 12 (VALUETYPE ): 1212 bytes
Type 13 (SZARRAY ): 2184 bytes
Type 15 (FREEBLOCK ): 27768 bytes
Type 16 (CACHEDBLOCK ): 96 bytes
Type 17 (ASSEMBLY ): 18684 bytes
Type 18 (WEAKCLASS ): 48 bytes
Type 19 (REFLECTION ): 24 bytes
Type 1B (DELEGATE_HEAD ): 216 bytes
Type 1D (OBJECT_TO_EVENT ): 216 bytes
Type 1E (BINARY_BLOB_HEAD ): 5976 bytes
Type 1F (THREAD ): 1152 bytes
Type 20 (SUBTHREAD ): 144 bytes
Type 21 (STACK_FRAME ): 912 bytes
Type 27 (FINALIZER_HEAD ): 240 bytes
Type 31 (IO_PORT ): 252 bytes
Type 34 (APPDOMAIN_HEAD ): 72 bytes
Type 36 (APPDOMAIN_ASSEMBLY ): 2784 bytes
GC: 3msec 42948 bytes used, 21432 bytes available
Type 0F (STRING ): 408 bytes
Type 11 (CLASS ): 3996 bytes
Type 12 (VALUETYPE ): 1572 bytes
Type 13 (SZARRAY ): 6024 bytes
Type 15 (FREEBLOCK ): 21432 bytes
Type 17 (ASSEMBLY ): 18684 bytes
Type 18 (WEAKCLASS ): 48 bytes
Type 19 (REFLECTION ): 48 bytes
Type 1B (DELEGATE_HEAD ): 324 bytes
Type 1D (OBJECT_TO_EVENT ): 240 bytes
Type 1E (BINARY_BLOB_HEAD ): 5976 bytes
Type 1F (THREAD ): 1152 bytes
Type 20 (SUBTHREAD ): 144 bytes
Type 21 (STACK_FRAME ): 912 bytes
Type 22 (TIMER_HEAD ): 72 bytes
Type 27 (FINALIZER_HEAD ): 240 bytes
Type 31 (IO_PORT ): 252 bytes
Type 34 (APPDOMAIN_HEAD ): 72 bytes
Type 36 (APPDOMAIN_ASSEMBLY ): 2784 bytes
Failed allocation for 685 blocks, 8220 bytes
GC: 2msec 42948 bytes used, 21432 bytes available
Type 0F (STRING ): 408 bytes
Type 11 (CLASS ): 3996 bytes
Type 12 (VALUETYPE ): 1572 bytes
Type 13 (SZARRAY ): 6024 bytes
Type 15 (FREEBLOCK ): 21432 bytes
Type 17 (ASSEMBLY ): 18684 bytes
Type 18 (WEAKCLASS ): 48 bytes
Type 19 (REFLECTION ): 48 bytes
Type 1B (DELEGATE_HEAD ): 324 bytes
Type 1D (OBJECT_TO_EVENT ): 240 bytes
Type 1E (BINARY_BLOB_HEAD ): 5976 bytes
Type 1F (THREAD ): 1152 bytes
Type 20 (SUBTHREAD ): 144 bytes
Type 21 (STACK_FRAME ): 912 bytes
Type 22 (TIMER_HEAD ): 72 bytes
Type 27 (FINALIZER_HEAD ): 240 bytes
Type 31 (IO_PORT ): 252 bytes
Type 34 (APPDOMAIN_HEAD ): 72 bytes
Type 36 (APPDOMAIN_ASSEMBLY ): 2784 bytes
Failed allocation for 685 blocks, 8220 bytes
#### Exception System.OutOfMemoryException - CLR_E_OUT_OF_MEMORY (2) ####
#### Message:
#### SUtility.Text.StringBuilder::.ctor [IP: 001b] ####
#### PoolController.Program::.cctor [IP: 006a] ####
A first chance exception of type 'System.OutOfMemoryException' occurred in PoolController.exe
So milliseconds into the program start, before it runs any application code, before creates any Spiral controls, before it initialised any TCP or HTTP, and with an alleged 21k of free memory, it fails to allocate room for 8kbytes.
and this is the constructor that is being called:
public StringBuilder( int AllocationSize )
{
lngAllocSize = AllocationSize;
lngAllocated = lngAllocSize;
_storedString = new char [lngAllocSize];
}
In this second instance, the program is not even running in main yet (I think the global allocations happen first), so there is no code path to call GC.
In the earlier instances, I was calling GC after every string append to try and clear up the fragmentation - didn’t seem to work.
Our main demo for Panda II is a home automation project which uses the display constantly and also hosts a web server that controls lights and reads light and temperature levels 10 times a second using java script! So it is proven that Panda II can handle web and graphics very well.
Are you sure yo are not adding some large assembly that is to needed in your project? Like TinyCore?
Thanks for the comments guys.
I’m not loading any strange assemblies for any funny hardware addons. Its just the basic Panda II
hardware assemblies, spot core, FEZ touch drivers. When the Wiznet DHCP/http code starts, 10k
gets eaten, so those assemblies take a fair chunk of memory.
I don’t think loading strings of an SD card will help me, but thanks for the idea.
I’m seeing the memory errors in two main places - building up the response string (more below) and then
when the string is copied into a byte[] array for the http response.
I build the response string dynamically, depending on what the config parameters are, and what the current time/temperature/pump status is etc. I was building this in a dynamic string and was getting out of memory errors when the string got to about 2k long. Each time you “string +=” the CLR allocates a new chunk of memory for the string, copies the old string and the appended bit into the new space and then allegedly frees the old string space. I was forcing a GC after each append to try and defrag the freed memory, but it wasn’t helping.
Then I found the string builder class in the code section here, which pre-allocates a chunk of memory as a char[] and then copies the stuff into it as needed, and also dynamically grows the array when needed. By allocating the initial array size big enough for the http response, I got rid of the memory error as the string grew, but I then ran out of memory with the conversion of the string into the byte[] array for the final response.
I think I have seen someone else post about splitting the response into multiple parts, which would let you halve smaller blocks to pass all this through. I can’t for the life of me find it now though.
We requested a new feature in 4.2 from Microsoft where you can read part of a binary resource… it was approved and added to 4.2. I can’t wait so it in GHI’s devices.
