Out of memory but free memory is around 4MB

I get out of memory exception, but i don’t know why, because it is enough memory available (4867572).

GC: 13msec 2472096 bytes used, 4867572 bytes available
Type 0F (STRING ): 4404 bytes
Type 11 (CLASS ): 23628 bytes
Type 12 (VALUETYPE ): 4548 bytes
Type 13 (SZARRAY ): 1616592 bytes
Type 01 (BOOLEAN ): 60 bytes
Type 03 (U1 ): 1578852 bytes
Type 04 (CHAR ): 852 bytes
Type 06 (U2 ): 27192 bytes
Type 07 (I4 ): 1728 bytes
Type 0F (STRING ): 2292 bytes
Type 11 (CLASS ): 5112 bytes
Type 12 (VALUETYPE ): 504 bytes
Type 15 (FREEBLOCK ): 4867572 bytes
Type 17 (ASSEMBLY ): 55260 bytes
Type 18 (WEAKCLASS ): 144 bytes
Type 19 (REFLECTION ): 216 bytes
Type 1B (DELEGATE_HEAD ): 1800 bytes
Type 1C (DELEGATELIST_HEAD ): 204 bytes
Type 1D (OBJECT_TO_EVENT ): 432 bytes
Type 1E (BINARY_BLOB_HEAD ): 750588 bytes
Type 1F (THREAD ): 3072 bytes
Type 20 (SUBTHREAD ): 336 bytes
Type 21 (STACK_FRAME ): 3636 bytes
Type 22 (TIMER_HEAD ): 216 bytes
Type 26 (WAIT_FOR_OBJECT_HEAD): 96 bytes
Type 27 (FINALIZER_HEAD ): 552 bytes
Type 31 (IO_PORT ): 324 bytes
Type 34 (APPDOMAIN_HEAD ): 72 bytes
Type 36 (APPDOMAIN_ASSEMBLY ): 5976 bytes
Free memory: 4867572
GC: performing heap compaction…
#### Exception System.OutOfMemoryException - CLR_E_OUT_OF_MEMORY (7) ####
#### Message:
#### SmartHouse.NET.HttpServer::ParseMultiPartFormData [IP: 0055] ####
#### SmartHouse.NET.HttpServer::RunServer [IP: 00c9] ####
#### SmartHouse.NET.HttpServer+<>c__DisplayClass1::b__0 [IP: 0007] ####
A first chance exception of type ‘System.OutOfMemoryException’ occurred in SmartHouse.exe

The code is from http://www.tinyclr.com/codeshare/entry/186.

        private static Hashtable ParseMultiPartFormData(Stream stream, long contentLength, string boundary)
            Hashtable keyValuePairs = null;

            byte[] data = new byte[POST_MULTIPART_PAYLOAD_SIZE + 1];
            byte[] boundaryData = Encoding.UTF8.GetBytes(boundary);
            int position = 0;

            // Receive all data
            while (true)
                int bytesRead = stream.Read(data, position, POST_MULTIPART_PAYLOAD_SIZE);
                position += bytesRead;

                if ((contentLength >= 0) && (position >= contentLength))

                if (bytesRead == 0)

                // Enlarge buffer
                int newLength = data.Length + POST_MULTIPART_PAYLOAD_SIZE;
                if (newLength > POST_MULTIPART_MAX_SIZE)
                    throw new HttpException("Post data exceeds maximum allowed size");
                byte[] temp = new byte[newLength];

                Debug.Print("Free memory: " + Microsoft.SPOT.Debug.GC(false).ToString());
                data.CopyTo(temp, 0);
                data = temp;

The size is 5MB. I manually copy file to sd and try to open it for sending to music module. I get the same error: out of exception.

        public static byte[] ReadFile(string filePath)
            FileStream stream = new FileStream(filePath, FileMode.Open);
            byte[] buffer = new byte[stream.Length];  //exception raised
            stream.Read(buffer, 0, (int)stream.Length);
            return buffer;
            //filePath = \SD\Sounds\music.mp3        stream.Length = 5455206      	Debug.GC(false) = 6424032

Ok, i understand that. I will change code to write to stream.
But what about this:

        public static byte[] ReadFile(string filePath)
            FileStream stream = new FileStream(filePath, FileMode.Open);
            byte[] buffer = new byte[stream.Length];  //exception raised
            stream.Read(buffer, 0, (int)stream.Length);
            return buffer;
            //filePath = \SD\Sounds\music.mp3        stream.Length = 5455206      	Debug.GC(false) = 6424032

GC: performing heap compaction…
#### Exception System.OutOfMemoryException - CLR_E_OUT_OF_MEMORY (1) ####
#### Message:
#### SmartHouse.SD::ReadFile [IP: 000e] ####
#### SmartHouse.Program::ProgramStarted [IP: 0037] ####
A first chance exception of type ‘System.OutOfMemoryException’ occurred in SmartHouse.exe
An unhandled exception of type ‘System.OutOfMemoryException’ occurred in SmartHouse.exe

Before you allocate your buffer, you can force GC to run which might get you a contiguous block to use. This is most likely the issue you’re seeing, not having enough contiguous available memory… but you’re always going to be struggling when you try to do this kind of operation in one, you need to handle the data in smaller chunks.

@ Makla -

Yes, it can go on if memory is 4MB and the file is 5MB…There is no disk swap in NETMF

Try to use a buffer with a fixed length and read the file by piece (for example 1MB). I dont know what you need to do with your returned buffer, but using pieces of 1MB will mean change some things in your parsing methods.

I had the same problem when parsing HTML files for an embedded web server…

When you read large files you should never read all at once, because this would need an very large array.
Even if the array would fit in memory (which is not the case here) this is extreamly time consuming.
You should read it in smaler pices in an loop.
You can even reuse the same buffer multiple times.

Here is a small sample (just typed in here, so might not compile, but shows you how to do this)

FileStream stream = new FileStream(filePath, FileMode.Open);
   byte[] buffer = new byte[1024]; // size depends on your needs, larger consumes more memory but is usually faster
   int remainingBytes = (int)stream.Length;
   while(remainingBytes > 0) // you could use also something like while (!stream.EOF) here
      // assuming Read returns the number of read bytes and takes the maximum number of bytes to read in 3rd parameter
      remainingBytes -= stream.Read(buffer, 0, 1024);
      // process data here

   // always clean up in a finally

An optimization would be to use 2 or 3 buffer that size. You can then :
read to 1st buffer
read to 2nd buffer and process data in 1st buffer at same time
read to 3rd buffer and process data in 2nd buffer at same time
read to 1st buffer and process data in 3rd buffer at same time

@ Reinhard Ostermeier -

I fully agree, even if your proposal to use several buffer ( in threads I guess) is also very hard to debug and master in embedded systems, as it is rarely the case when you need to play with GB files, such as it is often the case in large SQL databases on Servers…

As you can not decide where buffers are initialized in time, this can also provide unattended out of memory when used in conjonction with other treatment threads…Or you have to be sure that you will never exceed the left memory even when all the threaded buffers are initialized at the same time…

I understand what are you trying to tell me. I agree that i must change the code to use stream, but still:
stream.Length = 5455206 Debug.GC(false) = 6424032
MSDN tells, that Debug.GC return free memory. This means that this should work:

byte[] buffer = new byte[stream.Length];

I have 6MB free memory and i need 5MB.

@ Makla - I think the largest buffer size is set to 4mb…

Do not forget that your runtime also take some place in memory and the debug stack also. A good test would be to iniialize your buffer with a constant value length and increase it til it throw an exception. Then you will be able to see where is the max cursor for memory allocation…

@ andre.marschalek - Yup - Gus :slight_smile:


There are other posts about it…

I think that an array needs a free block of memory. If your memory is fragmented, then there migfht be no free block which is big enough.
The GC tries to defragment memory (at least I think so).
But I think it is still a bad idea to allocate a single array which is nearly as big as available memory.

I am 99% sure you can’t create any objects greater than 4mb

Processing the data in a 2nd thread is not that hard to do (and debug) if you have a bit experiance with it. (But I agree, without experiance it can be a hard and frustration job)
About memory allocation: I would create the two or three buffers before start reading.
So during processing no more memory allocations are done and the memory usage is constant (more or less).
In fact this is a verry common procedure when reading and processing data in ‘realtime’ like for video/audio playback or capturing.
I have implemented several video capture drivers using this approach.

That is correct, but how else would you do realtime playback of lage audio files?
You can usually not pause playback when loading the next part from file.

Another approach:
If you are sure that the whole file fits in memory, then you could load the file into an array of byte arrays (or an ArrayList).

// code is simply typed in forum editor and might not compile
var bufSize = 1024 * 1024;
var cnt = (stream.Length / bufSize) + 1;
var buffers = new byte[][cnt];
for (in n= 0; n < cnt; ++n)
   buffers[n] = new buffer[bufSize];
  // read here or in a seperate loop

This would bypass any array size limitation, because the single arrays can be at different memory locations.

@ Reinhard Ostermeier - You can’t have an array that big

*Edit - sorry your buffer is 1mb which is ok - 4mb is the limit

If you look at @ MischaBoender code entry here http://www.tinyclr.com/codeshare/entry/368 it’s a good example of reading large mp3 files and processing them on a seperate thread.

I use it to play big mp3 files without issue and run addressable leds at the same time.

How big?
if stream.Length is 5MByte then I have 1 array of 5 pointers and 5 arrays of 1MByte.
Or do I see anything wrong here?

On Win .NET arrays are alway passed by reference. is this different in NETMF?