GC: performing heap compaction…
#### Exception System.OutOfMemoryException - CLR_E_OUT_OF_MEMORY (1) ####
#### Message:
#### SmartHouse.SD::ReadFile [IP: 000e] ####
#### SmartHouse.Program::ProgramStarted [IP: 0037] ####
A first chance exception of type ‘System.OutOfMemoryException’ occurred in SmartHouse.exe
An unhandled exception of type ‘System.OutOfMemoryException’ occurred in SmartHouse.exe
Before you allocate your buffer, you can force GC to run which might get you a contiguous block to use. This is most likely the issue you’re seeing, not having enough contiguous available memory… but you’re always going to be struggling when you try to do this kind of operation in one, you need to handle the data in smaller chunks.
Yes, it can go on if memory is 4MB and the file is 5MB…There is no disk swap in NETMF
Try to use a buffer with a fixed length and read the file by piece (for example 1MB). I dont know what you need to do with your returned buffer, but using pieces of 1MB will mean change some things in your parsing methods.
I had the same problem when parsing HTML files for an embedded web server…
When you read large files you should never read all at once, because this would need an very large array.
Even if the array would fit in memory (which is not the case here) this is extreamly time consuming.
You should read it in smaler pices in an loop.
You can even reuse the same buffer multiple times.
Here is a small sample (just typed in here, so might not compile, but shows you how to do this)
FileStream stream = new FileStream(filePath, FileMode.Open);
try
{
byte[] buffer = new byte[1024]; // size depends on your needs, larger consumes more memory but is usually faster
int remainingBytes = (int)stream.Length;
while(remainingBytes > 0) // you could use also something like while (!stream.EOF) here
{
// assuming Read returns the number of read bytes and takes the maximum number of bytes to read in 3rd parameter
remainingBytes -= stream.Read(buffer, 0, 1024);
// process data here
}
}
finally
{
// always clean up in a finally
stream.Close();
}
An optimization would be to use 2 or 3 buffer that size. You can then :
read to 1st buffer
read to 2nd buffer and process data in 1st buffer at same time
read to 3rd buffer and process data in 2nd buffer at same time
read to 1st buffer and process data in 3rd buffer at same time
…
I fully agree, even if your proposal to use several buffer ( in threads I guess) is also very hard to debug and master in embedded systems, as it is rarely the case when you need to play with GB files, such as it is often the case in large SQL databases on Servers…
As you can not decide where buffers are initialized in time, this can also provide unattended out of memory when used in conjonction with other treatment threads…Or you have to be sure that you will never exceed the left memory even when all the threaded buffers are initialized at the same time…
I understand what are you trying to tell me. I agree that i must change the code to use stream, but still:
stream.Length = 5455206 Debug.GC(false) = 6424032
MSDN tells, that Debug.GC return free memory. This means that this should work:
Do not forget that your runtime also take some place in memory and the debug stack also. A good test would be to iniialize your buffer with a constant value length and increase it til it throw an exception. Then you will be able to see where is the max cursor for memory allocation…
I think that an array needs a free block of memory. If your memory is fragmented, then there migfht be no free block which is big enough.
The GC tries to defragment memory (at least I think so).
But I think it is still a bad idea to allocate a single array which is nearly as big as available memory.
Processing the data in a 2nd thread is not that hard to do (and debug) if you have a bit experiance with it. (But I agree, without experiance it can be a hard and frustration job)
About memory allocation: I would create the two or three buffers before start reading.
So during processing no more memory allocations are done and the memory usage is constant (more or less).
In fact this is a verry common procedure when reading and processing data in ‘realtime’ like for video/audio playback or capturing.
I have implemented several video capture drivers using this approach.
That is correct, but how else would you do realtime playback of lage audio files?
You can usually not pause playback when loading the next part from file.
Another approach:
If you are sure that the whole file fits in memory, then you could load the file into an array of byte arrays (or an ArrayList).
// code is simply typed in forum editor and might not compile
var bufSize = 1024 * 1024;
var cnt = (stream.Length / bufSize) + 1;
var buffers = new byte[][cnt];
for (in n= 0; n < cnt; ++n)
{
buffers[n] = new buffer[bufSize];
// read here or in a seperate loop
}
This would bypass any array size limitation, because the single arrays can be at different memory locations.
If you look at @ MischaBoender code entry here http://www.tinyclr.com/codeshare/entry/368 it’s a good example of reading large mp3 files and processing them on a seperate thread.
I use it to play big mp3 files without issue and run addressable leds at the same time.
