G120 OutOfMemoryException on reading file

NETMF & Gadgeteer
1

Hi all,

I’m trying to read in the contents of a file using the Method ReadStringFromFile below. When i go to instantiate the byte[] array with a size of 1245464 (thats the file size), it fails with the following exception. 1.2MB file, 6.8ish free.

I assume this is a heap problem, that’s the extent of my knowledge. Is there a work around where I can get this data in and parse it?

Thanks!


    #### Exception System.OutOfMemoryException - CLR_E_OUT_OF_MEMORY (1) ####
    #### Message: 
    #### Redliners.PCMSimulator.Helpers.SdCardHelper::ReadStringFromFile [IP: 0033] ####
    #### Redliners.PCMSimulator.Program::ProgramStarted [IP: 0013] ####

public static string ReadStringFromFile(string fileName)
        {

            if(Mainboard == null)
                throw new Exception("Mainboard has to have a value in SdCardHelper");

            string rootDirectory = VolumeInfo.GetVolumes()[0].RootDirectory;
            FileStream FileHandle = new FileStream(rootDirectory +
                     fileName, FileMode.Open, FileAccess.Read);
            byte[] data = new byte[FileHandle.Length];
            int read_count = FileHandle.Read(data, 0, data.Length);
            FileHandle.Close();
            Debug.Print("The size of data read is: " +
                        read_count.ToString());
            Debug.Print("Data from file:");
            Debug.Print(new string(Encoding.UTF8.GetChars(data), 0,
                  read_count));

            return new string(UTF8Encoding.UTF8.GetChars(data));
        }


GC: 4msec 481272 bytes used, 6858396 bytes available
Type 0F (STRING              ):    636 bytes
Type 11 (CLASS               ):   7908 bytes
Type 12 (VALUETYPE           ):   1104 bytes
Type 13 (SZARRAY             ):   4284 bytes
  Type 03 (U1                  ):    636 bytes
  Type 04 (CHAR                ):    648 bytes
  Type 07 (I4                  ):    468 bytes
  Type 11 (CLASS               ):   2532 bytes
Type 15 (FREEBLOCK           ): 6858396 bytes
Type 17 (ASSEMBLY            ):  34200 bytes
Type 18 (WEAKCLASS           ):     96 bytes
Type 19 (REFLECTION          ):    168 bytes
Type 1B (DELEGATE_HEAD       ):    756 bytes
Type 1D (OBJECT_TO_EVENT     ):    168 bytes
Type 1E (BINARY_BLOB_HEAD    ): 424560 bytes
Type 1F (THREAD              ):   1920 bytes
Type 20 (SUBTHREAD           ):    240 bytes
Type 21 (STACK_FRAME         ):   1056 bytes
Type 27 (FINALIZER_HEAD      ):    120 bytes
Type 31 (IO_PORT             ):    108 bytes
Type 34 (APPDOMAIN_HEAD      ):     72 bytes
Type 36 (APPDOMAIN_ASSEMBLY  ):   3876 bytes
6858396
2

yep, work in chunks. The heap limit is something like 700k, but I’d expect to work on smaller chunks, in the sub-100k size.

What are you trying to parse?

3

Thanks for the reply, that was my suspicion.

Its a text file, which contains a large JSON object. This is going to be challenging… :confused:

4

Should be easy with StringBuilder class

5

Whether you use StringBuilder or not, if you try to hold the whole thing in memory, I would expect memory problems. StringBuilder might be even worse if it creates heap fragmentation as it enlarges.

For Json, especially if you don’t care about every value, I would recommend using a ‘sliding window’ parser. Read a small window of text (“small” is relative to your available space, but let’s say 32 or 64K), and parse it as you would if you had the full text available. If you reach the end of the window while in the middle of parsing something, save whatever incomplete text fragment you have (the tail end bit that you could not finish parsing) and read the next window and continue parsing. You can use the same approach for XML, but you have to save a lot more state information (a list of open element tags for instance).

Algorithmically, this is slightly more challenging than if you had everything in memory, but parsing json is a nice one-pass forward-only parsing exercise (how often do you move your parsing pointer backwards even when everything is in memory?). You can pull this off with a window as small as one byte wide. Larger windows are just a time-performance optimization.

6

With 13mb memory I would try StringBuilder first and see how it performs

7

Well, I definitely don’t disagree - if you can get away with StringBuilder, it will certainly be simpler from an implementation standpoint. Having read more, I see that there are two internal implementations of StringBuilder - one which I would expect to work and one which I would expect to fail and I don’t know which one NETMF uses. In either case, this is probably (as I think Mike is implying) a case of “if it works, don’t fix it”.

1 Like
8

Definitely, learn to use Sliding window parsers. There will come a day when you have to parse a file that’s GBs in size; which will be more than your desktop can handle.

1 Like