Unmanaged heap

TinyCLR OS
1

Righto brains trust - how do we play with large objects in the unmanaged heap?

I can happily read a 70kb file into a byte array from SD but want to then create a string from the byte[] and promptly run out of memory, so i assume we need to play in the unmanaged heap, but buggered if i know how to do it proper like. :flushed:

using System;
using System.Collections;
using System.Diagnostics;
using System.IO;
using System.Text;
using System.Threading;
using GHIElectronics.TinyCLR.Devices.Storage;
using GHIElectronics.TinyCLR.IO;
using GHIElectronics.TinyCLR.Native;

namespace USBTest
{
    class Program
    {
        private static byte[] _data;

        static void Main()
        {
            UnmanagedBuffer uBuffer = new UnmanagedBuffer(100000);
            byte[] uTable = uBuffer.Bytes;
            var sd = StorageController.FromName(@"GHIElectronics.TinyCLR.NativeApis.STM32H7.SdCardStorageController\0");            
        
            string file = $@"{drive.Name}BP2.xml";

            if (File.Exists(file))
            {
                using (var fs = new FileStream(file, FileMode.Open, FileAccess.Read))
                {
                    uTable = new byte[(int)fs.Length];
                    fs.Read(uTable, 0, (int)uTable.Length);
                }
            }
            FileSystem.Flush(sd.Hdc);
            string str = new string(Encoding.UTF8.GetChars(uTable)); // death by lack of ram
            uTable = null;
            uBuffer.Dispose();
            Thread.Sleep(Timeout.Infinite);
        }
    }
}
1 Like
2

Unmanaged heap is for bye arrays only. So your question is on how to create larger strings. Good question and I do but have an answer yet. We can easily enable external memory as the main heap and you would have unlimited memory but then, while difficult, someone can use a logic analyzer to see the data.

3

This 70kb string is a problem in managed memory?

4

70KB is not a problem.

Problem is:

Encoding.UTF8.GetChars(uTable)

uTable is 100,000 byte, not 70KB.

char in C# it is 2 bytes.

It cost about 300KB for converting internally.

you may want to try this:

           var c = new char[70 * 1024];

            for (var i = 0; i < c.Length; i++)
            {
                c[i] = 'a';
            }

            var s = new string(c);
1 Like
5

I started porting code but have no hardware yet but this would be an issue?

image
image.png1135×712 41.2 KB

6

Yes it will be. You need about 2MB to process that one line! Even if we enable full heap on external memory, is it smart to use 2MB in a single line? This is an IoT device, not a server.

7

:rofl:

but the G120 has no issues with this :crazy_face:
just joking, we will see how we handle this but that kind off string size is something we need to handle.

8

We had a meeting about this today and for customers that want unlimited RAM and do not care if data is stored in external SDRAM, we are adding a new config to enable external heap. (G120 did it in external DRAM).

If all tested well, this will be added to preview.

6 Likes
9

Added, there is big pool 32MB, you can swim all day.

6 Likes