SC20260 Freezing on Garbage Collection

I have an application that I am writing on the SC20260 which, in addition to using UART and Network communications, has a GUI with five tabs (five 480x272 screens of information and buttons). It continually reads some analog inputs and displays the values in the GUI. Because of the memory requirements, I have the SC20260 set to use extended heap (also extended deployment, but probably not relevant). The general memory behavior I see is that the free memory starts out high (31 MB) and gradually decreases over time (at about -100,000 Bytes/s) until maybe ~20,000 Bytes are free, and then the garbage collector will run. Sometimes this is successful, and this process starts over again and repeats. First of all, is this normal free memory behavior? My other (smaller) application seems to have a more constant amount of free memory over a long period of time.

The major problem that I am having is that sometimes at the point where the garbage collector runs (before any GC output is printed or the free memory increases) the application will completely freeze up and, if running in the debugger, Visual Studio will stop responding in Windows. I have to power cycle the SC20260 to do anything further, and then Visual Studio window starts responding again and I can press the stop button for the debugger. The same thing happens if I manually execute “GC.Collect” vs. letting it run itself when needed. Depending on what parts of my code I comment out, it can happen on every garbage collection, or if I trim down my code it can succeed for several and then freeze.

My main question is, are there any known issues involving freezing when the garbage collector runs? I have another previously written and smaller application that doesn’t use a GUI or extended heap that I have not previously seen this happen with. Could it have to do with one of those factors?

To follow up, I briefly posted about this in an older somewhat related thread, and the advice was that it probably had to do with two objects that have Dispose methods, and one of them being garbage collected and disposed, which eliminates the underlying native resource that the second instance is still using. I went through all of the uses of hardware resources through serial ports, pins, etc. but couldn’t track down or prevent the freezing.

Since I was compiling the TinyCLR libraries from source, one debugging tactic that I tried was to place a “Debug.WriteLine” statement in every finalizer and Dispose method. Should this show where the problem is?

When I tried this and would see the freezing, the only things printed out from this prior to the freeze were many times the finalizer of a “System.Drawing.Bitmap” ran, each one paired with the Dispose method of “System.Drawing.Graphics”. I tracked this down to Buttons being created dynamically during the application on an ongoing basis. I stopped this from happening, but unfortunately my application still freezes even though it doesn’t show any finalizers or Dispose methods being run. How can that be?

Another thing I tried was to comment out parts of the code and see if any particular section caused it to freeze. Commenting things out would help, in that it would go for several garbage collections without freezing, but which parts of the code I uncommented to get it to break didn’t make sense to me. I’m not sure if it ever worked forever.

I also added a static property that calls “GC.Collect” in its “get” accessor, and added this property to the Watch window in Visual studio, so that I could step through the code and the watch window updating would trigger a garbage collection on every step that I took. The line of code that it would freeze on did not make sense to me.

Are there other approaches like this that can find the problem in some kind of methodical fashion? What else can I do? I am going crazy. Thank you for any help you can give me!

This is your main clue.

You are not managing memory usage……

With embedded system, the excessive use of dynamic allocation/deallocation of memory is asking for trouble. Just like what you are experiencing.

Go to Chat-GPT and ask “how to manage memory on an embedded system”. It will give you lots of good advice.

I would say that I am trying to be memory-conscious, but because TinyCLR is garbage-collected I am not managing memory (like malloc/free in C) like you would in embedded programming. So while the program is running, pins are being read, the GUI is being updated, the user may be clicking on things, typing things into the GUI, or sending serial commands to it, memory is going to be used. But I will say that in my other TinyCLR application I see the reported free memory steady over time. I’m not sure what the difference is between that and here. I’m wondering if it has to do with this being an application with a GUI, or using the extended heap? Do have experience with those conditions?

When I copy/pasted the sentence that you quoted that gave the memory behavior, Chat GPT says:

“Yes, what you’re seeing is normal behavior for a managed runtime like TinyCLR OS (on the SC20260, a SITCore SoC by GHI Electronics).

…

This is classic managed memory allocation and garbage collection behavior, consistent with .NET-based runtimes.”

The invocation of GC is normal. Excessive memory allocation, invoking GC, is not normal.

You are doing something wrong, which you don’t see because you are not thinking how memory allocation is being done.

Do you have a lot of ArrayLists? Not understanding how memory allocation is done with ArrayLists can cause memory allocation issues.

make another thread like this and run to see if any difference.

    new Thread(() =>
    {
        while (true)
        {
            Thread.sleep(1000);

            GC.Collect();
            GC.WaitForPendingFinalizers();
        }

    }).Start();

Thank you, I will try this. I have done something similar, but not the “GC.WaitForPendingFinalizers()”. Is the idea that if finalizers don’t complete, and other code executes, it could cause the freezing that I am seeing?

From my understanding, when the system is low on memory, it triggers garbage collection (GC). If you’re debugging using breakpoints, the system may freeze temporarily, but then it resumes and allows debugging again. It doesn’t stay frozen forever, right?

Yes, I believe it is forever. I have never waited more than a few minutes. But it requires me to power cycle the SC20260 even to get the Visual Studio to respond again.

We received some notes indicating that the issue only lasts for a few seconds before returning to normal. If you can provide a simple code example that reproduces the problem, we’ll be able to investigate further.

But please let us know the result after applying our patch above.

Just a note: if you’re trying to reproduce the issue, don’t apply the patch—it might either fix the problem or make it harder to reproduce.

Thank you for your efforts! The patch above is to run the thread that you gave? If so, that did not make the problem go away for me. I’ll try to make a simple example.

I have also found the automatic GC to be unreliable, so I’ve always implemented what @Dat_Tran suggested. The if statement below is running in a threaded loop. My minimum threshold is 10% of the total memory.

//Execute the garbage collector if the available RAM is below the minimum threshold
if (Memory.ManagedMemory.FreeBytes < MinMemory)
{
    //Execute the garbage collector
    ExecuteGarbageCollector();

    //Verify that the garbage collector worked
    if (Memory.ManagedMemory.FreeBytes < MinMemory)
    {
        Restart(ControllerId, "RAM lockout error.");
    }
}
///<summary> Manually executes the garbage collector, freeing up volatile memory space. </summary>
public static void ExecuteGarbageCollector()
{
    Logger.LogMessage(Logger.Level.Debug, ControllerId, $"Executing garbage collector (Free bytes: {Memory.ManagedMemory.FreeBytes:N0})...");
    GC.Collect();
    GC.WaitForPendingFinalizers();
    Logger.LogMessage(Logger.Level.Debug, ControllerId, $"Garbage collector executed (Free bytes: {Memory.ManagedMemory.FreeBytes:N0}).");
}
1 Like

Whoa….

If you are going to disparage GHI software you have to be more specific.

Without specifics, the issue is more likely to be mismanagement of memory usage.

Dat’s “magic” code will help in certain situations, but will not cure the underlying problem.

My use cases are typically with the heap extended, and the GC + finalizers can often hold up the application for up to 200ms. Sometimes when I’m receiving a large volume of data through various channels and the GC executes with available memory is close to zero, the GC pause can cause the buffers to overflow or even cause memory leaks.

Could very well be me, but the code above was a simple fix that’s never let me down. Since Dat recommended something essentially identical, I wanted to speak up to let them know I’ve had similar issues to the OP. I certainly wouldn’t call it disparaging GHI software.

2 Likes

I have been pulled in other directions, but to update my progress - I have not been able to make a simple example with the same issue that I’m having. In fact, in my full application, I can comment out any combination of 1 or 2 of the tabs of my tabbed UI (5 total) and the application works, but with all of them enabled it freezes. So parts of the code that freeze it sometimes can work other times if something else is commented out. Commenting out the more elaborate tabs by themselves works, but commenting out the sparsest tabs requires 2 of them to be commented out. It seems like it has something to do with the amount of UI elements, memory used, or something.

Is it possible for you to completely disable the Ethernet interface and check if the problem occurs?

Yes, I can try that. Do you have reason to believe that is the problem?