Garbage collection randomly crashes program

TinyCLR OS
1

Hi All,
My colleagues and I had an issue with intermittent program crashes. We’ve been able to narrow it down to the garbage collector, by narrowing it down to a thrown out of memory exception and tracking our available memory. It appears that when the memory is low enough, garbage collection will automatically trigger. Over time, our managed memory drops to zero, is cleaned up, then begins counting down again. Occasionally, when garbage collection triggers, the entire program locks up.

Are there any known issues with particular types of method calls that interfere with garbage collection? None of the existing topics I could find were related to TinyCLR. Either GC enters an infinite loop or throws an exception that causes the entire program to crash.

2

This is a good read Memory Management

3

Hi sgtyar95. We had similar issues with GC. Our program got locked after a while, apparently due to GC running when available memory was too low.

We solved that adding a thread that calls System.GC.GetTotalMemory(true); periodically. We also keep a track of available memory to fine tune the interval. Here is the code we use:

public static void Thread_GC()
{
    long freeRAM;

    while (true)
    {
        Thread.Sleep(P.WBOX.g_gctm);        // P.WBOX.g_gctm: configurable interval, in milliseconds

        // P.RAMfreeMIN keeps track of minimun available RAM (we can reset it whenever we want)
        freeRAM = GHIElectronics.TinyCLR.Native.Memory.ManagedMemory.FreeBytes;
        if (freeRAM < P.RAMfreeMIN) P.RAMfreeMIN = freeRAM;

        // Call to GetTotalMemory method with wait for GC
        System.GC.GetTotalMemory(true);
    }
}

I hope this helps.

4

If GC is a problem, you should be researching why. especially if your application is still under development.

with smart design and coding, GC can be minimized.

A thread doing GC is treating the symptom not the disease.

5

What are some things to do to minimize the impact of GC ?

6

Hi jcplaza,
Thanks for the reply. This solution seems to be the consensus for now until we can figure out what the garbage collection is locking horns with - good to know that it works for you.

1 Like
7
  1. Preallocate and reuse buffers
  2. Use StringBuffer for assembling strings
  3. Be aware when allocation is being done (the keyword new is a hint)
  4. Preallocate and reuse objects
8

HI mike,

Thanks for your input. I’ll be on the lookout for things in the code that are constantly creating new objects. The way the application is designed utilizes a lot of static variables - we don’t do dynamic strings at all and instead use string libraries, all of our parameters and values are stored and manipulated using static libraries, etc. The memory still seems to be filling up very quickly. I am trying to go through the code to see if there is anything frequently called that creates objects, but it will take some time as the project is really large.

9

We have a variable shown on the LCD display indicating the current available memory. We can see when the GC occurs because the available memory is reset to 32mb. I occasionally get an ‘out of memory’ exception when the GC is running. Is there a chance that an event, timer or serial receive, would try to allocate memory during the GC. ? It seems to be at this time that the system hangs ?

10

Can you share your simple code?

11

Events can eat up memory. There is an internal queue for the event handler. Each event creates an entry for the queue. If you don’t process events fast enough, memory disappears into the queue.

I suppose a high event rate would also cause GC to occur.

Dat, are the event queue elements reused?

12

force garbage collection:
System.GC.GetTotalMemory(true);

Does this call block ?
if an event occurs during collection, and the event handler requests memory, what happens ?

13

Yes, I believe so.

14

If an asynchronous event occurs, like a timer or serial, during garbage collection, and that event handler allocates from the heap, what happens ?

15

Which asynchronous events are you asking about?

There is the interrupt handler, and the event handler.

The interrupt handler is pre-allocating memory and reusing elements.

The event handler, which is in managed code, has no issue with GC. In fact, it could contain the stimulus for GC.

But, if events are being put into the event queue faster than they are being handled, you will run out of managed memory.

16

We are currently doing garbage collection manually.(System.GC.GetTotalMemory(true):wink: We monitor free memory and initiate GC when free memory is under 4MB. Occasionally, I still see out of memory exception. What would cause that ?

17

Doing GC manually does not guarantee that you will not run out of memory.

As I said earlier, manually invoking GC addresses the symptom, not the problem.

If you had 6MB of memory left, and requested 7MB you could get a memory exception.

Memory fragmentation could also cause an issue. That is when there appears to be enough memory, but not in a continuous piece to meet the allocation request.

The bottom line is you need to review and improve your use of memory. Continual GC is a red flag, with an unpredictable future.

18

Running out of memory is not the issue. I’m concerned with the garbage collection procedure. I’m trying if the GC procedure is being interrupted

19

Okay, I have offered all I have…

20

Thanks for your help. I think I have determined that the GC procedure is not being interrupted. There must be an event queue that is processed in foreground.