Main Site Documentation

Problems with RLP calls from multiple Managed threads


#1

I’m using RLP with FW V4.3.8.1 on a G120.
Everything works quite fine as long as all RLP calls comes from the same managed thread (app runs for hours, no crash).
I also have the feeling RLP calls from multiple threads do work, as long as I do not call the same function from them (just a Feeling).
But when I make calls from 2 or more threads, my app crashes quite quickly (within minutes).
Sometimes a sleep before the call increases stability, but thats not guaranteed.
The exact behavior ist: Debug-Output to VS stops, no managed Code is executed anymore, but my harware timer is still running (I can see the SPI IO on my osciloscope). Connecting with FEZCconfig does not work (No reponse from device).
Mostly the CPU reboots after a couple of seconds then, but not every time…

Additionally Ethernet (ENC28), multiple UARTS and CAN are initialized, but not actively used when I run my Tests.
In RLP I have set up a timer which communicates to an ADC via SPI, and the RLP functions accesses the latest value from there, so they do Memory Access only and have a quite short runtime.
I even simplified the RLP function to a degree where I can be sure it does not look up.

@ GHI, anyone have similar experience or some additional advise?


#2

That’s why all of my calls to RLP are surrounded like this:

lock (_someLock) {
    try {
        _rlpDisableProcedure.Invoke((int)canChannel);
    } catch (Exception) {
        //Sometimes we get exception here!!! It means there's a glitch in RLP code itself... 
        CanMysticErrorState |= 0x02;
    }
}

“lock” if for inter-thread synchronization. “try” is because, every once in great while, RLP calls to even a very trivial native function “return 1” crashes. For whatever reasons.


#3

This idea came to my mind as well as I was writing the above post (sometimes talking (or writing) to someone about the Problem actually helps :smiley:
I replaced:


with

```cs
lock (_getAverageValue)
{
   return _getAverageValue.Invoke(uints, floats);
}

and it runs fine till then (way longer than usually).
But the try/catch is a good idea.
I have a mysterical ArgumentException on SerialPort.Write(), where I have to Close and open the port and everythis is fine again.


#4

Follow up question to @ GHI:
Since RLP calls are blocking all managed threads:
Any idea why a lock is needed if a RLP function is invoked from multiple threads?


#5

Perhaps the locks aren’t for the RLP. Perhaps it’s the managed code that is calling the RLP causing a problem.

I’d like to be able to reproduce this problem; I’m curious.


#6

@ Mr. John Smith - Just start 2 threads, each with a Loop Invoking the same RLP function, and you should get a Crash within minutes.
I also think now that even so the C Code is atomic, and by this thread safe, the Invoke method around it is not.


#7

@ Reinhard Ostermeier - That’s what i’m thinking also. That the Invoke method isn’t thread safe. Is there a way to put a lock inside of the RLP?


#8

@ Reinhard Ostermeier - The actual native invoke method does block managed threads. That method is “private void NativeInvoke();”. “public int Invoke(params object[] argumentList);” that you call is not actually a native function and performs several argument marshalling functions in managed code that are not thread safe before finally calling NativeInvoke.


#9

So NativeInvoke(), which calls RLP is thread safe, due to the fact that it blocks all managed threads to run the RLP, and .netmf devices are single core. Invoke() performs managed code marshaling before calling Native Invoke, and as such, a call from Thread B can outrun the call from Thread A if there are more parameters being passed to the RLP method (i.e. more marshalling).

@ John - Is it possible to do the marshaling in your own code, then call the NativeInvoke function directly?


#10

@ Mr. John Smith - Doing such a thing wouldn’t get you anything because you’d have to protect your own code from race conditions when you reimplement what we do. public int Invoke calls several native functions depending on the arguments you supply. Any private code in our libraries is subject to change so relying on the internal implementation is always inherently fraught with risk.


#11

@ John - So Simon says Lock on all RLP function calls.


#12

@ Mr. John Smith - Only if the same NativeFunction instance will be accessed by multiple threads. That applies to most of our libraries. We’ve made no thread safe guarantees.


#13

@ Reinhard Ostermeier -

How much heavy RLP take in native code, please?

I tried simple RLP just run a loop for 5 seconds and 2 threads in C# call to it, no sleep.

no need to use lock statement yet, look like I can not reproduce it. :frowning:


#14

@ Dat - We’re thinking that the difference comes when you have a lot of parameters being passed by one thread and not the other.


#15

@ Mr. John Smith -

Sound it can be, I will give it a try


#16

@ Dat - My RLP function has 2 Parameters. one Array of uint and one Array of float (I Need some out values)
Having 2 threads calling the same function works for a couple of minutes before it actually reboots or hangs (both can happen).


#17

@ Reinhard Ostermeier -

what code do you write in RLP function? empty, just while loop, or do something else?


#18

@ Dat - For a test I reduced the RLP function to some verry simple dummy Code. No Loops, just 1 or 2 if’s and some assignments to my out Parameters.
This did not change anything, it crashed the same as with the original code, which is more complex and has a while loop in it.
I assume the problem occurs if the 1st thread has started marshalling the parameters in Invoke, and then the 2nd thread gets Control bevor the RLP function is actually called in the 1st thread.