I’ve been wrestling with a problem for a while now and some input would be appreciated. My current project makes use of both the .NetMF Ethernet UDP stack and a couple of UARTs (Com1 and Com2).
The problem I am encountering is that when I have a reasonable amount of data going through the Ethernet (both TX and RX) I start getting SerialError.Overrun events. If I suspend all Ethernet traffic no overruns occur. (Baud rate is a lowly 19200 on Com1 and 115200 on Com2). I have tried using the Data_Received event, polled at 20Hz, 50Hz and as fast as the Processor can do it all with the same result. As the issue seems to be the hardware character buffer nothing I can do other than stopping the ethernet transmission seems to make a difference (which makes sense).
Over Ethernet I am sending roughly 78 bytes a second and receiving 238 bytes a second. The tx consists of roughly 11 packets a second and the rx is 14 packets a second (Therefore the main overhead will probably be the stack constructing the packet & crc).
I can choose to route the same data over Com2 (A backup link). In this scenario no error events are raised.
Reading through the documentation this error event is raised when bytes are lost because the USART low-level character buffer is full and does not get serviced in time.
I have performed some process profiling and the Ethernet stack is taking up even more time than my graphics library. (I’ve also tried disabling everything except the UART and Ethernet Stack with the same results).
I have tried to raise the priority of the USART using the Register function with no joy (I’m obviously missing a step somewhere but can’t find any documentation on the issue).
If anyone could shed some light on this it would be great,
Apologies but I don’t have enough work time available to do this just now - I was hoping someone else had experienced (and solved!) the very issue I am encountering.Basically tasking the Ethernet and serial drivers at the same time and seeing if the serial port starts generating serial overrun errors.
I will try to pull something together this week that I can share.
It would be very difficult to diagnose your problem without you breaking the code down to the smallest portion that will cause the error consistently.
If you truly don’t have time to do that than you should go with your backup link for now. Knowing that one method works and the other doesn’t (while being under a time constraint) it seems the most obvious route to take
I will take the time to do it properly - A convenient solution may well be more power! (In the form of a G400-D, masking my shoddy code)
To give you an idea of the scope of the project we built an in-house GUI with multiple panes, windowing, events etc… something you are rather well versed with The project has multiple communication paths to keep in touch with the device it is controlling (Ethernet, 2 x Serial UARTs and so on). I’ve done some basic profiling and essentially it seems that I am tasking the processor with something that is high enough in priority that it causes the system to fail to service COM1 / 2 in time before the next character comes in - even at 19k200.
Are there negative consequences to representing hardware with statics?
E.g.
public static class UDPSockets
{
private static Socket UDPSocket = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
private static IPEndPoint TXEndPoint;
private static EndPoint RXEndPoint;
private static byte[] TempBuffer = new byte[1500]; // 1500 byte buffer for an entire received packet
// Ethernet is always the first interface netif[0] - ChipworkX
static NetworkInterface[] netif = NetworkInterface.GetAllNetworkInterfaces();
public class Message
{
public byte[] Data;
public string IPAddress;
public int Port;
public Message(string IPAddress, int Port, byte[] Data, int Length)
{
this.Data = new byte[Length];
Array.Copy(Data, this.Data, Length); // Copy data locally
this.Port = Port;
this.IPAddress = IPAddress;
}
}
public static void InitUDPSockets(string ipAddressString, string HostIPString, int TXPort, int RXPort)
{
InitialiseEthernet(HostIPString);
IPAddress ipAddress = IPAddress.Parse(ipAddressString); // IP to TX to (usually global 255.255.255.255)
TXEndPoint = new IPEndPoint(ipAddress, TXPort);
RXEndPoint = new IPEndPoint(IPAddress.Any, RXPort);
try
{
UDPSocket.Bind(RXEndPoint);
}
catch (Exception ex)
{
Debug.Print("Ethernet Bind Failed: " + ex.Message);
}
}
static void InitialiseEthernet(string HostIP)
{
if (netif[0].IPAddress != HostIP) netif[0].EnableStaticIP(HostIP, "255.255.255.0", "0.0.0.0");
if (!Ethernet.IsEnabled) Ethernet.Enable();
}
public static void SetIPAddress(string NewIPAdd)
{
netif[0].EnableStaticIP(NewIPAdd, "255.255.255.0", "0.0.0.0");
}
public static void EthTXData(string IPAdd, int Port, byte[] outBuffer)
{
int Count = 0;
if (Ethernet.IsCableConnected) // If not data is discarded
{
if (TXEndPoint.Address != (IPAddress.Parse(IPAdd)))
{
TXEndPoint = new IPEndPoint(IPAddress.Parse(IPAdd), Port); // Only create a new endpoint if necessary
}
while ((outBuffer.Length - Count) > 0)
{
Count += UDPSocket.SendTo(outBuffer, Count, outBuffer.Length - Count, SocketFlags.None, TXEndPoint);
}
Program.EthTXBytes += Count;
}
}
public static Message GetPacket() // Do ethernet Receive
{
Message RxPacket = null;
// Receive all data in the Eth UDP Socket buffer - assume it is a single packet
bool Available = UDPSocket.Poll(1, SelectMode.SelectRead);
if (Available)
{
EndPoint ep = RXEndPoint;
int count = UDPSocket.ReceiveFrom(TempBuffer, SocketFlags.None, ref ep);
string RxIP = (ep as IPEndPoint).Address.ToString();
if ((RxIP != OCPHardware.OCP.IPAddress) && (RxIP != "127.0.0.1")) // Mask Local IP and Loopback IP to prevent receiving own messages
{
RxPacket = new Message((ep as IPEndPoint).Address.ToString(), (ep as IPEndPoint).Port, TempBuffer, count);
}
Program.EthRXBytes += count;
}
return RxPacket;
}
}
Originally all my hardware drivers were dynamically created at startup (during an intialisation phase). The program used multiple queues and threads allowing me to service the ethernet and serial drivers at independent rates.
All data read from a driver or sent to a driver was handled via independent blocking queues. The logic being that I didn’t have to worry about reentrancy or any having to remember to lock using common objects etc…
This quickly became unmanageable as I couldn’t name threads to track what was and was not running. (Another thing to note is that if you pause while debugging the serial driver continues to run in the background - stick a break point over a Port.BytesAvailable and you will see what I mean).
As a consequence I decided to pretty much eliminate the need for timers or multiple threads by allocating processing time to tasks in a main loop using System Ticks. I now had the ability to divide the processing time into phases - same concept as a timer running at 200Hz but running a different task each time (4 tasks = / 4 so 50Hz). Without relying on the dispatcher to relinquish control when I needed it to. I included a Thread.Sleep(0) to allow the dispatcher to intervene when needed in between function calls.
The next step was to use the number of system ticks elapsed before a function call returned and this gave me a reasonable metric (averaged over a few thousand calls) of how various functions were performing.
The serial character buffer overruns are a result of the low level arm peripheral interrupt failing to execute in time before the next serial character enters the holding register. I am unsure of the technique to use to ensure that the low level interrupts are serviced in time beyond randomly inserting Thread.Sleep(0) everywhere.
What high level functions can be called that are blocking and thus prevent low level (Priority 5) interrupts to fire and how can I avoid this scenario?
Next time around I am going to get someone who knows what they are doing to do the job for me
(edit to fix code layout)
if (Timer200Hz.ElapsedMilliseconds >= 5)
{
Timer200Hz.Reset();
switch (Phase200Hz) // Each phase is serviced at 50Hz
{
case PhaseEnum.PhaseA:
Phase200Hz = PhaseEnum.PhaseB; // Move to Phase B in next cycle
TSC2046.ProcessTouch(); // Service Touchscreen
break;
case PhaseEnum.PhaseB:
Phase200Hz = PhaseEnum.PhaseC; // Move to Phase C in next cycle
TSC2046.CheckForTouch(); // Order important - call must occur after ProcessTouch
break;
case PhaseEnum.PhaseC:
Phase200Hz = PhaseEnum.PhaseD; // Move to Phase D in next cycle
GUILib.ProcessGUIEventUpdate(); // Process all outstanding GUI Events
GUILib.ProcessGfxUpdate(); // Service GUI Library
break;
case PhaseEnum.PhaseD:
Phase200Hz = PhaseEnum.PhaseA; // Move to Phase A in next cycle
Channels.ProcessDIOFromStatusInfo(); // Process DIO changes from status packet (Needs to run at 50hz for responsiveness)
break;
}
Timer200Hz.Start();
}
Unless you are enabling and disabling interrupts, you should not be able block the serial port interrupts from occurring. Hardware interrupts are preemptive.
Assuming you are not playing with enabling/disabling interrupts… (if you are, don’t )
I suspect, with your dispatcher, you are not servicing the serial ports often enough, and the internal serial port memory buffers are overflowing.
I think your original approach, process interrupts and place the data into a queue, is the correct design. Keeping track of multiple threads can be difficult, but is easier than developing and trying to debug your own dispatcher.
You mentioned that debugging was problem because the serial driver runs in the background while in the debugger. This occurs regardless of whether you use queues or poll.
Please ignore the two deleted posts (the wonderful quick reply buttons strike again).
I promise I am not playing with enabling / disabling interrupts (Not intentionally anyway - hence enquiring about blocking calls).
Reading through some of the porting kit source code - the USARTs are at a priority level 5. Logically the only way to inadvertently prevent this interrupt from being serviced is to repeatedly trigger a higher priority interrupt. (Fast IRQ, Ethernet etc…?) in line with what you are saying regarding the hardware interrupts.
I have managed to generate the same error in a standalone program with just the Serial and Ethernet drivers running - though it requires much greater throughput to trigger it (All USARTS at 115200)
The specific serial port error that is being generated: -
I have seen over 2k of data happily buffer in the .Net FIFO with the program paused prior to reading it out.
Duly noted regarding the background .Net processes. It would be nice to be able to monitor these processes to ensure that they are receiving enough CPU time.
Skewworks - if you are still monitoring this thread - you’ve built an entire OS using a combination of .NetMF and RLP - have you encountered anything similar?
Going back to basics - I eliminated as many memory allocations as possible by replacing allocations with static buffers and length indexes. The number of UART character buffer overruns has significantly reduced (but are still occurring when serial traffic increases slightly).
Does the allocation of memory take precedence over the pre-emptive hardware interrupts? (i.e. the “new” keyword blocks).
There are limitations to the number of allocations I can remove. At the moment I am using FIFO queues for certain data streams that are asynchronous (such as user input) to be serviced when the main dispatcher swings around again. The only way to remove these would be to use fixed circular buffers or equivalent.
At your data rate, memory allocation can be a killer. I suspect that hardware interrupts are disabled during garbage collection. You would have to go
to the MF core code to determine if this is true.
Rather than circular buffers, I would use a shared pool of buffers. The free buffer pool would be a linked list. The linked list
would be allocated at initialization time. Should the list be exhausted, then additional buffers can be allocated. I would
set an indicator if the free buffers have been exhausted, so additional buffers could be allocated at initialization.
It’s looking more and more likely that the memory allocation is indeed the culprit. Of the two primary UARTs I use, one is running at 115200, the other at 19200. I no longer receive overruns on the 19200 port during normal operation (due to the slower data rate the system has a longer interval to service the interrupt prior to the next character arriving).
I will go with the shared pool of buffers as suggested… basically writing a custom malloc / free at a higher level to get around the pesky GC overhead (now to choose an efficient cluster size…)
@ Mike - your input was much appreciated.
As an aside - a small observation. I am using the following:
A complete custom GUI Library with windowing as the WPF was too slow (Hence glide etc) Including events etc…
A custom Touchscreen driver and 9 point calibration (TSC2046) as the normal one is pretty bad.
A custom SPI manager to handle multiple SPI devices without blocking etc…
My own dispatcher to ensure timely servicing of critical components as tracking thread activity is difficult in .NetMF 4.1 (4.2 allows naming of threads)
…and now my own “memory management” in rough form.
This is all my own fault for trying to do a combination of low level deterministic processing running under a non-deterministic GUI!
The only thing that has kept me going is seeing others manage to produce significantly more impressive projects and providing assistance to those such as myself. As time goes on I hope to be able to contribute more to the community.
Actually, I am sure GC is your friend, but also weilds a knife and isn’t afraid to stick it in your back when you’re not paying attention
UARTs and serial data are actually one of the hardest streams of data to handle in my opinion, since they almost always result in string data and parsing, all of which then require creation and destruction of more storage. I am sure its an art-form I could get to love but so far that’s not how I feel
)