UART dropping characters G80

After starting to implement the first of several serial ports I have run into a problem of dropped characters. I have read many forum threads and posts about different ways to solve this problem and none of them have really fixed my problem.

So far in my search of the problem I have reduced my event handlers to :


static byte[] v = new byte[100];
static void _debugPort_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
    int totalBytesRead = 0;
    totalBytesRead = _serialPort.BytesToRead;
      
    _serialPort.Read(v, 0, totalBytesRead);
}

static void _serialPort_ErrorReceived(object sender, SerialErrorReceivedEventArgs e)
{            
}
            

and set a break point in the ErrorReceived method. I send a string of around 60bytes once a second and after a few messages (<10) I hit a overrun error in _serialPort_ErrorReceived. Can the datareceived interrupt be getting held up somewhere? Is there a way to increase the priority of the interrupt? Am I thinking down the wrong path?

I do have some other threads running in the background, but they are all idle waiting for other interrupts.

I am really stumped with this, and would massively appreciate some tips or examples of solid serial data receiving.

I can post more code if needed.

Is your event handler just mis-named with ā€œdebugā€ not ā€œserialā€ ?

Can I ask, how do you know that your full 60-byte message has arrived? Maybe thatā€™s related to your problem?

Typically serial ports are ultra reliable. If youā€™re dropping content itā€™s because you havenā€™t made atomic read operations, or have not got a circular buffer working properly (your code is just using that as a straight buffer, reading one chunk and you must be processing it out the other side)

Iā€™d also consider using a different pattern. Read the full buffer size, and use the port.Read() return value that is the bytes that you read (helps the full transfer problem). There are many circular buffer implementations using serial ports that Iā€™ve seen here that donā€™t seem to have your specific challenge - is it possible to repro this with just something on a PC spewing 60-byte messages every second and your code snippet?

I started fleshing out a serial receiver mechanism with a Serial Buffer (from CodeShare) but started stripping it all back trying to find the problem and basically ended up with what I posted: when the data arrives, read whatever is in the buffer and exit as quick as possible.

With my full implementation (SerialBuffers etc) when I send a string it is read by the dataReceived handler in small chunks by checking BytesToRead (1-5 bytes) which is expected, and it worked most of the time. But every so often bytes would go missing in the reception (I had it printing the byte count) which I canā€™t figure out why. I thought it could be the internal buffer being overrun from the system being held up in some other critical section somewhere?

I will try create a small app that demonstrates the problem and post it.

I just think you need to turn your quick linear buffer into a circular one and process it differently. At the moment you must be offloading data between byte arrays outside of the handler, plenty of opportunity to overwrite important data (especially as you have no locking object).

@ joshuaberry - I would implement a while loop, in the data received event handler, which checks for bytes to read. Donā€™t leave until you are sure the system buffer is empty.

Also, what baud rate are you using?

This is the issue I think.
_debugPort_DataReceived isnt thread safe. If any data arrives while you are processing the last batch, it just uses the same buffer and overwrites it, which could cause all sorts of data missing/overrun issues.

At least this would be the issue in desktop .netmf.

1 Like

Thanks for the input everyone.
As Iā€™m preparing my concise example (with circular buffer, locking object etc), I have uncovered that if I do not setup Ethernet, all is well: at 115200 baud I can continuously send strings to the G80 many times a second and there are no drops, whereas with Ethernet ā€˜turned onā€™ even at 9600 the UART receives Overrun errors.

I have fleshed out the errorReceived handler, and Iā€™ve found that each time characters are dropped an ā€˜Overrunā€™ error is fired, but there is only ever 1-2 bytes to read in the bufferā€¦?


        static void _serialPort_ErrorReceived(object sender, SerialErrorReceivedEventArgs e)
        {
            errorCount++;
            switch (e.EventType)
            {
                case SerialError.Frame:
                    Debug.Print("Frame error " + ((SerialPort)sender).BytesToRead);
                    break;
                case SerialError.Overrun:
                    Debug.Print("Overrun error "+ ((SerialPort)sender).BytesToRead);
                    break;
                case SerialError.RXOver:
                    Debug.Print("RXOver error" + ((SerialPort)sender).BytesToRead);
                    break;
                case SerialError.RXParity:
                    Debug.Print("RXParity error" + ((SerialPort)sender).BytesToRead);
                    break;
                case SerialError.TXFull:
                    Debug.Print("TXFull error" + ((SerialPort)sender).BytesToRead);
                    break;
                default:
                    break;
            }
            
        }

I see that in previous releases of the SDK (for eg. https://www.ghielectronics.com/support/netmf/sdk/24/netmf-and-gadgeteer-package-2014-r5) : [quote]ā€œUART loses data during heavy loadsā€[/quote] and that in 2015 R1 in the G400 section: [quote]ā€œImproved serial port reliability, Improved UART performance and reliability.ā€[/quote]

Could having the Ethernet enabled be putting too much load on the G80? Would the G400 be better suited to running a network interface with several serial ports?

Hereā€™s something to demonstrate my problem.

With SetupEthernet() commented, I can send serial data to the G80 at ~1kBytes/s (115200 baud) without drop. With Ethernet enabled I get dropped bytes with the following debug output:
ā€¦
Overrun error 2
Overrun error 1
Overrun error 19
Overrun error 1
Overrun error 3
Overrun error 1
Overrun error 1
Overrun error 0
Overrun error 0
Overrun error 2
ā€¦

All the help is much appreciated.


using System;
using Microsoft.SPOT;

using System.Collections;
using System.Threading;
using Microsoft.SPOT.IO;
using Microsoft.SPOT.Hardware;
using System.IO.Ports;

using GHI.Pins;
using GHI.Networking;
using Microsoft.SPOT.Net;
using Microsoft.SPOT.Net.NetworkInformation;
using System.Text;

namespace SerialCorruptionTest
{    public class Program
    {
        static Thread busyThread;
        static Thread writerThread;
        static SerialBuffer mySerialBuffer;
        static string dataLine = "";
        static int lineNo = 1;

        static EthernetENC28J60 netif;
        static bool isNetworkReady = false;

        static SerialPort _serialPort;
        static OutputPort LED0 = new OutputPort(GHI.Pins.G80.Gpio.PE14, true); // led pd3

        static StringBuilder stringBuidler;

        static AutoResetEvent dataAvailable;
        const int RESPONSE_SIZE = 100;
        static byte[] responseBuffer = new byte[RESPONSE_SIZE];

        public static void Main()
        {
            Debug.Print(Resources.GetString(Resources.StringResources.String1));

            Debug.Print("Program Started");

            try
            {
                _serialPort = new SerialPort("COM1", 115200);
                _serialPort.ReadTimeout = -1;
                _serialPort.DataReceived += serialPort_DataReceived;
                _serialPort.ErrorReceived += _serialPort_ErrorReceived;

                _serialPort.Open();
            }
            catch (Exception e)
            {
                Debug.Print(e.ToString());
            }

            stringBuidler = new StringBuilder();

            // End Initialize RS232 Input
            mySerialBuffer = new SerialBuffer(256);
            //SendMessage.Start();

            dataQueueChanged += dataQueueChanged_Handler;

            busyThread = new Thread(new ThreadStart(BusyThreadFunction));
            writerThread = new Thread(new ThreadStart(WriterFunction));
            busyThread.Start();
            writerThread.Start();

            SetupEthernet();    // If this is commented, the UART drops stop

            while (true)
            {
                LED0.Write(!LED0.Read());

                Thread.Sleep(100);
            }

        }
        
        static void BusyThreadFunction()
        {
            double d;
            while(true)
            {
                for (int i = 0; i < 10000; i++)
                {
                    d = i / 3.0;
                }
            }
        }

        static void WriterFunction()
        {

            dataAvailable = new AutoResetEvent(false);
            string receivedLine;

            while(true)
            {
                if(dataAvailable.WaitOne())
                {
                    lock (stringBuidler)
                    {
                        receivedLine = mySerialBuffer.ReadLine();
                        if (receivedLine == null)
                        {
                            continue;
                        }

                        stringBuidler.Clear();
                        stringBuidler.Append("Received: \"");
                        stringBuidler.Append(receivedLine);
                        stringBuidler.Append("\" ");
                        stringBuidler.AppendLine(errorCount.ToString());

                        SendResponse();
                    }
                }
                
            }
        }

        static void SendResponse()
        {
            int bytesLeft = stringBuidler.Length;
            int count;
            string stringTosEnd = stringBuidler.ToString();

            lock (stringBuidler)
            {

                while (bytesLeft > 0)
                {
                    count = bytesLeft > responseBuffer.Length ? responseBuffer.Length : bytesLeft;
                    byte[] buf = Encoding.UTF8.GetBytes(stringTosEnd.Substring(stringBuidler.Length - bytesLeft, count));

                    Array.Copy(buf, responseBuffer, count);

                    _serialPort.Write(responseBuffer, 0, count);

                    bytesLeft = bytesLeft - (bytesLeft > RESPONSE_SIZE ? RESPONSE_SIZE : bytesLeft);

                }
            }

        }
        
        
        static int errorCount = 0;
        static void _serialPort_ErrorReceived(object sender, SerialErrorReceivedEventArgs e)
        {
            errorCount++;
            switch (e.EventType)
            {
                case SerialError.Frame:
                    Debug.Print("Frame error " + ((SerialPort)sender).BytesToRead);
                    break;
                case SerialError.Overrun:
                    Debug.Print("Overrun error " + ((SerialPort)sender).BytesToRead);
                    break;
                case SerialError.RXOver:
                    Debug.Print("RXOver error" + ((SerialPort)sender).BytesToRead);
                    break;
                case SerialError.RXParity:
                    Debug.Print("RXParity error" + ((SerialPort)sender).BytesToRead);
                    break;
                case SerialError.TXFull:
                    Debug.Print("TXFull error" + ((SerialPort)sender).BytesToRead);
                    break;
                default:
                    break;
            }

        }


        static byte[] v = new byte[1000]; 
        static void serialPort_DataReceived(object sender, SerialDataReceivedEventArgs e)
        {
            int bytesToRead = 0;
            int totalBytesRead = 0;

            lock (_serialPort)
            {
                bytesToRead = _serialPort.BytesToRead;
                while (bytesToRead > 0)
                {
                    totalBytesRead += _serialPort.Read(v, 0, bytesToRead);
                    mySerialBuffer.LoadSerial(v, 0, bytesToRead);

                    bytesToRead = _serialPort.BytesToRead;

                }

                if (totalBytesRead > 0)
                {
                    dataAvailable.Set();

                }
            }

        }

        static void SetupEthernet()
        {
            Debug.Print("Setting up ethernet...");

            netif = new EthernetENC28J60(SPI.SPI_module.SPI2, G80.Gpio.PE2, G80.Gpio.PE1, G80.Gpio.PC1);

            NetworkChange.NetworkAvailabilityChanged += new NetworkAvailabilityChangedEventHandler(NetworkChange_NetworkAvailabilityChanged);
            NetworkChange.NetworkAddressChanged += new NetworkAddressChangedEventHandler(NetworkChange_NetworkAddressChanged);

            netif.Open();

            netif.EnableStaticIP("192.168.47.92", "255.255.255.0", "192.168.47.1");
            netif.EnableStaticDns(new string[] { "192.168.55.1", "8.8.8.8" });
            netif.PhysicalAddress = new byte[] { 0, 33, 3, 0, 0, 5 };

            int waitcnt = 0;
            while (isNetworkReady == false)
            {
                Debug.Print("Wait for network ready " + waitcnt++);
                Thread.Sleep(500);
            }
        }

        static void NetworkChange_NetworkAvailabilityChanged(object sender, NetworkAvailabilityEventArgs e)
        {
            Debug.Print("*****************************Network has changed!************************** ");
            if (e.IsAvailable)
            {
                Debug.Print("The device can takes up to 30 seconds to optain an IP address. Please wait... ");

                //if (_webserver != null)
                //    _webserver.Start();

            }
            else
            {
               
            }
        }

        static void NetworkChange_NetworkAddressChanged(object sender, EventArgs e)
        {
            Debug.Print("New address for the Network Interface ");
            Debug.Print("Is DhCp enabled: " + netif.NetworkInterface.IsDhcpEnabled);
            Debug.Print("Is DynamicDnsEnabled enabled: " + netif.NetworkInterface.IsDynamicDnsEnabled);
            Debug.Print("NetworkInterfaceType " + netif.NetworkInterface.NetworkInterfaceType);
            Debug.Print("Network settings:");
            Debug.Print("IP Address: " + netif.NetworkInterface.IPAddress);
            Debug.Print("Subnet Mask: " + netif.NetworkInterface.SubnetMask);
            Debug.Print("Default Gateway: " + netif.NetworkInterface.GatewayAddress);
            string MACAddress = "";
            for (int i = 0; i < netif.NetworkInterface.PhysicalAddress.Length; i++)
            {
                MACAddress += " " + netif.NetworkInterface.PhysicalAddress[i].ToString();

            }
            Debug.Print("MAC Address: " + MACAddress);
            Debug.Print("Number of DNS servers:" + netif.NetworkInterface.DnsAddresses.Length);
            for (int i = 0; i < netif.NetworkInterface.DnsAddresses.Length; i++)
                Debug.Print("DNS Server " + i.ToString() + ":" + netif.NetworkInterface.DnsAddresses[i]);
            Debug.Print("------------------------------------------------------");
            if (netif.IPAddress != "0.0.0.0")
            {
                isNetworkReady = true;
            }
        }

    }

@ joshuaberry - are you seeing a lot of GC messages?

I dont know if locking the serial port object itself is the best pattern - but Im not the one to askā€¦ Myself, I would have just created a generic object as a mutex and used it to lock the handler function itself, so the processing that used your buffer was prevented from being reentered.


        static byte[] v = new byte[1000]; 
        private Object thisLock = new Object();  

        static void serialPort_DataReceived(object sender, SerialDataReceivedEventArgs e)
        {
            lock (thisLock)
            {
            int bytesToRead = 0;
            int totalBytesRead = 0;

                bytesToRead = _serialPort.BytesToRead;
                while (bytesToRead > 0)
                {
                    totalBytesRead += _serialPort.Read(v, 0, bytesToRead);
                    mySerialBuffer.LoadSerial(v, 0, bytesToRead);

                    bytesToRead = _serialPort.BytesToRead;

                }

                if (totalBytesRead > 0)
                {
                    dataAvailable.Set();

                }
            }

        }

@ Mike - No, Iā€™m not seeing any with or without Debug.EnableGCMessages(true); which is odd. Is there anything else I need to do to turn them on?

Hereā€™s where I call it:

        
...
public static void Main()
       {
            Debug.EnableGCMessages(true);
...

@ mtylerjr - Iā€™ve added your suggestion, but with no success.

@ joshuaberry - in the while loop of the code you posted, the value of bytestoread will not change. you want to check the serial object for more bytes to read.

@ Mike - The dataReceived handler works all the time when the network has not been started, and [em]most[/em] of the time when it has been started, so the logic in the while loop works.

From what I can see this is quite a bare-bones app, which leads to me think I have missed something quite simple (or Iā€™m the first person to have Ethernet and serial connectivity in one applicationā€¦?? :O)

Itā€™d be great if someone could try this on their device to try replicate?

And just to be clear: Iā€™m not married to anything in the example post - it was just something to demonstrate. Iā€™m open and very interested to try any suggestions. I have read quite a few forum posts that seem to be having similar problems, but then the threads go quietā€¦

When I get something up and running Iā€™ll be sure to post it in CodeShare!

what data do you test with - how much, how often?

@ joshuaberry - in the code posted, you did not include the code for your serial buffer. what happens if you comment out the save to buffer code?

******* I think you are using the SerialBuffer class from CodeShare?

If so, then you are reading the serial port, and then the loadserial method is reading the port. The second read, if the internal serial buffer is empty, will cause the event thread to wait, which is not a good thing.

**** on second review, it looks like you might not be using the codeshare classā€¦ What is in your SerialBuffer class?

To start off with, Iā€™m using what looks like modified version of the CodeShare SerialBuffer that Iā€™ve picked up from the forum. Iā€™ve copied what Iā€™m using below.
Commenting out the mySerialBuffer.LoadSerial() doesnā€™t make a difference. I donā€™t think itā€™s the processing time of the ISR thatā€™s the problem. As I mentioned in the first post of this thread: overruns occur even when the only thing that happens in the ISR is reading of the serial port buffer and exit. Each time the event is serviced the amount of bytes read are typically quite small (1-10bytes) so for a message of ~60bytes the event handler is called a few times.

Here are a few of the other forum threads that I see have a similar problem:
[ul]RS232 buffer overrun on Cerberus when network is open https://www.ghielectronics.com/community/forum/topic?id=16549&page=1
Serial overruns with ChipworkX https://www.ghielectronics.com/community/forum/topic?id=13036&page=1
RS232 module drops characters using Raptor board https://www.ghielectronics.com/community/forum/topic?id=16336&page=1[/ul]

@ Brett - At the moment the data Iā€™m testing with is just a string from the PC: ā€œTest data 123456789Test data 123456789Test data 123456789\nā€
And the response from the G80 is typically something like this (each string is sent every 500ms):
Received: ā€œTest data 123456789Test data 123456789Test data 123456789ā€ 0
Received: ā€œTest data 123456789Test data 123456789Test data 123456789ā€ 0
Received: ā€œTest data 123456789Test data 1234567Test data 123456789ā€ 0
Received: ā€œTest data 123456789Test data 123456789Test data 123456789ā€ 0

Hereā€™s the SerialBuffer class:

    public class SerialBuffer
    {
        private System.Text.Decoder decoder = System.Text.UTF8Encoding.UTF8.GetDecoder();
        private byte[] buffer;
        private int startIndex = 0;
        private int endIndex = 0;
        private char[] charBuffer;

        public SerialBuffer(int initialSize)
        {
            buffer = new byte[initialSize];
            charBuffer = new char[256];
        }

        public void LoadSerial(byte[] data, int startIndex, int length)
        {
            if (buffer.Length < endIndex + length) // do we have enough buffer to hold this read?
            {
                // if not, look and see if we have enough free space at the front
                if (buffer.Length - DataSize >= length)
                {
                    ShiftBuffer();
                }
                else
                {
                    // not enough room, we'll have to make a bigger buffer
                    ExpandBuffer(DataSize + length);
                }
            }
            //Debug.Print("serial buffer load " + bytesToRead + " bytes read, " + DataSize + " buffer bytes before read");
            //Array.Copy(data, 0, buffer, endIndex, length);
            Array.Copy(data, startIndex, buffer, endIndex, length);
            endIndex += length;
        }


        private void ShiftBuffer()
        {
            // move the data to the left, reclaiming space from the data already read out
            Array.Copy(buffer, startIndex, buffer, 0, DataSize);
            endIndex = DataSize;
            startIndex = 0;
        }

        private void ExpandBuffer(int newSize)
        {
            byte[] newBuffer = new byte[newSize];
            Array.Copy(buffer, startIndex, newBuffer, 0, DataSize);
            buffer = newBuffer;
            endIndex = DataSize;
            startIndex = 0;
        }

        public byte[] Buffer
        {
            get
            {
                return buffer;
            }
        }

        public int DataSize
        {
            get
            {
                return endIndex - startIndex;
            }
        }

        public void ClearBuffer()
        {
            startIndex = endIndex;
        }


        

        public string ReadLine()
        {
            lock (buffer)
            {
                // HACK: not looking for \r, just assuming that \r and \n come together        
                int lineEndPos = Array.IndexOf(buffer, (byte)0x0A, startIndex, DataSize);
                if (lineEndPos > 0)
                {
                    int lineLength = lineEndPos - startIndex;
                    if (charBuffer.Length < lineLength)  // do we have enough space in our char buffer?
                    {
                        charBuffer = new char[lineLength];
                    }
                    int bytesUsed, charsUsed;
                    bool completed;
                    decoder.Convert(buffer, startIndex, lineLength, charBuffer, 0, lineLength, true, out bytesUsed, out charsUsed, out completed);
                    string line = new string(charBuffer, 0, lineLength);
                    startIndex = lineEndPos + 1;
                    //Debug.Print("found string length " + lineLength + "; new buffer = " + startIndex + " to " + endIndex);
                    return line;
                }
                else
                {
                    return null;
                }
            }
        }

    }

Iā€™ve found another forum thread (posted 3 years ago) that seems to describe my problem exactly: https://www.ghielectronics.com/community/forum/topic?id=11275&page=2

It would seem that UART interrupts are disabled during GC, and when the network stack is initialised and running, the GC is running more frequently.

So, my question then is: is there any way the UART can be relied on at all for critical data if the GC can run at any time and prevent UART interrupts?

Losing UART data during GC would be a pretty severe issue. Gus & co, do you know anything about it?

@ godefroi - you own G80?

I do not see how GC would kill UART data. But it is possible that a buddy system, including running GC, would have overrun of data. There is an event handler to detect overruns.