RS232 buffer overrun on Cerberus when network is open

Hi,

I’m having a problem with RS232 character drops on a cerberus (4.3).
No matter how fast I try to recevie the characters, when the wired network is open I have lots of overrun. When the network if closed, it doesn’t miss anything.

The speed is only 19200 bauds, and the other side doesn’t support flow control.

I have made 2 samples to reproduce the problem :
A PC program sends batches of data (0 0 0 + numbers from 1 to 250), the cerberus synchronizes on the 3 ‘0’, checks the numbers and sends the result to the PC.
It roughly simulates what the actual code do.

This first part runs on the cerberus


 #define USE_NETWORK
 #define CERBERUS
//#define RAPTOR

using System;
using System.IO.Ports;
using System.Text;
using System.Threading;
using Microsoft.SPOT;

 #if USE_NETWORK
using GHI.Networking;
using Microsoft.SPOT.Hardware;
using Microsoft.SPOT.Net.NetworkInformation;
 #endif

namespace TestComPort
{
    public static class SerialPortLoop
    {
        private static readonly ManualResetEvent stopRequest = new ManualResetEvent(false);
        private static readonly CircularBuffer buffer = new CircularBuffer(512);
        private static SerialPort com;
        private static int totalReceived, wrong;

 #if USE_NETWORK
 #if CERBERUS
        private static readonly EthernetENC28J60 netIf
            = new EthernetENC28J60(SPI.SPI_module.SPI1,
                                   (Cpu.Pin)13, // Socket 6 pin 6
                                   (Cpu.Pin)14, // Socket 6 pin 3
                                   (Cpu.Pin)26, // Socket 6 pin 4
                                   4000);
 #elif RAPTOR
        private static readonly EthernetENC28J60 netIf
            = new EthernetENC28J60(SPI.SPI_module.SPI1,
                                   (Cpu.Pin)28, // Socket 3 pin 6
                                   (Cpu.Pin)33, // Socket 3 pin 3
                                   (Cpu.Pin)87, // Socket 3 pin 4
                                   4000);
 #endif
 #endif


        public static void Main()
        {
 #if USE_NETWORK
            NetworkChange.NetworkAvailabilityChanged += NetworkAvailabilityChanged;
            netIf.Open();
 #endif

 #if CERBERUS
            const string comName = "COM2";  // Socket 2
 #elif RAPTOR
            const string comName = "COM1";  // Socket 10
 #endif

            com = new SerialPort(comName, 19200, Parity.None, 8, StopBits.One);
            com.Handshake = Handshake.None;
            com.Open();
            com.ErrorReceived += ComOnErrorReceived;

            var reader = new Thread(Reader);
            reader.Start();

            byte value = 0;
            while (!stopRequest.WaitOne(0, false))
            {
                int zCnt = 0;
                while (!stopRequest.WaitOne(0, false))
                {
                    // Synch on 3 0x00
                    if (buffer.Read(100, ref value) && (value == 0))
                    {
                        zCnt++;
                        if (zCnt == 3) break;
                    }
                    else
                        zCnt = 0;
                }

                DateTime timeout = DateTime.Now.AddTicks(500 * TimeSpan.TicksPerMillisecond);
                int received = 0;
                while (DateTime.Now < timeout)
                {
                    received = buffer.Length;
                    if (received >= 250)
                        break;

                    Thread.Sleep(1);
                }

                bool ok = true;
                for (int i = 1; i <= 250; i++)
                {
                    if (!buffer.Read(0, ref value) || (value == 0))
                    {
                        ok = false;
                        break;
                    }
                }

                if (!ok) wrong++;

                string msg = "Batch : " + received + ", total bytes " + totalReceived + ", wrong " + wrong;
                Debug.Print(msg);

                var toSend = Encoding.UTF8.GetBytes(msg + "\r\n");
                com.Write(toSend, 0, toSend.Length);

                buffer.Clear();
            }
        }

 #if USE_NETWORK
        private static void NetworkAvailabilityChanged(object sender, NetworkAvailabilityEventArgs e)
        {
            if (e.IsAvailable)
            {
                Debug.Print("Network UP");
                Debug.Print("IP : " + netIf.IPAddress + " / " + netIf.SubnetMask);
                Debug.Print("GW : " + netIf.GatewayAddress);
                foreach (var address in netIf.DnsAddresses)
                    Debug.Print("DNS : " + address);
            }
            else
                Debug.Print("Network DOWN");
        }
 #endif

        private static void ComOnErrorReceived(object sender, SerialErrorReceivedEventArgs serialErrorReceivedEventArgs)
        {
            string msg;
            switch (serialErrorReceivedEventArgs.EventType)
            {
                case SerialError.Frame: msg = "Com framing error"; break;
                case SerialError.Overrun: msg = "Com buffer overrun, next char lost"; break;
                case SerialError.RXOver: msg = "Com input buffer overflow"; break;
                case SerialError.RXParity: msg = "Com parity error"; break;
                case SerialError.TXFull: msg = "Com output buffer full"; break;
                default: msg = serialErrorReceivedEventArgs.EventType.ToString(); break;
            }
            Debug.Print(msg);
        }

        private static void Reader()
        {
            var tmp = new byte[8];
            while (!stopRequest.WaitOne(1, false))
            {
                if (com.BytesToRead != 0)
                {
                    int read = com.Read(tmp, 0, 8);
                    totalReceived += read;
                    buffer.Append(tmp, read);
                }
            }
        }
    }

    public class CircularBuffer
    {
        private readonly byte[] buffer;
        private readonly ManualResetEvent dataAvailable = new ManualResetEvent(false);
        private readonly int last;
        private int readPos;
        private int writePos;
        private int free;

        public CircularBuffer(int size)
        {
            buffer = new byte[size];
            last = size - 1;
            Clear();
        }

        public int Length { get { return buffer.Length - free; } }

        public void Clear()
        {
            lock (buffer)
            {
                dataAvailable.Reset();
                readPos = writePos = 0;
                free = buffer.Length;
            }
        }

        public int Append(byte[] data, int length)
        {
            lock (buffer)
            {
                if ((free == 0) || (length < 1)) return 0;

                free--;
                buffer[writePos] = data[0];
                if (writePos == last) writePos = 0; else writePos++;
                dataAvailable.Set();
                length--;

                if (length < free)
                    free -= length;
                else
                {
                    length = free;
                    free = 0;
                }

                for (int i = 1; i <= length; i++)
                {
                    buffer[writePos] = data[i];
                    if (writePos == last) writePos = 0; else writePos++;
                }
            }
            return length + 1;
        }

        public bool Read(int waitMs, ref byte value)
        {
            if (!dataAvailable.WaitOne(waitMs, false)) return false;

            lock (buffer)
            {
                value = buffer[readPos];
                if (readPos == last) readPos = 0; else readPos++;
                free++;
                if (free == buffer.Length)
                    dataAvailable.Reset();
            }

            return true;
        }
    }
}

Then the PC part :


    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Hit a key to start");
            Console.ReadKey();

            var input = new StringBuilder();
            var data = new byte[253];
            var com = new SerialPort("Com1", 19200, Parity.None, 8, StopBits.One);
            com.DataReceived += (sender, eventArgs) =>
                                {
                                    if (eventArgs.EventType == SerialData.Chars)
                                    {
                                        input.Append(com.ReadExisting());
                                        var s = input.ToString();
                                        int cr = s.IndexOf("\r\n");
                                        if (cr >= 0)
                                        {
                                            Console.WriteLine(s.Substring(0, cr));
                                            input.Remove(0, cr + 1);
                                        }
                                    }
                                };
            com.Open();

            int p = 0;
            for (byte i = 0; i < 3; i++)
                data[p++] = 0;
            for (byte i = 1; i <= 250; i++)
                data[p++] = i;

            p = 0;
            while (!Console.KeyAvailable)
            {
                com.DiscardOutBuffer();
                com.Write(data, 0, data.Length);
                Console.WriteLine("Sent batch {0}", ++p);
                Thread.Sleep(500);
            }
        }
    }

I tried on cerberus, raptor with or without network (by commenting #define), the raptor works fine in both modes.

Thanks in advance for any help…

Best regards,

Steph

I think Andre might have a point. I haven’t studies the code in too great a depth, but these while loops look pretty taxing on the cpu. I don’t see any sleep statements so all threads are probably running flat out and not giving too much time for stuff to happen in the background. I don’t really fully understand the waitone sync mechanism so I could be wrong.

You can try running the CPU monitor from this thread to see.https://www.ghielectronics.com/community/forum/topic?id=15769&page=1#msg156785

Hi,
perhaps it helps to insert a thread.sleep(1) in your Reader.


while (!stopRequest.WaitOne(1, false))
             {
                 if (com.BytesToRead != 0)
                 {
                     int read = com.Read(tmp, 0, 8);
                     totalReceived += read;
                     buffer.Append(tmp, read);
                 }
             Thread.Sleep(1);
             }

Thanks everyone,

Andre.m : the com port is read within a thread. The actual code was initially using the datareceived event, having the same problem. And the code work perfectly when network is not enabled…

RoSchmi : the stopRequest.WaitOne(1… in the reader thread release the cpu 1ms. Sleeping in the loop does not improve :frowning:

hagster : the main “waiting” function, the buffer.Read also relies on waithandler, so it should not overload CPU. Thanks for the tip, I will try monitoring it.
WaitHandlers in "waiting’ state should consume almost no CPU (this is true on real OS like Windows or LInux, I assume it works in the same way in MF…)

Hi,
in each loop you read only few bytes (8) from the serial port.
I have an application where I used a value of 400 instead.


int read = com.Read(tmp, 0, 400);

another point is that you append the received data bytewise to your circular buffer. That’s rather slow in NETMF.
I used the Array.Copy command to append data to a circular buffer, that’s way faster since it is executed in native code behind the Scenes of NETMF.

1 Like

@ andre.m -
Good question. I think that I wanted to avoid action of the carbage collector.
If you first read how many bytes are availabe you have different bytecounts and have to create in each sequence of the while loop a new bytearray which gives work to the garbage collector. When you take one bytearray with a fixed length which is declared outside the loop you can avoid this.

@ andre.m -
I’m not quite sure how you mean this. If I declare the bytearray (here named tmp) as Steph did outside the loop with a length of e.g 400 Bytes and, which can easily happen, more than 400 bytes are available in the buffer of the Serial port I will get an exception if I copy more than 400 bytes into the bytearray buffer tmp which has the length of only 400 bytes. Not right?

@ RoSchmi : I use 8 bytes because, even if netmf is slow, the cerb runs at 160Mhz, and in 1 ms the 19200 bauds com port can’t transmit more than a few bytes. In the non network version of the code, I have “plenty of time” :slight_smile: to debug.print the actual number of bytes read : 2 or 3 each loop…
For the history, the same kind of code ran for monthes on a Spider (@ 70Mhz) perfectly, and it had a lot of other things to do. But it was in 4.1 and I have not converted that app yet.

What mostly annoys me is that those errors are overrun, not receive buffer overflow. If the code can’t handle the bytes (by adding “datetime.now loops” just to make the reader thread too slow), I get buffer overflow…

I will try to make a piece of code that just counts the bytes received, without even treating them.

Thank for your interest !

Ok, so the code that just counts byte :


 #define USE_NETWORK

using System;
using System.IO.Ports;
using System.Text;
using System.Threading;
using Microsoft.SPOT;
 #if USE_NETWORK
using GHI.Networking;
using Microsoft.SPOT.Hardware;
using Microsoft.SPOT.Net.NetworkInformation;
 #endif
using NSS.MF.Tools;

namespace TestComPort
{
    public static class SerialPortLoop2
    {
        private static readonly ManualResetEvent stopRequest = new ManualResetEvent(false);
        private static readonly AutoResetEvent dataAvailable = new AutoResetEvent(false);
        private static SerialPort com;
        private static int received;

 #if USE_NETWORK
        private static readonly EthernetENC28J60 netIf
            = new EthernetENC28J60(SPI.SPI_module.SPI1,
                                   (Cpu.Pin)13, // Socket 6 pin 6
                                   (Cpu.Pin)14, // Socket 6 pin 3
                                   (Cpu.Pin)26, // Socket 6 pin 4
                                   4000);
 #endif


        public static void Main()
        {
            var usage = new CpuUsage();
 #if USE_NETWORK
            NetworkChange.NetworkAvailabilityChanged += NetworkAvailabilityChanged;
            netIf.Open();
 #endif

            com = new SerialPort("COM2", 19200, Parity.None, 8, StopBits.One); // Socket 2
            com.Handshake = Handshake.None;
            com.Open();
            com.ErrorReceived += ComOnErrorReceived;

            var reader = new Thread(Reader);
            reader.Start();

            while (!stopRequest.WaitOne(0, false))
            {
                while (dataAvailable.WaitOne(100, false))
                {
                    // Wait until nothing received for 100 ms
                }
                if (received != 0)
                {
                    var toSend = Encoding.UTF8.GetBytes(received.ToString() + "\r\n");
                    com.Write(toSend, 0, toSend.Length);
                    received = 0;
                }
            }
        }

        private static void Reader()
        {
            var tmp = new byte[16];
            while (!stopRequest.WaitOne(3, false))
            {
                if (com.BytesToRead != 0)
                {
                    int read = com.Read(tmp, 0, 16);
                    received += read;
                    dataAvailable.Set();
                }
            }
        }

 #if USE_NETWORK
        private static void NetworkAvailabilityChanged(object sender, NetworkAvailabilityEventArgs e)
        {
            if (e.IsAvailable)
                Debug.Print("Network UP");
            else
                Debug.Print("Network DOWN");
        }
 #endif

        private static void ComOnErrorReceived(object sender, SerialErrorReceivedEventArgs serialErrorReceivedEventArgs)
        {
            string msg;
            switch (serialErrorReceivedEventArgs.EventType)
            {
                case SerialError.Frame: msg = "Com framing error"; break;
                case SerialError.Overrun: msg = "Com buffer overrun, next char lost"; break;
                case SerialError.RXOver: msg = "Com input buffer overflow"; break;
                case SerialError.RXParity: msg = "Com parity error"; break;
                case SerialError.TXFull: msg = "Com output buffer full"; break;
                default: msg = serialErrorReceivedEventArgs.EventType.ToString(); break;
            }
            Debug.Print(msg);
        }
    }
}

Still lots of overruns :frowning:
No more ideas…

Just trying to run your code. Learned a littel bit how to work in plain NETMF, only used Gadgeteer applications before.

@ Steph - in your reader you should read BytesToRead bytes. anything else is asking for trouble.

if you don’t receive anything for 3ms will your reader thread terminate?

@ Steph -
I have it running on my Cerbuino Bee. Can confirm the issue…

Hi,
did some search in the Internet.
You are not the first with a similar issue.

https://www.ghielectronics.com/community/forum/topic?id=5952

Here are buffer overrun errors, not buffer overflow errors.
These overrun errors may mean that the background processes of the firmware are to slow to catch the signals of the incoming data stream at the serial port. Then it would not be a problem of the application code but of the firmware/hardware-complex.

http://social.msdn.microsoft.com/Forums/vstudio/en-US/da50167c-cee5-440a-b4e1-5dcae3392ca5/serialport-overrun-errors

Made some tests with my “RS232 Gadgeteer <=> PC Round-n-Round” application on Cerbuino Bee with added Ethernet Module. Runs without errors but when the Ethernet gets plugged in there are data errors and the application freezes.

I think now somebody should show an example where serialPort and ethernet work together with no problems.

The ball is on your court, GHI ?

Hi RoSchmi,

That’s what I was afraid of. I hope it could be fixed, as part of the application is relaying data from serial port to ethernet…
I could develop and test on the raptor but the memory pressure is not exactly the same :slight_smile: , and such board if oversized for the final project.
Thanks for your support !

@ MIke : never had problem reading “too much” data. Maybe a little time lost while timing out on reception. Not shure, so I’ll test with the exact amount of data.
No, the reader thead will keep looping until the stopRequest WaitHandler is signaled (Set) somewhere else. Not used in the sample, but in the real app it’s how I control background threads or synchronize them.

Off for a few days (on the beach !) I don’t think the board would appreciate the sand or sea water… or that my wife would appreciate the board on holyday ;D

Have a good time!

Did anyone from GHI see this thread?
Are you going to have a look at this issue?

The problem is in the limited memory of the Cerberus. If we increase the buffer size for UART then we will be decreasing memory to everything else! Maybe we can determine a happy place in the middle but since every need is different, this is not going to be easy.

@ Gus - Thanks for your answer. I have thought that overruns would be more a matter of speed than of buffer size but I have not enough knowledge to enter a discussion. Ethernet and one serial port, at least with low baudrate, would be fine. I’m hopeful that you get it fixed. Enjoy your holiday.

Hi there,
I encounter the same problem using my xbee wifi module. I loose characters from the module in case of multiline responses received from the rn171 wifi-module.
(Operating at 9600 baud). I use the 171 library of GHI , which I modified for the xbee module.

I am afraid that the firmware is not capable of handling the data rate, I hope that GHI can clearify this, and hope I am wrong.

PS: and what about the USB , SPI firmware?

Everything is interrupt driven