Cobra II serial port issues

I’ve been experiencing some odd behavior with the serial ports on the Cobra II. I’m using two GHI RS232 Modules plugged into a FEZ Cobra II with an extender. I’m communicating with audio equipment (57600 baud & 9600 baud sending 3-50 character lines of info & command/response). I’ve been using each of the 3 ‘u’ gadgeteer sockets on the board. I’ve a number of tests with each device on each socket and RS232 module. I’ve tried code using LineReceivedEventHandler (prefered), SerialDataReceivedEventHandler, and direct polling.

Gadgeteer Socket 7 (COM4) works perfectly. Perfect communicaiton with either device.

Gadgeteer Socket 5 (COM1) works great 99% of the time. One character goes missing every 100 messages or so it seems. Same with either device…

e.g. Device A on socket 7 works perfectly, device B on socket 5 skips a character once-in-a-while. When I swap gadgeteer plugs & change the diagram in Program.gadgeteer, I get device A on socket 5 skipping characters and device B on socket 7 working perfectly.

Buffers are not full – I’ve seen it happen on a single message. I think using LineReceivedEventHandler may miniize the errors, but the issue is random and I don’t have any statically significant data.

Gadgeteer Socket 8 (COM2) does not respond at all to either device (same code)

I tried all different HardwareFlowControl options (since Socket 8/COM2 seems to have RTS/CTS pins and the RS232 module seems to support them as well), but did not have any luck. I double checked what should be simple solder connections attaching the extender board. I did a conductivity check between the G120 module pins and the corresponding Socket 8 pins - just to make sure nothing was odd.

I am missing something subtle?

I don’t think you’re missing anything subtle, but I haven’t looked at the Cobra2 serial ports in detail.

Can you describe what you mean by missing characters? If I send a long text string, “the quick brown fox jumps over the lazy dog” repeated, do you just miss a random character in the middle? Can you repro this sufficiently to demonstrate it with a simple (polling) app? If so, I’d suggest to give the best chance of anyone else demoing this, a sample app so it can be tested independently. Even if we have to have a PC or a different Fez device on the other end of each serial line…

Interesting to note someone else having serial port issues as I had similar issues with a ChipworX on COM3. I was trying to use a GSM modem on it at 57600 bps and kept getting lost characters and overun errors. I ended up running at 19200 and not loss of data at this speed.

I also have a Cobra 2 and trying to use the same code I get overruns and lost characters above 19200 bps. This is also on COM3. I have not tried any other com port as I would need a different cable but I will get around to checking this out based on your testing.

Looks like an internal driver issue?

The code ran fine on a Device Solutions Meridian board at 57600 but that was on COM1.

yes, the port might receive something like “the uick brown fox jumps over the lazy dog”.
I’ll work on reproducing this. Sending seems fine, so I will try connecting the two serial ports with a null modem cable and send stuff. (the board decided to have deployment issues this morning, so I need to try to fix that first - FEZ_Config see the board, but it won’t ping. TinyBooter seems ok, so I will try re-updating)

Hi,
Im not sure but I think that hardware flowcontrol is still not working. Question to GHI !?
How did you compare the sent and received data? I saw sometimes that when the received data were sent to Debug.print that characters got lost but this was not due to the serial read process but due to the further processing of the the received data.

The lost characters were detected in the parsing of the response string, so they should be exactly what came out. (that code should be the same regardless of which serial port the data comes from).

I did some testing using the cobra II board. connected the two RS232 modules with a null modem and sent data both ways, then swapped things around. I tried various traffic patterns that I would attempted to model the physical devices and could not reproduce the missing characters. (100% data matches on over a million characters) . I’ll keep working on this.

However, COM2 does not send or receive when I plug the GHI RS232 module in Socket 8 (and change the connection in Program.gadgeteer.diagram). If I am the only one with the problem, it could be a soldering mistake.

While doing the tests, I measured the latency of each message (# ticks from send to receive) and learned a bit more about how the EventHandlers work: (about 25 character messages):

  • LineReceivedEventHandler: 200-260ms average (87ms fastest, 138-188ms typical, plus a few 400-500ms)
  • DataReceivedEventHandler: 150-400ms average
    [ If I am not already handling an event: read ‘BytesToRead’ sized chunks until ‘BytesToRead’ is zero. Find newlines and send each line string to same “LineReceived” handler.]
    Looking at each message… it starts off good (90-100 ms latency), then gets worse and worse 6000-9000ms latency. Then it just starts working quickly (36-70ms latency for the last half) It does not correspond to traffic on the other port, as far as I can tell. So I have concluded that it works great, except for a few situations where there is data waiting and the Handler does not fire (there is always new data coming in too… )
  • polling: 89ms ave (typically 40ms to 70ms with a few longer one that bring up the average)
    [simple loop: call DataReceived event handler if there are BytesToRead; sleep(5)]

@ eolson - “then gets worse and worse 6000-9000ms latency”.
hard to believe that it works so bad.
If you could post your code, I could try to reproduce it on my Cobra II.

I was puzzled as well. I cleaned up the code a little and found the case where it behaved strangely. I’m pretty sure it is the other serial port (in Line mode) that is causing interference, things speed up after that test set finishes.

Here are a few sample results:
// 5 x messages, delay
Test 0: 1000/1000 messages from COM4 to COM1 with 0 errors, average latency=190 (rx:DataReceived)
Test 1: 1000/1000 messages from COM1 to COM4 with 0 errors, average latency=232 (rx:LineReceived)

Test 0: 1000/1000 messages from COM4 to COM1 with 0 errors, average latency=226 (rx:Polling)
Test 1: 1000/1000 messages from COM1 to COM4 with 0 errors, average latency=242 (rx:LineReceived)

// 5 x messages, sync
Test 0: 500/500 messages from COM4 to COM1 with 0 errors, average latency=99 (rx:Polling)
Test 1: 500/500 messages from COM1 to COM4 with 0 errors, average latency=255 (rx:LineReceived)

Test 0: 500/500 messages from COM4 to COM1 with 0 errors, average latency=406 (rx:DataReceived)
Test 1: 500/500 messages from COM1 to COM4 with 0 errors, average latency=205 (rx:LineReceived)

Here is the code set-up for the odd case: (the RS232 ports are named rs232_a and rs232_b)

using System;
using System.Collections;
using System.Threading;
using Microsoft.SPOT;
using Microsoft.SPOT.Presentation;
using Microsoft.SPOT.Presentation.Controls;
using Microsoft.SPOT.Presentation.Media;
using Microsoft.SPOT.Touch;

using Gadgeteer.Networking;
using GT = Gadgeteer;
using GTM = Gadgeteer.Modules;
using Gadgeteer.Interfaces;

namespace CobraII_serial_test
{
    public partial class Program
    {
        string port_a;   // test 0 tx (test 1 rx)
        string port_b;   // test 1 tx (test 0 rx)

        // This method is run when the mainboard is powered up or reset.   
        void ProgramStarted()
        {
            Debug.EnableGCMessages(false);
            rs232_a.Initialize(57600, Serial.SerialParity.None, Serial.SerialStopBits.One, 8, Serial.HardwareFlowControl.NotRequired);
            rs232_b.Initialize(57600, Serial.SerialParity.None, Serial.SerialStopBits.One, 8, Serial.HardwareFlowControl.NotRequired);

            port_a = rs232_a.serialPort.PortName;
            port_b = rs232_b.serialPort.PortName;

            // setup rx for port a (line received)
            rs232_a.serialPort.Close();   // need port closed when setting AutoReadLineEnabled - set_ReadTimeout gets called
            rs232_a.serialPort.AutoReadLineEnabled = true;
            rs232_a.serialPort.LineReceived += new Serial.LineReceivedEventHandler(Test_LineReceived);
            rs232_a.serialPort.Open();

            // setup rx for port b (data received)
            // use this line for data received
            rs232_b.serialPort.DataReceived += serialPort_DataReceived;
            
            // use these lines for polling
            //Thread pollThreadB = new Thread(() => serialPort_Poll(rs232_b.serialPort));
            //pollThreadB.Start();
            
            // use these lines for line received
            //rs232_b.serialPort.Close();   // need port closed when setting AutoReadLineEnabled - set_ReadTimeout gets called
            //rs232_b.serialPort.AutoReadLineEnabled = true;
            //rs232_b.serialPort.LineReceived += new Serial.LineReceivedEventHandler(ReadLine1);
            //rs232_b.serialPort.Open();          

            Thread.Sleep(1000);  // make sure things are settled before starting the tests?

            // these need to be on threads - timers or the main thread don't work because
            // they can't happen in parallel with the EventHandlers
            data[0] = new Data("rx:DataReceived");
            Thread TestThread0 = new Thread(() => Test_Transmit(rs232_a.serialPort));   // tx:a rx:b
            TestThread0.Start();
            data[1] = new Data("rx:LineReceived");
            Thread TestThread1 = new Thread(() => Test_Transmit(rs232_b.serialPort));   // tx:b rx:a
            TestThread1.Start();
        }

        class Data
        {
            const int TEST_SIZE = 500;

            public string[] message = new string[TEST_SIZE];       // message text (check for transmission errors)
            public DateTime[] startTime = new DateTime[TEST_SIZE]; // rx time (to compute latency)
            public long[] lantency = new long[TEST_SIZE];   // ms
            public int rxCount = 0;
            public int errCount = 0;
            public long totalLatency = 0;
            public Serial rxPort = null;     // confirm rx port
            public string info = "";
            public Data(string info)
            {
                this.info = info;
            }
        }
        Data[] data = new Data[2];

        private void Test_Transmit(Serial sender)
        {
            int test = (sender.PortName == port_a) ? 0 : 1;  // tx test
            Thread.Sleep(1000);
            Debug.Print("Starting test "+test);
            for (int i = 0; i < data[test].message.Length; i++)
            {
                string msg = "#M" + i + " T=" + DateTime.Now.Ticks.ToString();
                data[test].message[i] = msg;
                sender.WriteLine(msg);
                Mainboard.SetDebugLED(true);   // blink lights for fun
                data[test].startTime[i] = DateTime.Now;

                // play with the traffic pattern

                if ((i % 5) == 0) // send 5 messages and then wait for them to be received
                    while (data[test].rxCount < i + 1)
                        Thread.Sleep(20);

                //if ((i % 5) == 0) Thread.Sleep(500);  // burst of 5 messages and then delay
            }

            // wait for all messages (or no change for .5 seconds)
            int lastCount = -1;
            while ((data[test].rxCount < data[test].message.Length) && (data[test].rxCount != lastCount))
            {
                lastCount = data[0].rxCount;
                Thread.Sleep(500);
            }
            long ms = (data[test].totalLatency / data[test].message.Length);
            Debug.Print("Test "+test+": " + data[test].rxCount +"/"+ data[test].message.Length + " messages from " + sender.PortName + " to "
                + (data[test].rxPort != null ? data[test].rxPort.PortName : "?")
                + " with " + data[test].errCount + " errors, average latency=" + ms + " ("+data[test].info +")");

            // print out latencies for both tests
            if (test == 0)
            {
                while ((data[1].rxCount < data[1].message.Length))
                    Thread.Sleep(500);
                Thread.Sleep(500); // let other message finish 
                for (int i = 0; i < data[test].message.Length; i++)
                {
                    Debug.Print("M" + i + " " + data[0].lantency[i] + " \t" + data[1].lantency[i]);
                }
            }
            

        }
   
        private void Test_LineReceived(Serial sender, string line)
        {
            int test = (sender.PortName == port_a) ? 1 : 0;  // rx test
            int i = Convert.ToInt32(line.Substring(2, line.IndexOf(' ') - 2));
            TimeSpan timediff = DateTime.Now - data[test].startTime[i];
            data[test].lantency[i] = timediff.Ticks / TimeSpan.TicksPerMillisecond;
            data[test].totalLatency += data[test].lantency[i]; 
            Mainboard.SetDebugLED(false);
            if (i == 0)
                Debug.Print("Test" +test+" sample - " + line + " :" + data[test].lantency[i]+"ms");


            if (data[test].message[i] != line)
            {
                Debug.Print("Data mismatch: " + data[test].message[i] + " => " + line);
                data[test].errCount++;
            }
            data[test].rxCount++;
            data[test].rxPort = sender;
        }


        void serialPort_Poll(Serial sender)
        {
            System.IO.Ports.SerialData data = new System.IO.Ports.SerialData();   // dummy for data call
            while (true)
            {
                while (sender.BytesToRead > 0) 
                    serialPort_DataReceived(sender, data);
                Thread.Sleep(5); 
            }
        }

        bool[] dataReceivedRunning = {false, false};
        string[] serialLine = {"",""};
        void serialPort_DataReceived(Serial sender, System.IO.Ports.SerialData data)
        {
            int test = (sender.PortName == port_a) ? 1 : 0;  // rx test
            // only run one at a time
            if (dataReceivedRunning[test]) return;
            dataReceivedRunning[test] = true;
            int total_bytes = 0;

            // turn interrupt off for now
            int read_bytes = sender.BytesToRead;
            while (read_bytes > 0)
            {
                byte[] readBuf = new byte[read_bytes];
                int read = sender.Read(readBuf, 0, read_bytes);
                total_bytes += read;
                for (int i = 0; i < read; i++)
                {
                    if (readBuf[i] == 13) { } // chomp newline
                    else if (readBuf[i] == 10)
                    { // split commands on /r
                        Test_LineReceived(sender, serialLine[test]);
                        serialLine[test] = "";
                    }
                    else
                    {
                        serialLine[test] += (char)readBuf[i];
                    }
                }
                read_bytes = sender.BytesToRead;
            }        
            // hook back up
            dataReceivedRunning[test] = false;
        }

    }
}

@ eolson - thank you for the interesting test code. I just had a look on my Cobra II and saw that I have no extender and so I have no sockets 7 and 8. Perhaps I can use an extender or breakout module (not before the weekend I think).

An idea: Can it be that it lasts to long to process the code in the event handlers that the consecutive events pile up?

DataReceived performed well when things were not busy (5 messages, then .5 second rest). Things broke down when it sent 5 messages, then waited just long enough for all 5 to get received. So they are not allowed to pile up, but they certainly can get busy. From what I can tell, the EventHandlers for both ports are operating on the same thread, so having the other port kept very busy can certainly affect performance. I am not sure if these are driven by actual hardware interrupts, I don’t think so. I’m pretty sure I’ve seen Thread.Sleep methods in an EventHandler (or Timer) effectively block other EventHandlers.

The processing is pretty minimal on the rx. (In my real code, I just shove the line onto a queue and process it on a different thread.)

I’m most interested in having somebody confirm if the software works on socket 8. Thank you for your interest. It has been quite informative about what is happening behind all the software.

That is correct. There is only one thread for events.

That is why, in situations with multiple ports and/or high data rates, it is not advisable to process the received data in the event handler. Instead, the data should be placed into a queue for processing by another thread(s).

A sleep should never be used in an event handler. It will cause all sorts of undesirable things to happen. :slight_smile:

1 Like

Hi Mike, what kind of queuing do you use? The queue class of NETMF or a self-coded buffer? Perhaps you have a link to a field-tested code example?
Regards
Roland

@ RoSchmi - I do not use the queue class. I use custom linked lists.

I define a small data buffer class with a “next” field. at startup time, I create a number of these buffers and chain them in a “free” list. for the queue, I maintain a head and tail reference.

preallocation of the buffers and maintaining a free buffer queue avoid GC issues.

I add a data ready autoreset event for the queue, and an object to lock on.

in the dataReceived event, I lock, get a free buffer, and put the available data into the buffer(s). I then set the event.

in a process thread, I wait on the event, extract a buffer from the queue, process it, and return the buffer to the free queue. of course, locking at the appropriate times.

should I run out of free buffers, I allocate additional buffers.

using buffer objects allows them to be shared among multiple serial ports.

I usually incapsulate all this logic in a queue class, with the free buffers stored as class variables.

this type of processing is called the “producer consumer pattern”.

in simple situations, I sometimes use a circular buffer for each port.

@ Mike -
Aaah…, knew that you have some jewels in your toolbox! As andre Id be grateful to see the code and to have a tutorial.
Thanks Roland

I have no generic code for queuing. I usually use queuing when performance is required, and each situation usually has special needs. Here is some code from a project under development which uses the type of queue I described.

The application reads 12 byte messages. The RF Pipe, which is used, handles 12 byte packets atomically. But there is a a race condition that occurs when the program is first started, which will result in loss of message sync. I need to fix this. This info is not necessary to understand the queue mechanism, but you will have to determine what is the granularity of the queue entry data portion.

There is some stuff that I have not included, but it is easy to see where the queuing and processing is being done.

Justin’s RF Pipe module is used with this code. The pipe is half duplex, with receiving having priority over send.

By use of the proper pre-compile flags, the code compiles on MF or full .NET.

#if MF_FRAMEWORK_VERSION_V4_2
using System;
using Microsoft.SPOT;
using System.Collections;
using System.Threading;
using System.IO.Ports;
 #else
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.IO.Ports;
using System.Threading;
 #endif

namespace MANETMF_Library
{
    public class SerialCommunications : CommunicationsInterface
    {
        private class QueueEntry
        {
            public byte[] message;
            public QueueEntry next;
        }

        private SerialPort serialPort = null;

        private AutoResetEvent readyEvent = new AutoResetEvent(false);
        private QueueEntry queue = null;
        private QueueEntry lastInQueue = null;
        private QueueEntry freeQueue = null;
        private Thread processThread = null;

        public NodeManager Node { get; set; }

        private LogBase log = null;

        public SerialCommunications(LogBase log, int initialFreeEntryCount = 10)
        {
            this.log = log;
            for (int i = 0; i < initialFreeEntryCount; i++)
            {
                QueueEntry entry = new QueueEntry();
                entry.message = new byte[12];
                entry.next = freeQueue;
                freeQueue = entry;
            }
        }

        public void Start(SerialPort serialPort)
        {
            this.serialPort = serialPort;
            serialPort.DataReceived += new SerialDataReceivedEventHandler(serialPort_DataReceived);

            processThread = new Thread(MessageProcessingThread);
            processThread.Start();
        }

        private QueueEntry assembly = null;
        private int nextAssemblyIndex = 0;
        private bool receiving = false;
        private bool sending = false;

        private enum SeizePortModes { Send, Receive };
        private void SeizePort(SeizePortModes mode)
        {
            while (true)
            {
                lock (this)
                {
                    switch (mode)
                    {
                        case SeizePortModes.Receive:
                            if (receiving)
                                return;
                            if (!sending)
                            {
                                receiving = true;
                                return;
                            }
                            break;
                        case SeizePortModes.Send:
                            if (sending)
                                return;
                            if (!receiving)
                            {
                                sending = true;
                                return;
                            }
                            break;
                    }
                }
                Thread.Sleep(20);
            }
        }

        void serialPort_DataReceived(object sender, SerialDataReceivedEventArgs e)
        {
            SeizePort(SeizePortModes.Receive);

            while (serialPort.BytesToRead > 0)
            {
                // make sure we have an assembly buffer
                if (assembly == null)
                {
                    lock (this)
                    {
                        if (freeQueue == null)
                        {
                            assembly = new QueueEntry();
                            assembly.message = new byte[12];
                        }
                        else
                        {
                            assembly = freeQueue;
                            freeQueue = assembly.next;
                        }
                    }
                    nextAssemblyIndex = 0;
                }


                // determine how many byes we need to read to fill the current buffer
                int remainBytesToPacketleet = 12 - nextAssemblyIndex;
                int bytesToRead = (remainBytesToPacketleet < serialPort.BytesToRead) ? remainBytesToPacketleet : serialPort.BytesToRead;
                int bytesRead = serialPort.Read(assembly.message, nextAssemblyIndex, bytesToRead);
                log.Write("Bytes read - " + bytesRead.ToString());
                nextAssemblyIndex += bytesToRead;

                if (nextAssemblyIndex == 12)
                {
                    assembly.next = null;
                    lock (this)
                    {
                        if (queue == null)
                        {
                            queue = assembly;
                        }
                        else
                        {
                            lastInQueue.next = assembly;       

                        }
                        lastInQueue = assembly;
                    }

                    readyEvent.Set();
                    assembly = null;
                    nextAssemblyIndex = 0;
                }
            }
            receiving = (nextAssemblyIndex != 0);
        }

        private void MessageProcessingThread()
        { 
            while (true)
            {
                QueueEntry entry = null;
                lock (this)
                {
                    if (queue != null)
                    {
                        entry = queue;
                        queue = queue.next;
                        if (queue == null)
                            lastInQueue = null;
                    }
                }

                if (entry != null)
                {
                    // parse message
                    MessageBase msg = MessageBase.Parse(entry.message, 0);

                    // tell node to process message
                    Node.ProcessMessage(msg);

                    /// return to free queue
                    lock (this)
                    {
                        entry.next = freeQueue;
                        freeQueue = entry;
                    }

                    continue;
                }

                readyEvent.WaitOne();
            }
        }

        public void SendMessage(MessageBase message)
        {
            SeizePort(SeizePortModes.Send);

            serialPort.Write(message.GetBytes(), 0, 12);
            log.Write("Packetlette sent");
            sending = false;
        }

    }
}
1 Like

@ Mike - thanks for spending your time.
Roland

Update: John figured out why Socket 8 was not working!
His fix works for me.