Data Corruption with rs232 module

In our device we are trying to mimic data quickly and safely from the rs232 port on our device to the s232 port of our device (a loop back). However in with such a simple task we find our responses lackluster at best: we’re getting a reasonably significant amount of corruption.
Example Output looping the simple line “I am a test string\r”:

keep in mind that it’s our device both sending and receiving these strings so there’s no reason for there to be any data loss!

the relevant code:

void ProgramStarted()
        {          
            Debug.Print("Program Started");
           
            GT.Timer SendMessage = new GT.Timer(0, GT.Timer.BehaviorType.RunOnce);
            SendMessage.Tick += SendMessage_Tick;
            SendMessage.Start();
        }

        void SendMessage_Tick(GT.Timer timer)
        {
            rs232.Initialize(19200);
            rs232.serialPort.SynchronousUnsafeEventInvocation = true;
            rs232.serialPort.DataReceived += serialPort_DataReceived;
            while (1 == 1)
            {
                rs232.serialPort.WriteLine("I am a test string\r");
            }
        }

void serialPort_DataReceived(GT.Interfaces.Serial sender, System.IO.Ports.SerialData data)
        {
            var toRead = sender.BytesToRead;
            var v = new byte[toRead];
            sender.Read(v, 0, toRead);
            for (int i = 0; i < v.Length; i++)
            {

                if (v[i] == 13)
                {
                    //sender.WriteLine(output);
                    writeStream(output);
                    //writeStream("Serial," + output);
                    Debug.Print(output);
                    output = "";
                    
                }
                else
                {
                    output += (char)v[i];
                }
            }

        }

Hi,
I didn’t analyse your code in detail, but before doing this I would suspect that perhaps the Debug.Print command shows corrupted data that are not really corrupted. I saw that Debug.Print() sometimes eats data. Try to compare the sent strings with the received strings and Debug.Print a message if they are not equal.

@ RoSchmi - well the problem appears when we are logging data to the SD card as well. The “project” given here was just to eliminate all other factors and to produce the issue in simplistic terms; we’ve also tried testing the port by reading and mimicking the read lines to the pc with the same results. The problem appears to be in the frequency of the data being sent: slow streams (1 message per second for example) do not seem to cause any corruption, but ramping up that speed seems to bring these issues to forward. We originally diagnosed this as a latency issue but after considerable changes to the design we’ve reduced the latency to <1 microsecond. However there’s still corruption so were a little at a loss here.

try to lock the output variable, I think it’s not Threadsave

@ VB-Daniel - this may still not e threadsafe; but we saw the same behavior with line received in our previous tests with the unsafe property set to false which someposedly protects us from unsafe threading

@ jstone05 - According to MSDN, the “unsafe” property will only allow (true) the use of pointers or deny (false) them.

For thread safe applications you will need to make use of a lock:


lock(object)
{
    ...
}

Because it is not threadsafe, serial events can happen while you are still processing the event.
Also, sender.Read returns the actual number of bytes read (which could be different than ‘toRead’, especially with the non-threadsafeness)
Try something like this:

        bool stillReading = false;
        void serialPort_DataReceived(GT.Interfaces.Serial sender, System.IO.Ports.SerialData data)
        {
            if (stillReading) return;
            stillReading = true;
            var toRead = 0;
            while ((toRead = sender.BytesToRead) > 0)
            {
                var v = new byte[toRead];
                int didRead = sender.Read(v, 0, toRead);
                for (int i = 0; i < didRead; i++)
                {
                    if (v[i] == 13)
                    {
                        //sender.WriteLine(output);
                        writeStream(output);
                        //writeStream("Serial," + output);
                        Debug.Print(output);
                        output = "";

                    }
                    else
                    {
                        output += (char)v[i];
                    }
                }
            }
            stillReading = false;
        }  

There is still a very narrow window where we know there are no bytes to read and we have not reset ‘stillReading’, but it should get called again soon.

If you not using RTS/CTS (Hardware flow control) then your just over running the receiver.

but i think you get an buffer overrun error before losing data

@ James - Sorry for being unclear, but I was referring to the rs232.serialPort.SynchronousUnsafeEventInvocation property, should have known unsafe was it’s own thing. It was my understanding (It seemed implied) that setting this to false was intended to prevent event threading problems, if that’s not the case then what was this property meant to do?

@ eolson - Implemented this code: corruption still present when using a separate transmitter; It did clear the loopback test though. (we’ve tested the transmitter extensively It shows no signs of corruption with a receiving PC).

This is the current code in use now, data corruption persists

void serialPort_DataReceived(GT.Interfaces.Serial sender, System.IO.Ports.SerialData data)
        {
            if (IsReading232) return;
            IsReading232 = true;
            var toRead = 0;  
            while ((toRead = sender.BytesToRead) > 0)             
            {
                var v = new byte[toRead];
                lock (v)
                {
                    lock (output)
                    {
                        int didRead = sender.Read(v, 0, toRead);
                        for (int i = 0; i < didRead; i++)
                        {
                            if (v[i] == 13)
                            {
                                //sender.WriteLine(output);                         
                                //writeStream(output);                         
                                writeStream("Serial," + output);
                                Debug.Print(output);
                                output = "";
                            }
                            else
                            {
                                output += (char)v[i];
                            }
                        }
                    }
                }
            }
            IsReading232 = false; 
        }

Is there any change to the amount of corruption with this small change:

void serialPort_DataReceived(GT.Interfaces.Serial sender, System.IO.Ports.SerialData data)
        {
            if (IsReading232) return;
            IsReading232 = true;
            var toRead = 0;  
            while ((toRead = sender.BytesToRead) > 0)             
            {
                var v = new byte[toRead];
                lock (v)
                {
                    lock (output)
                    {
                        int didRead = sender.Read(v, 0, toRead);
                        for (int i = 0; i < didRead; i++)
                        {
                             while (writeStream.BytesToWrite > 0)
                                 Thread.Sleep(10);

                            if (v[i] == 13)
                            {
                                //sender.WriteLine(output);                         
                                //writeStream(output);                         
                                writeStream("Serial," + output);
                                Debug.Print(output);
                                output = "";
                            }
                            else
                            {
                                output += (char)v[i];
                            }
                        }
                    }
                }
            }
            IsReading232 = false; 
        }

As well as for this question, here is the Microsoft response to that property:

[quote=“Microsoft”]
An interrupt may occur on a thread other than the application thread. When SynchronousUnsafeEventInvocation is false (the default), the Interrupt event, which is raised in response to the interrupt on the interface, is not raised immediately, instead it is queued for raising on the application’s dispatcher thread. However, when SynchronousUnsafeEventInvocation is true, the Interrupt event is raised immediately on the same thread that generated the interrupt. This results in faster interrupt processing and may be useful to respond to realtime events, but extra care must be taken when using this facility to be thread-safe, i.e. to handle issues such as locking, atomic reading/writing of streams/files, deadlock avoidance, etc.[/quote]

@ James - writeSteam is a method i defined, so it has no bytestowrite property… However it does have a flag that indicates it’s currently writing so I replaced it with that

that does seem to reduce the corruption, but it does not remove it completely

I’ve included the writeStream method for clarity
note that this corruption is appearing in the debugger as well, despite this methods possible shortcomings

 /// <summary>
        /// writes the passed string to the SD card
        /// </summary>
        /// <param name="data">data to be written to SD card</param>
        void writeStream(string data)
        {            
            writing = true;
            if (!error)
            {
                lock (stream)
                {
                    stream.WriteLine(milliseconds + "," + data);
                    stream.Flush();
                    VInfo.FlushAll();
                }
            }
            writing = false;
        }

@ jstone05 - Is this class writing to the SD card or sending it out to be written elsewhere?

@ James - It’s Writing to the SD card, but not atm; I’m observing corruption through the debugger, and despite warnings against it, I’ve noticed no data inconsistancy between debugger and actual logged files

Hi jstone05,
I got some time to play around with your serial port issue and made a modified version that led to not corrupted data on a Cerbuino Bee mainboard.


using System;
using System.Collections;
using System.Threading;
using Microsoft.SPOT;
using Microsoft.SPOT.Presentation;
using Microsoft.SPOT.Presentation.Controls;
using Microsoft.SPOT.Presentation.Media;
using Microsoft.SPOT.Presentation.Shapes;
using Microsoft.SPOT.Touch;
using Gadgeteer.Networking;
using GT = Gadgeteer;
using GTM = Gadgeteer.Modules;
using Gadgeteer.Modules.GHIElectronics;
using Serial = Gadgeteer.Interfaces.Serial;


namespace jStone05SerialPortIssue
{
    public partial class Program
    {
        Thread writerThread;
        SerialBuffer mySerialBuffer;
        string dataLine = "";
        int lineNo = 1;
        
        void ProgramStarted()
        {
            Debug.Print("Program Started");
            GT.Timer SendMessage = new GT.Timer(1000, GT.Timer.BehaviorType.RunOnce);
            SendMessage.Tick +=SendMessage_Tick;
            try
            {
                rs232.Initialize(19200, Serial.SerialParity.None, Serial.SerialStopBits.One, 8, Serial.HardwareFlowControl.NotRequired);
            }
            catch (Exception e)
            {
                Debug.Print(e.ToString());
            }
            if (!rs232.serialPort.IsOpen)
            {
                rs232.serialPort.Open();
            }
            rs232.serialPort.DataReceived +=serialPort_DataReceived;
            // End Initialize RS232 Input
            mySerialBuffer = new SerialBuffer(256);
            SendMessage.Start();
        }


        void SendMessage_Tick(GT.Timer timer)
        {
            Debug.Print("SendMessage reached");
            writerThread = new Thread(new ThreadStart(runWriterThread));
            writerThread.Start();
        }

        void runWriterThread()
        {
            while (true)
            {
                rs232.serialPort.WriteLine("I am a test string");
                Thread.Sleep(100);
            }
        }

        void serialPort_DataReceived(GT.Interfaces.Serial sender, System.IO.Ports.SerialData data)
        {
            var toRead = sender.BytesToRead;
            byte[] v = new byte[toRead];
            sender.Read(v, 0, toRead);
            mySerialBuffer.LoadSerial(v, 0, v.Length);
            while ((dataLine = mySerialBuffer.ReadLine()) != null)
            {
                Debug.Print("Print line No. " + lineNo + ": "+ dataLine);
                lineNo++;
            }
        }
    }
}
 

I changed the following parts of your code:

  1. Initialized the serial port in Program.Started.
  2. I made a second thread to write the strings to the serial port.
  3. In the event handler I only read once from the serial port and stored the incoming
    data in a buffer (SerialBuffer) from where they were read back after a hex 0x0A was
    received.
  4. You set your GT.Timer SendMessage timer to fire after 0 msec, I set it to 1000.

I don`t know if all the points are important but I think that points 2 and 3 are important.
You can find the SerialBuffer Class in my CodeShare Project.
FEZ/PC Bluetooth File Transfer Server/Client
https://www.ghielectronics.com/community/codeshare/entry/823
(the namespace must be changed to the actual project)
Cheers
Roland

1 Like

@ RoSchmi - Wow, thanks a ton for all the work put in.

First off this seems to be working well, I saw a little corruption at the start, but that’s not a huge deal (1/50000 lines: likely because of a breakpoint)

However the incoming data rate is the heart of this problem: so I’m not surprised you saw no data corruption with a slow write rate

I’ll do a whole mess of testing and report back with the results soon

So, adding the SD card into this equation re-implimented the data corruption, I suspect the “smoking gun” here is the speed at which I can process these messages, so I moved the SD writing to a separate thread and had the read-data event toss it’s string into a Queue; doing more testing of course, but this seems to be working (long-term high speed transfers would likely bust this Queue, but We’re more concerned about a “burst” of writes, as opposed to a constant long-term stream of high-speed messages