Fast serial communication makes our ChipworkX crash!

Hi.

We are experimenting a strange and weird effect on our latest system.

Here is the scenario: in one of subsystems of whole machine, we have a couple of boards (a Fez-Panda2 and a ChipworkX) that work together to manage signals coming from two IR-fences set at both sides of a tunnel inside which a conveyor takes items through. Since each ir-fence is composed by many emitter/receiver couples (64), we designed it to let us address 8-couples banks at a time, reducing input ports count needed to process incoming signals. This way, indeed, we have to poll ir-couples instead of managing interrupts (we have to switch active bank and so on).

Without recurring to RLP, we have obtained a decent “frame rate” of about 50 fps per fence, but we had to reserve a board (a fez-panda2) to do continuous polling of fences. This board talks via serial port with a ChipworkX that runs main firmware, that processes incoming serial messages about fence “engaging” state variation (fence1 on, fence 1 off, fence 2 on, fence 2 off).

We checked with a serial usb bridge and teraterm messages coming from fez-panda2 and all works well, even with 50/60 fence variations per second (moving 2 wires very fast along both led strips).

At ChipworkX side, we process incoming messages using the class you can find below. Messages are delimited by STX (0x2) and ETX (0x3) bytes.

The strange and bad behavior we are experimenting is that triggering 50/60 fence variations per second (i.e. sending 50/60 messages per second) hangs ChipworkX firmware in a way we have to reboot it (even VS debugger is detached by running firmware).

We tried to avoid to process too many messages at ChipworkX side discarding them if too little time has elapsed since previous messages, but without success. Maybe it depends on low-level incoming data buffering?

Thanks!


using System;
using Microsoft.SPOT;
using System.IO.Ports;
using System.Text;

namespace SerialCommunication
{

    public class SerialCommunicationManager
    {
        const int MAX_LEN = 256;
        const int READING_MIN_INTERVAL_TICKS = 0;

        byte START_BYTE = 0x02;
        byte END_BYTE = 0x03;

        byte[] _data;
        SerialPort _com;
        int _datapos = 0;
        int _codeStartPos = 0;
        int _codeEndPos = 0;
        long _latestTicks;

        public event MessageReceivedEventHandler MessageReceived;

        public SerialCommunicationManager(string ComPort, int BaudRate)
        {
            _data = new byte[MAX_LEN];
            _com = new SerialPort(ComPort, BaudRate);
        }

        public void SendMessage(string message)
        {
            var msgbytes = Encoding.UTF8.GetBytes(message);
            var totMsg = new byte[msgbytes.Length + 2];
            totMsg[0] = START_BYTE;
            msgbytes.CopyTo(totMsg, 1);
            totMsg[totMsg.Length - 1] = END_BYTE;

            _com.Write(totMsg, 0, totMsg.Length);

        }

        public void Start()
        {
            _com.Open();

            _latestTicks = DateTime.Now.Ticks;

            _com.DataReceived += new SerialDataReceivedEventHandler(com_DataReceived);
        }

        public void Stop()
        {
            _com.Close();
            _com.Dispose();
        }

        private void processData()
        {
            int pos = -1;

            if (_codeStartPos == 0)
            {
                pos = FirstIndexOf(_data, START_BYTE);

                if (pos == -1) return;

                _codeStartPos = pos;
            }

            pos = FirstIndexOf(_data, END_BYTE);

            if (pos != -1)
            {
                _codeEndPos = pos;

                try
                {
                    raiseEvent();
                }
                catch { }

                resetData();
            }
        }


        private int FirstIndexOf(byte[] buffer, byte pattern0)
        {
            for (int i = 0; i < buffer.Length; i++)
            {
                if (buffer[i] == pattern0)
                {
                    return i;
                }
            }

            return -1;
        }

        private void resetData()
        {
            for (int i = 0; i < _data.Length; i++)
            {
                _data[i] = 0;
            }

            _datapos = 0;
            _codeStartPos = 0;

            int n = _com.BytesToRead;
            byte[] dummy = new byte[n];
            _com.Read(dummy, 0, n);
        }

        private void raiseEvent()
        {
            _latestTicks = DateTime.Now.Ticks;

            if (MessageReceived != null)
            {
                byte[] code = new byte[_codeEndPos - _codeStartPos - 1];

                Array.Copy(_data, _codeStartPos + 1, code, 0, code.Length);

                string stringcode = new string(Encoding.UTF8.GetChars(code));

                MessageReceived(this, new MessageReceivedEventArgs(stringcode));
            }
        }

        void com_DataReceived(object sender, SerialDataReceivedEventArgs e)
        {
            if (e.EventType != SerialData.Chars) return;

            try
            {
                processComStream();
            }
            catch (Exception ex)
            {
                Debug.Print(ex.Message);
            }
        }

        private void processComStream()
        {
            int n = _com.BytesToRead;

            if (_datapos + n < _data.Length &&
                DateTime.Now.Ticks - _latestTicks > READING_MIN_INTERVAL_TICKS)
            {
                _com.Read(_data, _datapos, n);
                _datapos += n;

                processData();

            }
            else
            {
                byte[] dummy = new byte[n];
                _com.Read(dummy, 0, n);

                resetData();
            }
        }

        string decode(byte[] arr)
        {
            return new string(Encoding.UTF8.GetChars(arr));
        }

    }

    public class MessageReceivedEventArgs : EventArgs
    {
        string _msg;

        public string Message
        {
            get { return _msg; }
        }

        public MessageReceivedEventArgs(string msg)
        {
            _msg = msg;
        }
    }

    public delegate void MessageReceivedEventHandler(object s, MessageReceivedEventArgs e);
}

 


All looks fine except this maybe your bottleneck. I would implement this in RLP. Another option is to change your protocol where panda sends a frame when chipworkx sends a “start now” byte?

private int FirstIndexOf(byte[] buffer, byte pattern0)
        {
            for (int i = 0; i < buffer.Length; i++)
            {
                if (buffer[i] == pattern0)
                {
                    return i;
                }
            }
 
            return -1;
        }

Or use Array.IndexOf

Aha! even better, indexof.

Ok, thanks, I’m going to try in short time.

Actually, our naive IndexOf implementation is due to a previous (obsolete) protocol version in which starting and ending message patterns were made of several bytes instead of just one. Method
hasn’t been refactored very well :-[

Concerning suggested solution, does it mean that if our managed firmware has an open serial port and didn’t succeed to process rx buffer at least at tx transmit rate it is doomed to crash (despite any managed exception handling)?

I tried to modify our code as suggested, but without luck… :’(

Any other suggestions or any answer to my question about tx rate higher than rx one?

Thanks a lot!