Com Trouble. Need help ASAP. Please…

NETMF & Gadgeteer
1

Iam new to micro framework and I have been ask to design a high speed serial logger. So I start with panda II. When receiving I am missing some of the RX data. My data packet is 92 bytes long and I am getting 32 packets in 1 second and I have 3 of these channels coming to the system at same time.

My simple code as follows

       public static void Comm2_Thread()
        {
            SerialPort UART = new SerialPort("COM4", 57600);
            byte[] rx_byte = new byte[2000];
            int read_count = 0;
            UART.Open();

            while (true)
            {
                Debug.GC(false);
                // Thread.Sleep(1000);
               
                read_count = UART.Read(rx_byte, 0, 2000);
                string pk1 = "";

                if (read_count > 0)// do we have data?
                {
                    int xx_cou = 0;
                    while (xx_cou < read_count)
                    {




                        try
                        {
                            byte[] a1 = new byte[1];
                            a1[0] = rx_byte[xx_cou];
                            string a2 = new     string((Encoding.UTF8.GetChars(a1)));

                            pk1 = pk1 + a2;
                        }
                        catch { }
       


                        xx_cou = xx_cou + 1;
                    }



                }
                Debug.Print("--------------------------------------------------------------------");
                Debug.Print("COM4  " + pk1);
            }  
        }

I am getting overlap of packets in 256th to ~ 266th byte, but next 1744 is perfect. My first thought was my TX data stream too fast and I set to transmit only 2 packets per second but error is still there. Then I try to get only first 200 bytes at a time but in some times I get few data missing in the middle. I know at the moment I am not synchronize the incoming data but it should not be a problem when I am interested in reading the buffer.

Sample of incoming data
0.07798058_A,000,001.09,-001.70,M,+344.22,+021.01,00,+0.0001,+0.0002,+0.0004,+0.0004,-50.00C,
Packet overlapping due to error
0.1786935_Y,360,236.25,-025.23,M,+F,090,002.10,+005.67,M,-345.21,+021.01,02,+0.0001,+0.0002,+0.0004,+0.0004,-45.21C,

Am I expecting too much from Panda II or I am missing some thing?
I really need a solution quickly
Thanks

2

Please use code tags

You should be to handle your data just fine

3

You create a string and a byte for each byte received and you expect it to run fast? Receive full packets and convert each packet in one conversion. Avoid ‘new’ keyword in a while loop.

4

Hi Wouter Huysentruit
Thanks for you suggestion. I am new to C# and it is true it will take long time convert them in to string this way. But according to my understanding after the line byte[] rx_byte = new byte[2000]; it has finish reading data from UART any delays due to my bad coding after that should not effect the data overlap in middle of the array. IS that correct or ‘new’ keyword in a while loop some now affecting the UART reading process?

Receive full packet at once is a good idea, Is there any method of reading data one packet at time I try DataReceived event method but could not get to read it from start of packet and stop at end of the packet. Is there a way to tell the buffer to start receiving at certain character and stop at certain character?

Thanks
niran
Re: sorry for my English.

5

Hi Gus
Support

My apologizes for posting many times. First I did not know what is the best place ask a question. I am also sorry for putting ASAP in the heading. Clearly GHI answer soon as they can.

Thanks
Niran

6

You must add code to recognize the beginning and end of your data packet.

There is not any thing in the framework to automatically do it for you.

I suspect you are having problems because you are not explicitly handling packets.

7

What is code tags?

8

it’s the second button in the toolbar (the button with the ones and zeros).
Just click modify to edit your post, select all your code and click on the code tag, and click on edit reply.

9

Hi Mike

Thanks. I can see the data loss with out having to break my big array in to packets.
Earlier I wrote something to break it in to packets.

public static void Comm1_Thread()
        {
            SerialPort UART = new SerialPort("COM3", 115200);
            byte[] rx_byte = new byte[2000];
            int read_count = 0;
            UART.Open();
            string end_offset = "";
            string st_offset = "";

            while (true)
            {
                Debug.GC(false);
               // Thread.Sleep(1000);
               // Debug.Print(UART.BytesToRead.ToString());
                read_count = UART.Read(rx_byte, 0, 2000);
                string pk1 = "";
              
                if (read_count > 0)// do we have data?
                {
                    int xx_cou = 0;
                    bool PK_reed_on =true;
                  //  string comp_from_offset = "";
                     bool st_offset_TK_on = true;
                    
                    while (xx_cou < read_count)
                    {

             

                        if (rx_byte[xx_cou] == 0x02)//to find header
                        {
                            PK_reed_on = true;
                           

                           // Debug.Print("start found at " + xx_cou );

                        }


                        if (rx_byte[xx_cou] == 0x03)//  to find footer
                        {

                            if (st_offset_TK_on == true)
                            {
                               
                             //   Debug.Print("offset pk = " + comp_from_offset);
                                st_offset_TK_on = false;
                            }

                            PK_reed_on =false;
                            save_data_scheduler("COM3," + pk1);
                         Debug.Print(pk1 + " pks " );
                            pk1 = "";
                            end_offset = "";
                            //Debug.Print(xx_cou + " end found " + temp_old.ToString());
                      

                        }

                        if (PK_reed_on == true)
                        {

                            try
                            {
                                byte[] a1 = new byte[1];/////really really inefficient do something later
                                a1[0] = rx_byte[xx_cou];
                                string a2 = new string((Encoding.UTF8.GetChars(a1)));

                                pk1 = pk1 + a2;
                                end_offset = end_offset + a2;

                            }
                            catch { Debug.Print("error " + xx_cou); }
                            
                        }



                        xx_cou=xx_cou +1;
                    }

This was my Output window

its 256th to ~ 266th byte, but next 1744 is perfect.
10

Try this concept: (I say concept, because I didn’t test this code, it just shows the way how I would try to solve this problem)


using (SerialPort serialPort = new SerialPort("COM3", 115200))
{
    const byte PACKET_START = 0x02;
    const byte PACKET_END = 0x03;
    const int MAX_PACKET_LENGTH = 200;

    byte[] readBuffer = new byte[2000];
    char[] charBuffer = new char[MAX_PACKET_LENGTH];
    int readBufferInUse = 0;
    serialPort.Open();

    while (!terminated.WaitOne(10, false))
    {
        while (serialPort.BytesToRead > 0)
        {
            int bytesRead = serialPort.Read(readBuffer, readBufferInUse, readBuffer.Length - readBufferInUse);

            if (bytesRead > 0)
            {
                readBufferInUse += bytesRead;

                // Extract as many packets from the buffer as we can
                int startOffset = -1;
                int endOffset = -1;
                for (int i = 0; i < readBufferInUse; i++)
                {
                    if (readBuffer[i] == PACKET_START)
                        startOffset = i;

                    if (readBuffer[i] == PACKET_END)
                    {
                        // Packet end found
                        endOffset = i;
                                    
                        // See if we have a valid packet start
                        if (startOffset >= 0)
                        {
                            // Complete packet found between startOffset and endOffset
                            int packetLength = endOffset - startOffset - 1;

                            if ((packetLength > 0) && (packetLength <= MAX_PACKET_LENGTH))
                            {
                                // Don't know if this trick works here (copy byte[] to char[]) (byte and char are both 8 bit values in NETMF)
                                // Otherwise we need to create a new array since Encoding.UTF8.GetChars doesn't take offset and length.
                                Array.Copy(readBuffer, startOffset + 1, charBuffer, 0, packetLength);
                                string packetText = new string(charBuffer, 0, packetLength);
                                Debug.Print(packetText);
                            }

                            startOffset = -1;
                        }
                    }
                }

                // Throw away processed data from the readBuffer
                if (endOffset >= 0)
                {
                    // Calculate remaining bytes in readBuffer
                    readBufferInUse -= endOffset;
                    readBufferInUse--;

                    // If one or more bytes remaining, move them to beginning of buffer
                    if (readBufferInUse > 0)
                        Array.Copy(readBuffer, endOffset + 1, readBuffer, 0, readBufferInUse);
                }
            }
        }
    }
}

And of course, you have defined a global manualresetevent to terminate your thread:


ManualResetEvent terminated = new ManualResetEvent(false);
11

Hi Wouter Huysentruit

Thanks for your time and effort. Excellent piece of code. I like the way you move remaining bytes to the beginning of the buffer. I still did have time to run it on the hardware. I run on the first chance I get and post the result here stay tune.
If array.copy doesnt work do I have use another loop of packet length(96) science Encoding.UTF8.GetChars take only one byte at time

Thanks
niran

12

Where do you get that quote from ???

GetChars takes a byte array.

http://msdn.microsoft.com/en-us/library/khac988h(v=VS.100).aspx

13

Hi Wouter Huysentruit

I ran your code, I have to change array.copy to Encoding.UTF8.GetChars
Have a look. It certainly improve packet lost but it still there now I am getting random lost (overlap) between every 64th and 120th packets. Also is there any other way to convert byte array to string?

Thanks again for your support.

        public static void Comm1_Thread()
        {
            SerialPort serialPort = new SerialPort("COM4", 57600);
            const byte PACKET_START = 0x02;
            const byte PACKET_END = 0x03;
            const int MAX_PACKET_LENGTH = 200;

            byte[] readBuffer = new byte[2000];
            byte[] a1 = new byte[1];
            char  a2;
            string packetText = "";
            char[] charBuffer = new char[MAX_PACKET_LENGTH];
            int readBufferInUse = 0;
            serialPort.Open();
            while (serialPort.BytesToRead > 0)
            {
                int bytesRead = serialPort.Read(readBuffer, readBufferInUse, readBuffer.Length - readBufferInUse);
               
                if (bytesRead > 0)
                {
                     //Debug.Print(bytesRead.ToString());
                    readBufferInUse += bytesRead;

                    // Extract as many packets from the buffer as we can
                    int startOffset = -1;
                    int endOffset = -1;
                    for (int i = 0; i < readBufferInUse; i++)
                    {
                        if (readBuffer[i] == PACKET_START)
                            startOffset = i;

                        if (readBuffer[i] == PACKET_END)
                        {
                            // Packet end found
                            endOffset = i;

                            // See if we have a valid packet start
                            if (startOffset >= 0)
                            {
                                // Complete packet found between startOffset and endOffset
                                int packetLength = endOffset - startOffset - 1;

                            
                                    // Don't know if this trick works here (copy byte[] to char[]) (byte and char are both 8 bit values in NETMF)
                                    // Otherwise we need to create a new array since Encoding.UTF8.GetChars doesn't take offset and length.
                                   // Array.Copy(readBuffer, startOffset + 1, charBuffer, 0, packetLength);
                                  

                                    packetText = "";
                                    for (int i2 = 0; i2 < packetLength; i2++)
                                    {
                                        //Debug.Print(readBuffer[startOffset+i2].ToString());
                                        a1[0]=readBuffer[startOffset+i2];
                                        packetText = packetText + new string(System.Text.UTF8Encoding.UTF8.GetChars(a1));
                                    }
                              
                                //string packetText = new string(charBuffer, 0, packetLength);
                                Debug.Print("newtest" + packetText);


                                startOffset = -1;
                            }
                        }
                    }

                    // Throw away processed data from the readBuffer
                    if (endOffset >= 0)
                    {
                        // Calculate remaining bytes in readBuffer
                        readBufferInUse -= endOffset;
                        readBufferInUse--;

                        // If one or more bytes remaining, move them to beginning of buffer
                        if (readBufferInUse > 0)
                            Array.Copy(readBuffer, endOffset + 1, readBuffer, 0, readBufferInUse);
                    }
                }
            }

        }
14

Hi EriSan500 (Eric)
Yes but I am not running .NET Framework 4. I using micro framework.
If there is any other method for Get Chars from a byte array (with out using a loop) please let me know.

Thanks
niran :-[

15

Niran, it’s the same in .Net MF 4.1: [url]Microsoft Learn: Build skills that open doors in your career

May I suggest to invest some time in going through the SDK’s.

16

HI EriSan500 (Eric)
Does it take offset, I need UTF8Encoding.UTF8.GetChars(bytearay as array,offset as int ,length as int);?
Thanks
niran

17

Did you actually looked at the page??

18

Why always use that for loop? Just convert all chars at once…


public static void Comm1_Thread()
        {
            SerialPort serialPort = new SerialPort("COM4", 57600);
            const byte PACKET_START = 0x02;
            const byte PACKET_END = 0x03;
            const int MAX_PACKET_LENGTH = 200;
 
            byte[] readBuffer = new byte[2000];
            byte[] a1 = new byte[1];
            char  a2;
            string packetText = "";

     byte[] packetBuffer = new byte[MAX_PACKET_LENGTH];

            int readBufferInUse = 0;
            serialPort.Open();
            while (serialPort.BytesToRead > 0)
            {
                int bytesRead = serialPort.Read(readBuffer, readBufferInUse, readBuffer.Length - readBufferInUse);
 
                if (bytesRead > 0)
                {
                     //Debug.Print(bytesRead.ToString());
                    readBufferInUse += bytesRead;
 
                    // Extract as many packets from the buffer as we can
                    int startOffset = -1;
                    int endOffset = -1;
                    for (int i = 0; i < readBufferInUse; i++)
                    {
                        if (readBuffer[i] == PACKET_START)
                            startOffset = i;
 
                        if (readBuffer[i] == PACKET_END)
                        {
                            // Packet end found
                            endOffset = i;
 
                            // See if we have a valid packet start
                            if (startOffset >= 0)
                            {
                                // Complete packet found between startOffset and endOffset
                                int packetLength = endOffset - startOffset - 1;
 
 
                                    // Don't know if this trick works here (copy byte[] to char[]) (byte and char are both 8 bit values in NETMF)
                                    // Otherwise we need to create a new array since Encoding.UTF8.GetChars doesn't take offset and length.
                                   // Array.Copy(readBuffer, startOffset + 1, charBuffer, 0, packetLength);
 
if ((packetLength > 0) && (packetLength <= MAX_PACKET_LENGTH))
{
      Array.Copy(readBuffer, startOffset + 1, packetBuffer, 0, packetLength);
      packetText = new string(Encoding.UTF8.GetChars(packetBuffer), 0, packetLength);
 
                                Debug.Print("newtest" + packetText);
}
 
                                startOffset = -1;
                            }
                        }
                    }
 
                    // Throw away processed data from the readBuffer
                    if (endOffset >= 0)
                    {
                        // Calculate remaining bytes in readBuffer
                        readBufferInUse -= endOffset;
                        readBufferInUse--;
 
                        // If one or more bytes remaining, move them to beginning of buffer
                        if (readBufferInUse > 0)
                            Array.Copy(readBuffer, endOffset + 1, readBuffer, 0, readBufferInUse);
                    }
                }
            }
 
        }
19

So as I mentioned in my code, GetChars doesn’t take offset, that would have been to easy.
In the modifications I made above, you’ll see how I first copy the packet to a smaller buffer, to speed up the GetChars… adjust MAX_PACKET_LENGTH to the maximum length of your packets.

Also note that Debug.Print will slow things down.

20

Hi Wouter Huysentruit

Thanks again for you ongoing support. This is very interesting project for me and I am learning C# and micro framework as I try to fix this problem. So please be patient with my cording.
I ran your modified code again and the data overlapping error still there (see the picture). Now it improves every 30 packet to every 80 to 100 packet. At normal situations I say it acceptable but in this case our customer expect to get all the data with out a loss.

Do you think faster system will work better?
Thanks
Niran