Serial Data Buffering Question

[contractorwolf]

so i am using the bluetooth module that used to be available on this site and a laptop with a usb bluetooth module and a program that sends a continuous stream of data. the fez domino attached uses this code to “see” the data stream

static void uart_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
    rx_data = new byte[uart.BytesToRead];
    uart.Read(rx_data, 0, rx_data.Length);
    s = new String(System.Text.Encoding.UTF8.GetChars(rx_data));
         
    Debug.Print("[" + messageCount.ToString() + "] " + s);

    messageCount++;
}

the problem is that i am sending chunks of data from the laptop that look like this:
“128,128” (with the newline \n) added by calling the serialPort.,WriteLine(“128,128”), but instead of getting nice “[12345] 128,128\n” data, it looks more like this:

[142192] 831
[142193] 28
[142194] ,1
[142195] 83
[142196] 1
[142197] 28
[142198] ,1

i am pretty sure that even though i am calling serialPort.WriteLine(“larger string”); the actual radio is breaking up the “larger string” into something like this: “lar” “ger” " st" “ring”. this makes it hard for me because i cant think of an efficient way to attach the strings together without doing something stupid like this:

            
rx_data = new byte[uart.BytesToRead];
uart.Read(rx_data, 0, rx_data.Length);
s = new String(System.Text.Encoding.UTF8.GetChars(rx_data));
 //wrong below, because it eats memory
line = line + s;

if (line.IndexOf("\n") > 0)
{
    ParseBTLine(line);
    line = "";
}

which actually eats up memory quickly (because i think the adding of strings always creates a larger amount of memory). i would prefer is i could set the “chunk size”, so instead of it looking at “lar” “ger” it would grab "larger string " so that i could see an actual reading each time without having to compile all the chunks. anybody done anything like that?

appreciate any advice

[Gus_Issa]

You allocate a buffer every time there is a serial event, which is disposed too…this will very much slow everything down. Object allocation and collection is very expensive.

[Brett]

An event, by definition, won’t wait until you get the last character in any “string”. So I’d expect you’re seeing exactly what I would expect.

Remember that a sequence of data 128,128\n128,128\n is just a big long sequence of bytes, there’s nothing that “makes” it special except your interpretation of the \n.

If you want to make the code handle \n terminators then you need to move away from events, something like this:

       
public static string ReadLine(SerialPort port, int delim = 10, int maxBufSize = 300)
         //for the delimeter, default = \n, for \r use delim=13
         {
           if (maxBufSize <= 0) throw new ArgumentOutOfRangeException("maxBufSize");
            byte[] buf = new byte[maxBufSize];
            int curPos = 0;

            while (curPos < maxBufSize)
            {
                int read = port.Read(buf, curPos, 1);
                if (read == 0) break;
                if (buf[curPos] == delim) //this won't work for multiline responses since this flushes anything left unread
                {
                    port.Flush();
                    break;
                }
                curPos += read;
            }

            // curPos will be zero if first read is zero or first read is delim byte.
            if (curPos == 0) return null;
            if (curPos == maxBufSize)
                throw new InvalidOperationException("Line size > maxBufSize.");
            if (buf[curPos] != delim)
            {
                port.Flush();
                return null; // if no delim found, assume comms failure. drop the current reading and move on
            }

            buf[curPos] = 0;
            char[] ca = Encoding.UTF8.GetChars(buf);
            return new string(ca);
        }

should help… Thanks to William (IIRC) who supplied most of that previously.

[contractorwolf]

so what your saying is instead of trying to add the drops up to a sip i should wait a specified period of time and take a full sip?

[Brett]

no what I’m saying is if you WANT the full message, set yourself up to collect things until the full message arrives (or a timeout occurs).

If you can find a nice way to handle data in drips, then an event driven methodology can work (but you need to specifically handle catching those drips in your cup until you have a sip ).

You can’t guarantee that when an event is triggered you’ll have a full set of data in the buffer waiting to be read, so you need to specifically handle that scenario by keeping a buffer outside the event handler, and then take your sips from that. You will probably need a circular buffer to help out.

So you can do it the way you’re trying, but you can’t assume that one pass through the event handler will produce you one clean message. But the code I showed should “just work”.

[WouterH]

This is what I use as serial line receiver:

private byte[] inputBuffer = new byte[1024];
private int inputBufferUsed = 0;

/// <summary>
/// Thread that reads all available data from the serial port.
/// </summary>
private void SerialThreadProc()
{
	int bytesToRead, bytesRead;
	byte[] buffer = new byte[64];
	while (!terminate)
	{
		bytesToRead = serialPort.BytesToRead;

		if (bytesToRead == 0)
			bytesToRead++;  // Wait for one byte when no bytes are available

		lock (inputBuffer)
			bytesToRead = Math.Min(bytesToRead, inputBuffer.Length - inputBufferUsed);

		if (bytesToRead == 0)
		{   // Buffer full
			Thread.Sleep(100);
			continue;
		}

		bytesRead = serialPort.Read(buffer, 0, Math.Min(bytesToRead, buffer.Length));

		if (bytesRead == 0)
			continue;

		lock (inputBuffer)
		{
			Array.Copy(buffer, 0, inputBuffer, inputBufferUsed, bytesRead);
			inputBufferUsed += bytesRead;
			dataReceived.Set();
		}

		try
		{
			// No need to lock the inputBuffer since the client will not read any data
			// while in updateMode!
			//
			// Search ASCII line ending (search for '\n' or '\r')
			for (int lineEnd = 0; lineEnd < inputBufferUsed; lineEnd++)
			{
				if ((inputBuffer[lineEnd] == '\n') || (inputBuffer[lineEnd] == '\r'))
				{
					if (lineEnd > 0)
					{   // Line contains one or more characters
						// Pass line to callback
						char[] inputChars = new char[lineEnd];
						int bytesUsed, charsUsed;
						bool completed;
						utf8Decoder.Convert(inputBuffer, 0, lineEnd, inputChars, 0, lineEnd, true, out bytesUsed, out charsUsed, out completed);                                    
						
						// Send string to event handlers
						OnMessage(new string(inputChars));
					}

					inputBufferUsed -= lineEnd;
					inputBufferUsed--; // termination char

					// Remove line from buffer
					Array.Copy(inputBuffer, lineEnd + 1, inputBuffer, 0, inputBufferUsed);

					lineEnd = -1;
				}
			}
		}
		catch (Exception ex)
		{
			inputBufferUsed = 0;
			Debug.Print(ex.Message);
		}
	}
}

Set the serial read timeout to f.e. 500ms, so the terminate bool is checked on a periodic basis.

Terminating goes like this:

if (thread != null)
{
    terminate = true;
    thread.Join();
    thread = null;
}

if (serialPort != null)
{
    serialPort.Dispse();
    serialPort = null;
}

[Jeff_Birt]

Here is an event driven serial reciever function I’m using that uses the “\r” charecter as a delimiter. It reads into a buffer until the “\r” is seen. This is not perfect but it works for our purposes right now.

    static void Data_Received(object sender, SerialDataReceivedEventArgs e)
    {
        //Debug.Print("Data Recieved");
        int read_count = xbee.Read(rx_data, buf_end, xbee.BytesToRead);

        if (read_count > 0)
        {
            int lineEndPos = Array.IndexOf(rx_data, endLine, buf_end, read_count);

            if (lineEndPos > 0)
            {
                char[] chars = Encoding.UTF8.GetChars(rx_data);
                string conv = new string(chars);
                Debug.Print("incoming: " + conv);
                buf_end = 0;

                if (conv.IndexOf("Trainee.") == 0)
                {

                    string bob = HandleCommands(conv.Substring(0, lineEndPos)) + "\r";
                    byte[] buffer = Encoding.UTF8.GetBytes(bob);
                    xbee.Write(buffer, 0, buffer.Length);
                }

            }
            else
            {
                buf_end += read_count; // points to end of incomplete Rx
            }
        }

    }

[contractorwolf]

i wanted to share something i discovered, and wanted to get some opinions on it.

when i orginally was just looking at the incoming serial data i had issues trying to combine pieces of the incoming stream trying to figure out how to build the full incoming message string. i started with code that looked like this:

static void uart_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
    rx_data = new byte[uart.BytesToRead];
    uart.Read(rx_data, 0, rx_data.Length);
    message = new String(System.Text.Encoding.UTF8.GetChars(rx_data));

    Debug.Print("[" + messageCount.ToString() + "] " + message);

    messageCount++;
}

the stream of data looked like this:

[233] 1
[234] 27
[235] ,1
[236] 28

[237] 1
[238] 27
[239] ,1
[240] 28

[241] 1
[242] 27
[243] ,1
[244] 28 

the prob was determining when the end of the stream of your individual reading was. my stream is a x,y sent over and over, but putting the pieces back together was a pain. the reason i saw was because the radio is breaking the whole string down to individual pieces. i thought that maybe i could buffer that a little. so i put a small time buffer when the data comes in so that enough data would build up in the buffer before i looked at it. my code idea was this:

static void uart_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
    Thread.Sleep(20);
    rx_data = new byte[uart.BytesToRead];
    uart.Read(rx_data, 0, rx_data.Length);
    message = new String(System.Text.Encoding.UTF8.GetChars(rx_data));

    Debug.Print("[" + messageCount.ToString() + "] " + message);

    messageCount++;
}

now the output i see looks like what i wanted:

[52] 127,128

[53] 127,128

[54] 127,128

[55] 127,128

does anyone see any issues with doing it that way? i can afford 20ms and control both the sender and receiver. the sender is a bluetooth usb running a COM port

thanks for any advice

[Brett]

still you won’t guarantee that this will always work - you just happen to find that 20ms will allow the first content to arrive in the buffer and you grab it; if you made it 24, would you still get the same results? Perhaps if GC kicks in occasionally you will see that extra delay; and that might just break you

You’re still not dealing with the real problem though; you are not “registering” the end of the data entry anywhere you just happen to be lucky enough that it comes into the buffer when your sleep is over. You still need to parse the whole data line and break out your values - well actually maybe I assume you need to for your app, but maybe you just need to output it as text.

[WouterH]

If you look at the serialthread function I proposed here, you can get what you want.

For each line the function will call the OnMessage method and pass the complete line as a string paramter…

[GxG]

Serial communication is a b***h… i know… i’m having the same sync problems you’re having… except i know the size of the chunks i receive…

You could try calling GC right before receiving data so it will give you a little wiggle time so u can be sure for the next couple of messages GC won’t kick in… but that’s an assumption, and you can’t base your code on assumptions.

Thread.Sleep(200) doesn’t bring GC to meddle with my serial receiving, but i don’t have big bits to handle…

What you can try is to set it up at the largest BAUD rate supported and see how fast you can send these blocks, higher the BAUD rate higher the speed… so you can afford to put something like Thread.Sleep for about 10ms maybe, but for large chunks it won’t guarantee anything. The safest way is to define an string end character like \0 which is used as a string terminator and to wait for that… \n or \r won’t ensure anything. \0 was used in C back in the day, and is still in use.

But first try with a higher BAUD rate maybe that will send everything in just one chunk…

[Brett]

or… just use one of the myriad of solutions that others have offered up here that guarantee you process data in the right chunks.