Main Site Documentation

Serial comms - using all the memory up


#1

ok i`m playing aroung with the serial comms on the domino , i have linked the com1 in and out together, and have deployed the following code.

the data is being outputted fine and received ok, the problem is if i leave the code running for a while after about 10 cycles i seem to run out of memory i say this because in the debug i can see “Type 1E (BINARY_BLOB_HEAD ): 47436 bytes” this value gets larger and poof project stops.

i have desabled the data received event and it runs forever so i know the problem is in their, but alas i`m at a loss, any ideas ??


using System;
using System.Threading;
using System.IO.Ports;
using System.Text;

using Microsoft.SPOT;
using Microsoft.SPOT.Hardware;

using GHIElectronics.NETMF.FEZ;

namespace FEZ_Domino_Application2
{  
    public class Program
    {
        static SerialPort uart = new SerialPort("COM1");
        static byte[] buffer = new byte[66];

        public static void Main()
        {
            uart.DataReceived += new SerialDataReceivedEventHandler(Test_DataReceived);
            byte[] TX_Data = Encoding.UTF8.GetBytes("$GPRMC,225446,A,4916.45,N,12311.12,W,000.5,054.7,191194,020.3,E*68");
            uart.Open();

            while(true)
            {
                uart.Write(TX_Data, 0, TX_Data.Length);
                Thread.Sleep(1000);
            }
        }

        static void Test_DataReceived(object sender, SerialDataReceivedEventArgs e)
        {
            uart.Read(buffer, 0, buffer.Length);
            string RX_Data = new string(System.Text.Encoding.UTF8.GetChars(buffer));
            Debug.Print("Received Data: " + RX_Data);
        }
    }
}


#2

In the event, you are only reading 66 bytes. If your received data is more than 66 then you are leaving some data in memory. Few loops later and you have filled the memory up


#3

Gus:

I thought the buffer for the UART was a fixed size? I would hope that if an overflow occurs, there woudl be an exception that could be handled.


#4

He is doing partial read and I think this moves the data from native to managed queue

Errors are handled but you also should plan your design to read all data or discard it


#5

You could check the number of bytes available to read by checking the value of BytesToRead before reading to the buffer.

Debug.Print("Bytes to read: " + uart.BytesToRead.ToString());

The next note may not matter much if you are only receiving once a second, and if your debug output doesn’t break up the GPS sentence and/or you don’t see any blank "Received Data: " sections.

I don’t know how similar the micro framework is to the full framework in it’s serial port implementation or how quickly the DataReceived event handler is called, but the DataReceived event may not fire for every byte received, and it may fire before the end of transmission (depending on transmission length and how quickly the framework switches between the putting the main thread to sleep and allocating a thread to fire the event handler) which could leave you working with a fragment of the transmission. At the very least, I usually check for a non zero value of BytesToRead before I perform the read because you could theoretically fire the event once for every byte received, even though you may have already removed those bytes from the buffer.

            if (uart.BytesToRead > 0)
            {
                byte[] buffer = new byte[uart.BytesToRead];
                uart.Read(buffer, 0, buffer.Length);
                //do work based on the buffer here
                //DoWork();
            }
//OR
            while (uart.BytesToRead > 0)
            {
                //use the buffer of fixed size like the original code
                if (uart.BytesToRead > buffer.Length)  //the buffer is smaller than the uart buffer
                {
                    while (uart.BytesToRead > buffer.Length)
                    {
                            uart.Read(buffer, 0, buffer.Length);
                            //do work on the full buffer
                            //DoWork();
                    }
                }
                uart.Read(buffer, 0, uart.BytesToRead);
                //do work on the partial buffer
                //DoWork();
            }

EDIT: updated code to be easier to follow and to eliminate incorrect array offsets


#6

The MF is very close to the full .NET serial implemenation for the functions supported by both.

For both, no assumptions can be made on how many characters of a total message will be read with one DataReceived event.

I can’t say I have ever received a DataReceived event when there is no data available.

I usually include a “while (uart.BytesToRead > 0)” around my event code to catch any additional bytes that arrive while I am processing the data that was available on entry to the interrupt routine.


#7

hey all,

thank you for the replies, im gonna add a couple of checks to the code as suggested like ensure there is some data, and check how many bytes are availble, well soon as my tea is cooked, umm sea bass tonight :) anyways what is still dont get is yes i have set the code to read 66 bytes but i`m only sending 66 bytes 1 second. and the byte to string conversion then debug print always displays the full message, so how could i be only partially reading the uart buffer ? interestingly if i change the sent bytes to say abcde and the buffer byte to 5 the code runs for longer but still falls over with no memory left.

i`ll give it a try have some food not nessasarily in that order and get back to you, oh and thanks for the help


#8

If you are reading what you are sending then you should have zero bytes in UART buffer, right? Just check that and you will know where the problem is.


#9

Good point Mike, you mentioning that made me realize I wrote that sample code more complicated than it needed to be. I’m editing it above.


#10

awesome guys many thanks for the help really appreciated. ok so i added some comments to check i understand whats happening, there is one while command that im not so sure i need what do you think ? so what have i learned well if i output x amount of bytes i shouldnt expect x amount of bytes to appear on each event trigger. oh and i can see each iteration carries on the sequence where it left off whilst using send data that is less in length than the buffer :smiley:


using System;
using System.Threading;
using System.IO.Ports;
using System.Text;

using Microsoft.SPOT;
using Microsoft.SPOT.Hardware;

using GHIElectronics.NETMF.FEZ;

namespace FEZ_Domino_Application2
{  
    public class Program
    {
        static SerialPort uart = new SerialPort("COM1");
        static byte[] buffer = new byte[66];
        public static int count = 0; // added a counter for each interation of code

        public static void Main()
        {
            uart.DataReceived += new SerialDataReceivedEventHandler(Test_DataReceived);
            byte[] TX_Data = Encoding.UTF8.GetBytes("0123456789"); // switched data out for testing //  $GPRMC,225446,A,4916.45,N,12311.12,W,000.5,054.7,191194,020.3,E*68
            uart.Open();

            while(true)
            {
                uart.Write(TX_Data, 0, TX_Data.Length);
                Thread.Sleep(5000);
            }
        }

        static void Test_DataReceived(object sender, SerialDataReceivedEventArgs e)
        {
            while (uart.BytesToRead > 0) // catch data that arrives during processing
            {
                if (uart.BytesToRead > buffer.Length) // ensure the buffer is full
                {
                    while (uart.BytesToRead > buffer.Length) // while the buffer is full [DO I NEED THIS WHILE AS THE IF CATCHES THIS ???]
                    {
                        count +=1; // increment the interation counter for display only
                        Debug.Print("************************************"); // break the debug code is its easier to read
                        Debug.Print("Bytes to read before uart read: " + uart.BytesToRead.ToString()); // test uart length before reading

                        uart.Read(buffer, 0, buffer.Length); // read buffer
                        string RX_Data = new string(System.Text.Encoding.UTF8.GetChars(buffer)); // convert buffer chars to string
                        Debug.Print(count + ": Received Data: " + RX_Data); // print the buffer contents for the preset buffer length

                        Debug.Print("Bytes to read after uart read: " + uart.BytesToRead.ToString()); // test uart length after reading
                        Debug.Print("************************************");// break the debug code is its easier to read
                    }
                }
            }
        }
    }
}


output from the debugger


Bytes to read before uart read: 68
1: Received Data: 012345678901234567890123456789012345678901234567890123456789012345
Bytes to read after uart read: 4



Bytes to read before uart read: 68
2: Received Data: 678901234567890123456789012345678901234567890123456789012345678901
Bytes to read after uart read: 6



Bytes to read before uart read: 68
3: Received Data: 234567890123456789012345678901234567890123456789012345678901234567
Bytes to read after uart read: 2



Bytes to read before uart read: 68
4: Received Data: 890123456789012345678901234567890123456789012345678901234567890123
Bytes to read after uart read: 6



Bytes to read before uart read: 68
5: Received Data: 456789012345678901234567890123456789012345678901234567890123456789
Bytes to read after uart read: 6



#11

I had that exact problem last week. :smiley: Mine was reading 6000bytes every 1500ms (1.5s).

My findings were the WHILE loop and the uart.READ causes the Fez to lock on that thread. But as is reads the incoming bytes using uart.read, “uart.DataReceived += new SerialDataReceivedEventHandler(Test_DataReceived);” is also constantly trying to fire the locked thread, thus the binary blob immediately maxes out.

reading 66 bytes = 66 SerialDataReceivedEvents ??

What I did was, unsubscribe to that event when the while/read is running then subscribe again after while/read finishes.
uart.DataReceived -= new SerialDataReceivedEventHandler(Test_DataReceived);

But, that fix is not recommended by experienced programmers.


#12

I believe it is necessary to subscribe to the DataReceived event after the uart has been opened, not before, as was done in the code above.


#13

i am facing a similar problem… waiting for solutions…


#14

The code you’re using isn’t handling partially filled buffers, which is why you are getting long strings of numbers instead of just the 10 characters you send every 5 seconds (and is probably maxing out the possible cpu time while it waits for the serial buffer to reach 66 characters). Try the below instead. This should completely empty the serial buffer every time the event is fired. And just for clarity, were you still having an issue with the FEZ crashing?


            count += 1; // this will now count fired but unused events too
            while (uart.BytesToRead > 0) // catch data that arrives during processing
            { // increment the interation counter for display only
                Debug.Print("************************************"); // break the debug code is its easier to read
                Debug.Print("Time: " + DateTime.Now.ToString());
                Debug.Print("Bytes to read before uart read: " + uart.BytesToRead.ToString()); // test uart length before reading

                uart.Read(buffer, 0, buffer.Length); // read buffer
                string RX_Data = new string(System.Text.Encoding.UTF8.GetChars(buffer)); // convert buffer chars to string
                Debug.Print(count + ": Received Data: " + RX_Data); // print the buffer contents for the preset buffer length

                Debug.Print("Bytes to read after uart read: " + uart.BytesToRead.ToString()); // test uart length after reading
                Debug.Print("************************************");// break the debug code is its easier to read
            }

Kein - I don’t have a hardware device yet to test on (waiting on my Panda to ship) but in theory the serial DataReceived event should be queued instead of trying to break into the existing thread. The way NUBIARN’s current “while” loop is written, it will consume as much processor time as can be alloted because it takes at least 7 serial transmissions (35 seconds) to get past the “if” condition. That is why I gave full and partial buffer handling in my original suggestion. As long as the event handler does not lock for a long period of time, you shouldn’t run into collisions.

In my experience, the two main reasons for locking up a cpu in the DataReceived event handler are waiting for more bytes directly in the handler, and processing the data directly in the event handler. I like to use the event handler to queue data to be processed in a separate (lower priority) thread.

Mike - I don’t know which you are [italic]supposed[/italic] to do, but in the full framework I regularly subscribe to the event prior to opening the port. I think the bigger issue is if you re-declare the variable, since the delegate would not be associated with the new object. I could be wrong, of course, but that has been my past experience.


#15

I found one solution, just send one line of data and wait for acknowledgement from the other side, and then send another one again.

The other side will send the acknowledgement right after it finishes processing the data it has. Thus it will prevent filling the RAM.


#16

actually, that is probably just reducing the likelihood of you getting a recieve event when there’s more data still coming in from the UART.


#17

well no, cuz the tx side will wait for the acknowledgement before sending out a new set of data, meaning there is no new transmissions when the current data is being processed.


#18

well, perhaps not. My hypothesis is still that in the case you describe, you only have a small number of bytes coming in, so your single recieved event pass works fine - your previous problem is caused when you have to process multiple passes to get what you expect. By chunking it down, it’s more likely that the incoming data will be complete.

Either way, if you’re happy and your code is functioning as you need it, then all is well :slight_smile:


#19

You are right. I am just sending remote control strings across uart. Each line won’t be more than a few characters.


#20

hey all, im still working on a robust solution and trying different things, damn ive learnt alot in a few days about buffers and uarts. ill post some more code soon as i get a robust solution, it is actually quite a interesting problem, well it is for me anyways :slight_smile: and looking at some of the posts i think i`m not the only one who has been faced with said issues, however its all part of a learning curve and thats what makes this so much fun.