Storing data on SD, removing when done

I’m currently working on a project which collects data from serial port and sends it to a server on the internet. That’s all working fine, but sometimes the internet connection is unavailable for some time (can be one minute, until several days.)

In case the connection is down, I need to store the serial data and send it as soon as the connection is restored. I take an GPS position every minute, so the amount of data collected can be quite a lot.

I tested this by using sqllite on EMX, but quickly ran into issues with the database file, mainly when the power to the module is interupted. After reboot, I often get messages like invalid file format.

I create json objects from the serial data I receive and could easliy store them in textfiles, but I can’t figure out a simple way to remove them from the file after I transfered them to the webserver.

I’m also struggeling with the diffrent tasks that my program has to perform. Right now, I have one thread collecting the data and another thread sending it to the server. I wonder what the best way is to approach all this, when implementing the third task of storing the data when connection is unavailable.

As always, thanks in advance for any suggestions.

I ran into performance problems when writing log files onto SD.
Also if power goes off while writing the file or even the whole directory gets corrupted.
To avoid that you could add a ‘UPS’ or battery backup and flush the file system on power down.

To remove your json objects from file you could overwrite them with zeros or remember the last ‘already sent’ position in a separate file.
Shortening a file at the beginning is not possible I think.
Also smaller files which are sent at once would be possible.

There’s no reason you can’t have a log file that holds your data and uploads it when the connection is back up. Something like:

Lock file object so new data doesn’t get added
Copy current logged data file to temporary file
remove lock
open connection to server
loop through for each data point
push data to server - if fails, write data to retry queue file
remove temporary file

Then you know each iteration has either pushed the data up or it has resulted in something you can retry later (again). Making sure nothing writes to your file at the start is important, and the only scenario that you really have to worry about is failure (power loss etc) in the middle of that process - which if you had sequential IDs on records for example you could just track what point in the file you were up to.

This whole scenario though doesn’t protect you from power disruption and what I expect the “corruption” you saw actually is. I’d invest in some power-detection circuitry and a “battery” backup. I’ve been thinking about using a supercap as the “power supply” and detecting when source power before the cap drops low, and go into “flush and shutdown” mode. The probable cause of the corruption is half-written writes to file entries causing you to not be able to parse the file correctly next time round, and that will be no different just using text files. I always use a protection block like the following when I log data to SD:

            if (SD_ready)
                using (FileStream myFile = File.Open(sdCard.GetStorageDevice().RootDirectory + "\\mylog.txt", FileMode.Append))
                    byte[] logMsg = UTF8Encoding.UTF8.GetBytes("Log: " + DateTime.Now.ToString() + "," + msg + "\r\n");
                    myFile.Write(logMsg, 0, logMsg.Length);
                    Debug.Print(DateTime.Now.ToString() + "," + msg);


So, it seems to me that the biggest concern is file corruption on power failure. Using a supercap seems a valid idea to overcome this. Detecting loss of power is straight forward.

@ Brett - Your solution looks simple enough. I’ll give that a try and see how it goes. Thanks for the suggestion.

That’s my biggest concern anyway :slight_smile:

And just to be clear, that block of code is only to minimise the risk of data loss/corruption during the logging. If the card isn’t properly unmounted you can still lose data (I use a button-press to safely unmount it and then you can remove it with little chance of issue)

@ Brett - I see what you mean, but I won’t be taking out the SD card anyway, so I have no need to unmount it.
I’m just using the SD to store the data, because I would run out of ram very fast when internet is gone for a couple of days and all gps positions should be stored in RAM.
Not to mention the loss of data when power loss would occur.

How about unmounting the SD after using it? Would mounting and unmounting it all the time take up extra resources? I’m asking this, because it seems that unmounting it will force the data to be flushed to the card right?

unmounting can require you to reinsert it physically, so don’t do that :slight_smile: For me, unmounting was the scenario where I needed to take the card and read it elsewhere, so doesn’t really apply in your circumstance. Re-mounting is also an expensive (time wise) process so not something you want when you have high data rates.

The construct I showed earlier is sufficient to protect against a “normal” powerloss situation, as long as it doesn’t happen when the write is going on (very small chance really) plus it’s reliable in that the handle gets disposed properly and doesn’t leak - thanks to the forum for sharing that. That’s all I’d do to protect against that scenario.

I have a Panda2-based product that continuously logs NMEA-0183 data to a microSD card (GPS, depth, etc.) at rates of up to 20 Hz. Power can be turned off at any time, and I do not have supercaps or any kind of power backup implemented. This product has been shipping to customers worldwide since 2011, and not one customer has yet complained about file corruption. I perform a FileStream.Flush() after every write, and I create a new log file after every power cycle, appended with an increasing index number in the filename.

1 Like

@ Iggmoe - I tried it in a similar way with a G120. and got corrupted Files and poor Performance all the time. My Network Connection was nearly down. And I only had to write about 20 lines per Minute!

I use this allot to access the SD and read log files or update config files.
I unmount SD and mount again to the USBClient

                        // check if USB is connected
                        USBClientController.State usbState = USBClientController.GetState();

                        // connect only once !
                        if (usbState == USBClientController.State.Running && MyUSBClientLoaded == 0)
                            // only if SD is actualy mounted !!
                            if (SDStorage != 0)
                                // flag 
                                MyUSBClientLoaded = 1;

                                // log naar syslog
                                SysLogFileWriteLine(RealDateTime + " : " + "USB Client connected");

                                // still mounted -> close all file handles

                                    MyUSBClient.AttachLun(0, MySD, "Emrol", "IRC Controller");
                                    SysLogFileWriteLine(RealDateTime + " : " + "Error at attachlun(0) and µSD");

As a workaround to make sure SD is mounted correctly (and to read my new config files) i reboot the device after i had external acces to the SD card.

                            SysLogFileWriteLine(RealDateTime + " : " + "Software Rebooting");
                            PowerState.RebootDevice(true, 1000);

Works just fine all the time.

For the power loss: i have a 500mA battery and charging circuit in my design to survive short power breaks and allows me to do a proper shutdown when external power is down for more than 5 seconds.

@ Reinhard - Must be a case of “your mileage may vary.” The Panda2 has been excellent, but it seems like the G120 does things a little differently.