Detect SD Card is full and overwrite oldest file

NETMF & Gadgeteer
1

I just acquired a Panda II and like to use it as a CAN datalogger in automotive application.

Before deploying it to the vehicle, I first tested it logging date and time data to SD card with the codes below (most are cut and paste from the FEZ beginners’ guide and from this forum). A new file is created with date and time as filename in every loop executed. It worked ok but stopped logging once the SD card is full.

Like to seek advise on how to make it detect the SD card is full and overwrite the oldest file. Ideally, I like to leave the device running in the vehicle and only retrieve data when something abnormal happens.

I am a novice in programming, any help is appreciated. Thanks in advance for any advise. Thanks!


using System.Threading;
using System.IO;
using Microsoft.SPOT;
using Microsoft.SPOT.IO;
using Microsoft.SPOT.Hardware;
using System;
using GHIElectronics.NETMF.IO;
using GHIElectronics.NETMF.Hardware;
using GHIElectronics.NETMF.System;


namespace Time_datalog
{
    public class Program
    {
        
        public static void Main()
        {
            // To keep track of time, set it at the beginning of your application from the RTC.

            // If the time was never set before, set it to a certain date and time.
            /*bool timeValid = false;
            if (timeValid == false)
                RealTimeClock.SetTime(DateTime.Today);
                timeValid = true;
            */

            // Set the system time. CAN messages will have a time stamp
            Utility.SetLocalTime(RealTimeClock.GetTime());

            bool CardIn = false;

            RemovableMedia.Insert += new InsertEventHandler(RemovableMedia_Insert);
            RemovableMedia.Eject += new EjectEventHandler(RemovableMedia_Eject);
                       
            try
            {
                // try to access the microSD
                PersistentStorage sdPS = new PersistentStorage("SD");
                sdPS.MountFileSystem();
                CardIn = true;
            }
            catch
            {
                // if failed, assume no card in slot
                Debug.Print("No SD card!");
                CardIn = false;
            }

            while (CardIn)
            {
                using (var w = new StreamWriter(@ "\SD\DataLogFile_" + DateTime.Now.ToString("MMddyyyy") + "_" + DateTime.Now.ToString("hhmmss") + ".txt", true))
                {// Write data
                    DateTime now = DateTime.Now;
                    string iniLine = "date " + now.ToString("ddd MMM dd h:mm:ss ") + now.ToString("tt ").ToLower() + now.ToString("yyyy");
                    Debug.Print(iniLine);
                    w.WriteLine(iniLine);
                    for (int i = 0; i < 2000; i++)
                    {
                        string Nextline = DateTime.Now.ToString(); 
                        //Debug.Print(Nextline);
                        w.WriteLine(Nextline);
                        Thread.Sleep(50);
                    }
                }
            }
        }
        
        
        static void RemovableMedia_Eject(object sender, MediaEventArgs e)
        {
            Debug.Print("SD card ejected");
        }

        static void RemovableMedia_Insert(object sender, MediaEventArgs e)
        {
            Debug.Print("SD card inserted");

            if (e.Volume.IsFormatted)
            {
                Debug.Print("Available folders:");
                string[] strs = Directory.GetDirectories(e.Volume.RootDirectory);
                for (int i = 0; i < strs.Length; i++)
                    Debug.Print(strs[i]);

                Debug.Print("Available files:");
                strs = Directory.GetFiles(e.Volume.RootDirectory);
                for (int i = 0; i < strs.Length; i++)
                    Debug.Print(strs[i]);  
            }
            else
            {
                Debug.Print("SD card is not formatted");
            }
        }
               
    }
}
2

I edited your code tags in your post.

You can do this easier by setting separate folders for each hour or minute of logging and then you have a pre determined values of how many folders to have, lets say 10. Then in your code you always loop and write data to those 10 folders. You will not use 100% of the card but if your card is 16GB for example and you are using half then you are using 8GB of data!!

3

That’s the easy option, yes, and works well in “typical” dataloogger applications, like temperature or GPS loggers, but if you all of a sudden hit a “bad” situation in your car OBD/CAN logger, you want to catch as much of that information as possible - and figuring out how much space you have left on the volume might be a good addition to the SDK :wink:

4

If you name the files with the format of yyyyMMddHHmmss then you can sort it alphabetically and the first file will be the oldest, then you just delete that file.

5

Yep, agreed, appropriately named files make it EASIER to figure out what file you should remove. But when to remove it is still a challenge, plus if you have to scan the entire directory that might contain thousands of files and sort it and pick one to remove, is that going to be effective on the small framework devices? I’m sure it’s a snap when you have lots of memory at your disposal, but…

I don’t see this as anything but a great opportunity to enhance what we have, when we start to think about some of the use-cases we are now able to easily deal with.

6

Like I recommended first, folders are your friends when it come to large count of files

7

You don’t necessarily need thousands of files, as Gus says you could use folders, so you could have the folder be yyyyMMdd with the individual files being HHmmss.

Or instead of creating a new file every time, just have one file per day and append to that file through the course of the day.

As for when to delete, find the average file size and when the space drops below 10x the average file size, delete the oldest day’s worth of files. Or only keep files going back 30 (pick a number) days.

It’s really easy to make something like this work, even on a device with limited memory and cpu.

8

Thanks for all the replies so far. Much appreciated.

I actually got this idea from a small credit card size windscreen mounted camcorder. It stores avi video to SD card at 10 minutes intervals. All the files are written to the root directory and named by date and time. Although each file is of different size, it is able to delete the oldest file(s) to make way for the newest file.

Wanted to try with the forum if there are any ideas how this can be done. Thanks.

9

Here’s a couple of things I’ve thought about in the past 24 hours.

When you start, create a “filler” file that represents a reasonable chunk of data, say 1.5x your day’s worth. That way you always have “free space” if you clean it up. You could even do something like create the size based on VolumeInfo TotalSpace. Wrap your writes in a try-catch block and if you hit a volume full condition then clean up your filler file and re-do your write, and then clean up the oldest file as your new filler file

Use VolumeInfo TotalFreeSpace periodically to check how storage is trending. Clean up the oldest file if you need.