Appending to large files on the SD card or USB

TinyCLR OS
1

Hi all,

I’ve been trying to figure out how to do this for some time now. What we’re doing is taking a reading every half a second and writing about 25-30 bytes of data to a CSV file on the SD card (we started with a usb drive, but this more stable).

However, that many writes causes it to lock up, so all of it is buffered and then written to the file stream once every 30 seconds. After about 10 minutes, trying to access the file will cause it to freeze and force me to reset the system.

this is opening code:

 public static bool Open(string filename, FileMode fm)
    {
        bool inError = false;
        SDfilename = filename;
        do
        {
            try
            {
                string ss = $@"{drive.Name}{filename}";
                if (fm != FileMode.Append)
                    file = new FileStream(ss, fm, FileAccess.ReadWrite);
                else
                    file = new FileStream(ss, fm, FileAccess.Write);
                inError = false;
            }
            catch(ArgumentException e)
            {
                throw new InvalidOperationException("File does not exist", e);
            }
            catch (IOException E)
            {

This is the Append Code:

public static void Append(byte[] _buf)
    {

        file.Seek(0, SeekOrigin.End);

        file.Write(_buf, 0, _buf.Length);
        file.Flush();
        do
        {
            Thread.Sleep(10);
        }
        while (!drive.IsReady);
        

    }

The largest file this has ever made is only like 32 KB before it completely locks up when trying to open the file, which is tiny. We were experimenting with just storing the data in ram before writing it at the end of a test but its more preferable to write directly to storage. Any thoughts?

2

Wasn’t there a file.append function?

Not at my pc at the moment, so can’t check

3

I believe just open file in append mode and write your data.

2 Likes
4

Is there anything else going on though? i checked what the append filemode did on github and it’s doing exactly what we are - is there something else that happens even deeper that’s different between appending and writing?

5

I’m kind of at my wits end with this - unless the filemode under the hood is doing more than I think it is based off of the source code, I’ve tried everything. Flushing the file, flushing the storage controller, locking access to the sd card to a single thread, limiting my writes to only once every 30 seconds. It’s opening the file alright, but then after 15-20 minutes, will throw the following exception (this one happened at 23 min) doing the exact same write it had done 50 times up to that point, and even frequently permanently breaking the ability to write to the SD card until I reformat it. Is there a possibility it is something going on electrically? I’m not familiar with the how an SD/MMC card works in an electrical sense. This issue also persists between SITcore modules.

image
image699×187 3.19 KB

6

I have no insight into the firmware level native code, and this exception is coming from the native code in the Write routine (so seems to have already found the position to write at), but that would seem to indicate something involved in talking to the SD card (native firmware or hardware) or a bad SD card. Have you tried more than one SD card?

7

It’s a common symptom between both my flash drive and my sd card, but I will try another SD card. I’ve also noticed that there’s a linearish response with how big the allocation size is in the FAT format and how long it will go before petering out. 4096 bytes only lasts a few minutes while 64 KB will go for over 20. I don’t know how that is related but it may be a hint.

8

Here is my suggestion… Make an great app on the dev board, try with SD and USB. Try multiple media brands. And finally give us the sinks project you have that show the issue and we help investigate, assuming the issue is there.

The dev board is my best friend in cases like this. And we are here ready to help.

1 Like