Super Slow SD card write

Archive
1

Hej!

I was wondering if one of the gadgeteer experts could help me with my code. Right now I am writing a 100,000 * 6 element int array to the sd card with the following and it is taking 10 to 15 minutes to write it. If I only do the string conversion it only takes 20 or so seconds so i assume it is the write to SD that is taking so long.


for (int i=0;i<100000;i++)
                    {
                        streamWriter.Write(myInt[i].ToString()+" "+myInt1[i].ToString()+" "+myInt1[i].ToString()+" "+myInt1[i].ToString()+" "+myInt1[i].ToString()+" "+myInt1[i].ToString()+" ");
                    }

I am initializing with the following:



using (Stream configStream = sdCard.GetStorageDevice().OpenWrite(@ "\test111"+".txt"))

StreamWriter streamWriter = new StreamWriter(configStream);

To be honest after 10 mintues I gave up. Any suggestions?

I do know it is writing because if I make the array shorter it writes what i want it to.

2

The problem is that your trying to call the StreamWriter 100,000 times to write out just a little bit of data each time. So for each write of just a few bytes NETMF has to package up your string to something native code can work with, call native method that handles StreamWriter, wait for the write to be done and then move back up to managed code. This whole process is very expensive in terms of time.

If would be much better to convert all of your data to a string and write all at once, or convert it in large chunks and write out the large chunks.

3

This should help http://tinyclr.com/forum/1/1858

4

I’d log the data as plain binary using FileStream instead. In that case using byte[] instead of int[] so as to write big chunks as Jeff and Gus advised should be better.

5

@ Farza,

The problem is I am reading analog values and they must be read as ints. If I create a byte array netmf (as far as I remember) will not let me save analog to a byte array. Anyways I appreciate the help from everyone. I am really banging my head against the wall. I have six analog sensors that need to be sampled at the very minimum 2 KSPS. This means 12000 integers per seconds for around 50 seconds. The I need to write these values to an SD card and be back and up and running in 10 to 20 seconds. I have tried a lot of different things like big strings or string arrays and it is still terribly slow. One idea I had was to sample from an Arduino and send it via serial and have the Hydra just saving the data. That is a waste of a Hydra though. It seems like a simple project and if I understood RLP I suspect it would be.

6

since the analog input is ten bits, each value can be stored ib two bytes. convert to individual bytes by shifting and masking and put in large array.

7

Using 2 bytes/sample, you end up with a total array of size 21200050=1.200.000 bytes=1171.875KBytes

You have 10-20 seconds to log that data, so your write speed should be at least 117.2KBytes/s worst case, which is a little above 1/3 what Gus mentioned on that thread, so it should be possible to achieve it.

I also think you should do the sampling in native code, because it’ll be much less time constrained.