Speed Up Contest! File transfert over CDC

Archive
1

Hi Everyone

I need to transfert some xml files over USB using the CDC. Files are stored on sdcard.

My code is the following one:



string[] files = new string[2];
                    files[0] = "\\SD\\file1.xml";
                    files[1] = "\\SD\\file2.xml";
                    for(int i = 0;i < files.Length;i++)
                    {
                        Watchdog.ResetCounter();
                        FileStream f = new FileStream(files[i],FileMode.Open,FileAccess.Read,FileShare.None,8);
                        byte[] data = new byte[f.Length];
                        f.Read(data,0,data.Length);
                        f.Close();
                        f.Dispose();
                        COM_PC.sendbytes(data);
                        COM_PC.send("\r");
                    }

The sen method is:

 public void send(string s)
        {
            byte[] b = System.Text.Encoding.UTF8.GetBytes(s);
            cdc.Write(b,0,b.Length);
        }

Is there anything that I can do to speedup this?

2

To have a little bit more details: there’s one file about 151KB and the second is 169KB. The average time to transfert is 2.8s.

3

Unfortunately, I don’t have my Panda II ready to test, so this is pure conjecture, but have you tried a static buffer and transferring in chunks? Memory allocation and the associated garbage collection is expensive.

Is the baud rate on CDC fixed? Have you isolated the slowest operation?

Out of curiosity, what hardware are you using?

4

Hi Gregg, I am using EMX.

I will try to use a static buffer and see if it can enhance results.

5

Here’s the code for 1024 bytes packets transfert:


                    string[] files = new string[2];
                    files[0] = "\\SD\\Dataset.xml";
                    files[1] = "\\SD\\Playset.xml";
                    long bytesRemaining;
                    int bytesRead;
                    byte[] data;
DateTime start = DateTime.Now;
                    for(int i = 0;i < files.Length;i++)
                    {
                        Watchdog.ResetCounter();
                        FileStream f = new FileStream(files[i],FileMode.Open,FileAccess.Read,FileShare.None,8);
                        bytesRemaining = f.Length;
                        while(bytesRemaining > 0)
                        {
                            data = new byte[(bytesRemaining > 1024) ? 1024 : bytesRemaining];
                            bytesRead = f.Read(data,0,data.Length);
                            COM_PC.sendbytes(data);
                            bytesRemaining -= bytesRead;
                        }
                        f.Close();
                        f.Dispose();
DateTime end = DateTime.Now;
Debug.Print("RecupXML in," + (end - start).Ticks / 10 + ", micro seconds");

Results shows 1,5s for the first file and 1.7s for the second. while reading with a buffer of file size is 1.25s and 1.4s respectively

6

I am preparing a graph that shows the transfert speed according to the buffer size. Do you know if the buffer size of the filestream will also have an impact? If the buffer used in f.read is greater than 2048, the app seems to freeze… is it something expected?

7

Here are the first results:
These result need to be considered carefully and can not be treated as a rule of thumb but may help you to enhance performance in some specific cases. These are extracted from EMX running SDK 4.1.8.

The first experiment shows the runtime to read 2 xml files on sd and send their contents over CDC when the byte array size used for the read method of filestream is modified. first file is 151kB and the second is 169kB.

The second experiment shows the impact of the filestream buffer size:

As a result I think that choosing a buffer size filestream of 128 or 256 byte is a good option.

8

@ leforban - Try allocating a fixed 1024 byte buffer before starting your for loop, and just keep track of how many bytes are in the buffer as you read in the files.

9

As Jasdev said, and what I meant, was a buffer that does not get reallocated every iteration, something like this:


string[] files = new string[2];
                    files[0] = "\\SD\\Dataset.xml";
                    files[1] = "\\SD\\Playset.xml";
                    long bytesRemaining;
                    int bytesRead;
//Allocate static buffer here:
                    byte[] data = new byte[1024];

DateTime start = DateTime.Now;
                    for(int i = 0;i < files.Length;i++)
                    {
                        Watchdog.ResetCounter();
                        FileStream f = new FileStream(files[i],FileMode.Open,FileAccess.Read,FileShare.None,8);
                        bytesRemaining = f.Length;
                        while(bytesRemaining > 0)
                        {
                            bytesRead = f.Read(data,0,data.Length);
//Add a parameter to COM_PC.sendbytes to denote number of bytes to send, as it may
//be less than 1024
                            COM_PC.sendbytes(data, bytesRead);
                            bytesRemaining -= bytesRead;
                        }
                        f.Close();
                        f.Dispose();
DateTime end = DateTime.Now;
Debug.Print("RecupXML in," + (end - start).Ticks / 10 + ", micro seconds");

You’ll probably find the sweet spot if you play around with the buffer size. Seems like the EMX should have plenty of RAM, so I’d start off trying 16KB, 64KB, and 256KB buffer sizes.

10

Actually setting the buffersize at 2KB results in EMX freezes…