Losing data with large transfers over wifi rn171 module

Archive
1

I’m trying to use my RN171 module to transmit somewhat large files over wifi. I’m using a fez Raptor main board and the files I’m attempting to transfer are stored on an sd card. Using the simple http example I was able to respond using using the code following; which successfully initiates and proceeds to attempt a download through any standard browser.

var SDCard = sdCard.GetStorageDevice();
            using (var file = SDCard.OpenRead(@ "\test.csv"))
            {
                request.Response.HeaderData["Accept-Ranges"] = "bytes";
                request.Response.HeaderData["Content-Type"] = "application/octet-stream";
                request.Response.HeaderData["Content-Length"] = file.Length.ToString();
                request.Response.HeaderData["Connection"] = "keep-alive";
                request.Response.StatusCode =                     						  GTM.GHIElectronics.HttpResponse.ResponseStatus.OK;
                //All you have to do is send the document data through the response                       		  //object.                 
                //Header data is automatically applied for you when you chose to send.                 
                //request.Response.Send(System.Text.Encoding.UTF8.GetBytes(document));
                //using(LargeBuffer buffer = new LargeBuffer((int)file.Length))
                //{
                //    file.Read(buffer.Bytes, 0, (int)file.Length);
                // request.Response.Send(buffer.Bytes);                    
                //}

                    var dataRemaining = (int)file.Length;
                    var buffer = new byte[1024]; // adjust size to your needs 
                    int cnt = 0;
                    while (dataRemaining > 0)
                    {
                        if (dataRemaining < buffer.Length)
                        {
                            buffer = new byte[dataRemaining];
                            
                        }
                        cnt = file.Read(buffer, 0, buffer.Length);
                        if (dataRemaining == (int)file.Length)
                        {
                            request.Response.Send(buffer);
                        }
                        else
                        {
                            wifi_RN171.Send(buffer);
                        }
                        dataRemaining -= cnt;
                    }
            }

Unfortunately This method has consistently left me lacking in bytes on the destination pc, and as such said download will never complete.

In an attempt to remedy this I discovered that using the code above the 1024th byte in the buffer is being repeatedly dropped. With that in mind I was able to add a 1025th byte to the buffer to achieve complete downloads However I later discovered that this is in no way scale-able as larger files grew more and more complex patterns of data loss, and smaller files risked having no data loss at all, leaving me with too much data.

I’m looking for any advice or solutions that can help solve this data loss problem.

2

Try setting the buffer to:

var buffer = new byte[500];

3

will try, not with the device atm, any reason that’d make a difference, or just a hunch worth trying?

4

It might be the router buffer.

Its that nice round number 1024 base 2.

5

@ Cowboy - I see, I’ve tried a whole bunch of different buffer sizes with no real change (different losses, but never none) but as soon as I can get back to the device I’ll try it out

6 


Are you sure the file length is within the limits of an int?
7

@ jstone05 - could you send your csv file to james.dudeck at ghielectronics dot com

8

@ jstone05 - We have been able to reproduce the issue. We will continue investigating the cause of the issue.

9

@ James - that’s “great” to hear, I look forward to any insight you can provide. I’d also like to note that if there’s an alternative method for transferring large(ish) files that would sidestep this issue were open to hearing it the method used above was just the only method we saw available

10

@ James - Any progress to report?
our development is completely stalled out until we can identify for a fix for this issue! :’(

11

@ jstone05 - I posted a set of example code on your other thread that allowed me to download consecutively without data corruption. There may be a few thread-safe issues to worry about with that example code still, but it should prove to be a good base point.

12

@ James - using the following code (adapted from your post) did not resolve the issue; Data loss occurred and as such I only managed to retrieve 9,580,728 of 9,582,930 bytes (ala fiddler)

using (var file = SDCard.OpenRead(@ "\test.csv"))
            {
                request.Response.HeaderData["Content-Type"] = "text/csv";
                request.Response.HeaderData["Content-Length"] = file.Length.ToString();
                request.Response.HeaderData["Connection"] = "close";
                request.Response.HeaderData["Cache-Control"] = "no-cache";
                request.Response.StatusCode = GTM.GHIElectronics.HttpResponse.ResponseStatus.OK;
                //All you have to do is send the document data through the response object.                 
                //Header data is automatically applied for you when you chose to send.                 
                //request.Response.Send(System.Text.Encoding.UTF8.GetBytes(document));
                //using(LargeBuffer buffer = new LargeBuffer((int)file.Length))
                //{
                //    file.Read(buffer.Bytes, 0, (int)file.Length);
                // request.Response.Send(buffer.Bytes);                    
                //}

                int baud_rate = 1024;
                int iterations = (int)(file.Length / baud_rate);
                int exceeding = (int)(file.Length % baud_rate);
                int segment_start = 0;
                if (exceeding > 0) iterations++;
                for (int i = 0; i < iterations; i++)
                {                         
                    //Calculate segment range                         
                    segment_start = i * baud_rate;                         
                    byte[] document = new byte[1024];                         
                    file.Read(document, 0, 1024);                         
                    //Write the current segment                         
                    if (i == 0)                             
                        request.Response.Send(document);                         
                    else                             
                        wifi_RN171.Send(document);                         
                    System.Threading.Thread.Sleep(50);                     
                }
            }