Sending large files over wifi

I think I also found a bug in WiFI_RN171:
If the buffer is larger than baudrate then it sends in chunks, but …

/// <summary>
    /// Send data to the currently connected client
    /// 
    /// </summary>
    /// <param name="data">Data</param>
    public void Send(byte[] data)
    {
      if (data.Length <= this._wifly.BaudRate)
      {
        this._wifly.Write(data, 0, data.Length);
      }
      else
      {
        int baudRate = this._wifly.BaudRate;
        int num = data.Length / baudRate;
        int count = data.Length % baudRate;
        for (int index = 0; index < num; ++index)
        {
          int offset = index * baudRate;
          this._wifly.Write(data, offset, baudRate);
          if (index == num - 1 && count > 0)
            this._wifly.Write(data, offset, count); // this line is wrong I think
            // should be
            //this._wifly.Write(data, offset + baudRate, count);
        }
      }
    }

If the offset is not increased by baudRate to send the remaining data, the final chunk will be a partial repetition of the previous chunk.

anyway: checking if this is the last complete chunk inside the loop is quite ineffective.
Also recalculating the offset with a multiplication could be replaced by a more simple addition.

But to come back to the original problem.
The number of bytes sent should be right.
Only the last bit of the data is wrong if the buffer is larger than baud rate.

@ RoSchmi - The number of missing bytes seems to vary, and it is usually very few. my file size is 1,898,880 or ~2MB, and through fiddler I can see only 1898876 bytes make it through. This also presents itself in the file: I the last line is missing characters.

@ Reinhard Ostermeier - I’m not sure what to tell you, If there’s a test you’d like me to try I’m more than willing.
Also to avoid the bug it’s not problematic for me to shrink the buffer to accommodate the baud rate, However I’m seeing it’s set by default to 115200 and since I’m only sending 10240 Bytes per send that seems like It’d be fine, is there a conversion happening that I’m not seeing?

Also worth noting is that these missing bytes only disappear some of the time, I’m frequently able to obtain the entire file in (Seemingly) perfect condition. That being said the failure rate is ~40% (4/10 attempting downloads fail) and being that these downloads take upwards of 15 min. each It’s unacceptable.

Update:

tried another file and got 1837 KB/1854 kb transfered. The wifi module threw no errors and sent no more data; as far as it’s concerned It’s job was done. This is the exact code I’m using (copy pasted directly from the project)

another test 1,898,711/1,898,887 bytes received; also worth noting is that the download prompt didn’t go off this time so I had to use fiddler to track the progress.

var SDCard = sdCard.GetStorageDevice();
            using (var file = SDCard.OpenRead(@ "\test.csv"))
            {
                request.Response.HeaderData["nope"] = "application/octet-stream";
                request.Response.HeaderData["Accept-Ranges"] = "bytes";
                request.Response.HeaderData["Content-Type"] = "application/octet-stream";
                request.Response.HeaderData["Content-Disposition"] = "attachment";
                request.Response.HeaderData["Content-Length"] = (file.Length).ToString();
                request.Response.HeaderData["Connection"] = "close";
                request.Response.HeaderData["Cache-Control"] = "no-cache";
                request.Response.StatusCode = GTM.GHIElectronics.HttpResponse.ResponseStatus.OK;
                //All you have to do is send the document data through the response object.                 
                //Header data is automatically applied for you when you chose to send.                 
                //request.Response.Send(System.Text.Encoding.UTF8.GetBytes(document));
                var dataRemaining = (int)file.Length;
                var buffer = new byte[10 * 1024]; // adjust size to your needs 
                int cnt = 0;
                while (dataRemaining > 0)
                {
                    if (dataRemaining < buffer.Length)
                    {
                        buffer = new byte[dataRemaining];
                    }
                    cnt = file.Read(buffer, 0, buffer.Length);
                    if (dataRemaining == (int)file.Length)
                    {
                        request.Response.Send(buffer);
                    }
                    else
                    {
                        wifi_RN171.Send(buffer);
                    }
                    dataRemaining -= cnt;
                }
            }

If the transfer is ok sometimes, then I would say the code is not the problem.
Then it might be WiFi, which would mean the TCP/IP stack on the RN171 is very bad.
Or it might be the serial interface between CPU and RN171 module.
Since I never used any WiFi module my self I’m out of the game here I think.

You could also check the Network traffic with WireShark, but I assume you see the same bytes missing as in the file.

Last but not least a Flush on the Serial object of RN171 might be necessary, but it’s not possible by the given interface.

One more idea:
It might be an buffer overflow in RN171 which is not detected somehow.
You could add a Thread.Sleep after each loop iteration.
Start with larger delay, if it works then reduce it until it fails again.
You could ale make the buffer smaller to do this too.

@ Reinhard Ostermeier - I’ll try both the sleep and the smaller buffer, those seem to be likely enough causes. Thanks so much for all your help.

@ jstone05 -
Your file examples give interesting results: In one case 4 bytes are missing, in another case 176 byte, in the third case 17 kbytes.
As Reinhard already suggested I would try to add a thread.sleep in the loop (see my post #5) and take a smaller buffer (e.g. 2000 bytes).
Another option to find bugs is that you do not add the module with the designer but add the driver class from codeplex:
http://gadgeteer.codeplex.com/SourceControl/latest

to your project. You then can include a counter in the driver class with which you can count the bytes really sent to the RN171 with the _wifly.Write command. If this counter gives the right file size, the fault must be in the RN171 itself (e.g. buffer overflow).

If you add the driver sources by to your project you also could call the Serial.Flush method.
Not sure if it is necessary or if it should be done after each write command. But this might help too.

To test the flush hypothesis, use SetBaudRate after you send the response. SetBaudRate closes and reopens the serial port; Close will flush buffer.

If you decide to modify the driver, this might help: https://www.ghielectronics.com/docs/122/gadgeteer-driver-modification

@ Jeff - Can you confirm the bug when sending buffers larger than the baud rate?

I have trouble understanding how buffer size can interact with baud rate? ???

The only reason for this I can think of is that the module needs a small break after a certain time.
Sending in chunks of “baud rate” bytes would mean a small pause after about 10 sec. right!?
Start +Stop bit + 8 data bits = 10 Bits / byte
(10 bit * “baud rate” bytes) / "baud rate) = 10 sec.

Quick update:

Using smaller buffers or adding a Thread.sleep() did nothing to effect the data loss problem, I’m doing 1 last “quick” test using setbaudrate as an alternative to a flush, and then I’ll proceed to poke around in the drivers (regardless of if that works, though I’ll post an update here if it does)

So I found some interesting data (though no fix yet) along with the setbaudrate solution I added an extra send to the end of my loop, that way I could get a “complete” file and from there check the data corruption for some kind of patern

var dataRemaining = (int)file.Length;
                    var buffer = new byte[1024]; // adjust size to your needs 
                    int cnt = 0;
                    while (dataRemaining > 0)
                    {
                        if (dataRemaining < buffer.Length)
                        {
                            buffer = new byte[dataRemaining];
                            
                        }
                        cnt = file.Read(buffer, 0, buffer.Length);
                        if (dataRemaining == (int)file.Length)
                        {
                            request.Response.Send(buffer);
                            wifi_RN171.SetBaudRate();
                        }
                        else
                        {
                            wifi_RN171.Send(buffer);
                            wifi_RN171.SetBaudRate();
                        }
                        dataRemaining -= cnt;
                    }
            }
//breakpoint here
            wifi_RN171.Send(new byte[] { 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, });
            

Whats shocking is that when I set a break point on my nonsense send method it was never reached this leads me to believe that the module must have crashed/been crashing.

I’m gonna poke around in the driver and see if I can’t get to the “bottom” of this, but I wanted to keep a sort of log here,

Hi,
your command line:
wifi_RN171.Send(new byte[] { 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00, 00,
seems not to be just behind the closing while Loop. There is one closing bracket where I cannot see where it belongs to

@ RoSchmi - sorry, there’s a using(var file = SDCard.OpenRead(@ “\test.csv”)){up there,

the file will be disposed before that last send is called. On that note not hitting said breakpoint may have been a fault on the part of the debugger, my systems acting strange in that regard so I’m gonna do a restart soon, just got to wrap up some other work

There is also a strange behaviour with Debugger, that it sometimes stops execution, even if there is no breakpoint at all.
If you hit pause, then Play in VS it will continue.
I never had this Problem when Debugger is not attached.

So lets say you always start with Debugger, and debugger stops in the middle of the file Transfer. Then you get exactly the Problem you have.

So next time your download does not complete, press pause and start in VS to see if this is the Problem.

first things first, I was mistaken with the breakpoint, for whatever reason the debugger wasn’t hooked in properly (other things that were confirm-ably working did not trigger breaks. A simple restart fixed this

now for the “good” news; I was finally able to retrieve a file with corrupted data and discovered a pattern. The data loss is somewhat consistent it happens to the last byte (ie; I sent 1024 bytes, the 1024-rth ditched and exactly 1023 bytes made it through.)

edit: I’m aware that this doesn’t sit well with the fact that the % of the file I receive is inconsistent, but I’m going to blame that on fiddler2 for now

hoorah!, making my buffer 1 byte larger than it’s intended to be has seemingly solved all my previous issues. I’ve got to run a bunch more tests to 100% this, but for now we may have a solution

1st Gratulation.
2nd. This is really strange.
I would assume it’s a bug in the WIFI module Firmware :think:

Solution persists using 1024 and 10240 size buffers, 102400 brings problems back (but offers no performance advantage so I’ll just ignore it.

to theorize: 102400 is the first buffer size that exceeds the baud rate, and so gets broken up in the driver software one would assume (and I may even test) that if you modified the driver to allow for an extra byte when sending multiple “sends” that it’d work for that size as well.