Sending large files over wifi

I’m having trouble transferring large(10MB) files with the rs171 wifi module. Since I cannot hold the object in a single byte[] or string, I’m not sure how to proceed with transfering the file over (the file is stored on a sd card module btw)

I’m not sure but I use this way to get a lage file


        Dim SDfile As System.IO.FileStream

       SDfile = mvarSD.OpenWrite("myfile.txt")
       SDfile.Write(response.RawContentBytes, 0, response.RawContentBytes.Length)
       SDfile.Close()

@ jstone05 - Read smaller blocks from the SD and send them. Read… write…read… write…

That’s the approach I’m taking currently


using (LargeBuffer bytes = new LargeBuffer(size))
{
         file.Read(bytes.Bytes, 0, size);
         request.Response.Send(bytes.Bytes);
}
 while (0 < dataRemaining)
{
         size = getBufferSize(ref dataRemaining);
         using (LargeBuffer bytes = new LargeBuffer(size))
          {
                    file.Read(bytes.Bytes, 0, size);
                    wifi_RN171.Send(bytes.Bytes);
          }
 }

But I’m not getting as much data as I’m sending; anywhere from 5-20% of the data is either being lost or not arriving properly, so the download never completes.

I was hoping to find an alternative approach to avoid the issue entirely, but if there’s a way to repair that problem that’d be great too.

Hi,
I hat success with code like this:

 
byte[] b = new byte[2048];
FileStream iFile;
iFile = new FileStream(sFile, FileMode.Open, FileAccess.Read);
Int lRemain = iFile.Length;

while (lRemain > 0)
{
       if (lRemain < 2048)
       b = new byte[lRemain];

       iFile.Read(b, 0, b.Length);
       // do something with the contents oft he buffer;
       wifi_RN171.Send(b);
       lRemain -= b.Length;
       // eventually add a thread.sleep(200); or another time here
}
iFile.Close();
 

How do you send the header data with this method? or is that simply not required?

Hi,
you did not yet tell, what you want to do. Do you want to transfer a file from a Gadgeteer device to a PC (or what else?). Which of the two shall initiate the transfer? I think that you need a protocol like ftp.
There are some examples in CodeShare:
https://www.ghielectronics.com/community/codeshare/entry/836
and
https://www.ghielectronics.com/community/codeshare/entry/662

@ RoSchmi - My bad; I’m using the wifi module as an http server currently. The PC hits a link on a webpage, and the transfer begins. this is the entirety of the current method I’m using.

private void getLogs(HttpStream request)
        {
            var SDCard = sdCard.GetStorageDevice();
            var file = SDCard.OpenRead(@ "\log.csv");
            request.Response.HeaderData["nope"] = "application/octet-stream";
            //having a bug here that necessitates an additional header
            request.Response.HeaderData["Accept-Ranges"] = "bytes";
            request.Response.HeaderData["Content-Type"] = "application/octet-stream";
            request.Response.HeaderData["Content-Disposition"] = "attachment";
            request.Response.HeaderData["Content-Length"] = file.Length.ToString();
            request.Response.HeaderData["Connection"] = "close";
            request.Response.HeaderData["Cache-Control"] = "no-cache";
            request.Response.StatusCode =   GTM.GHIElectronics.HttpResponse.ResponseStatus.OK;
            var errr = file.Length.ToString();
            var dataRemaining = (int)file.Length;
            var size = getBufferSize(ref dataRemaining);
            using (LargeBuffer bytes = new LargeBuffer(size))
            {
                file.Read(bytes.Bytes, 0, size);
                request.Response.Send(bytes.Bytes);
            }
            while (0 < dataRemaining)
            {
                size = getBufferSize(ref dataRemaining);
                using (LargeBuffer bytes = new LargeBuffer(size))
                {
                    file.Read(bytes.Bytes, 0, size);
                    wifi_RN171.Send(bytes.Bytes);
                }
            }
        }

Sorry, the code I used was in conjunction with ftp protocol. I don’t know if it can be used in conjunction with a http Server. There will be other forum members who can help you.

You could replace this:

var dataRemaining = (int)file.Length;
             var size = getBufferSize(ref dataRemaining);
             using (LargeBuffer bytes = new LargeBuffer(size))
             {
                 file.Read(bytes.Bytes, 0, size);
                 request.Response.Send(bytes.Bytes);
             }
             while (0 < dataRemaining)
             {
                 size = getBufferSize(ref dataRemaining);
                 using (LargeBuffer bytes = new LargeBuffer(size))
                 {
                     file.Read(bytes.Bytes, 0, size);
                     wifi_RN171.Send(bytes.Bytes);
                 }
             }

with this:

var dataRemaining = (int)file.Length;
var buffer = new byte[10 * 1024]; // adjust size to your needs
while (dataRemaining > 0)
{
   int cnt = file.Read(buffer, 0, System.Math.Min(buffer.Length, dataRemaining));
   request.Response.Send(buffer, 0, cnt); // assuming this overload with offset and size exists
   dataRemaining -= cnt;
}

By this the file is read and sent in chunks of 10k. This saves a lot of memory and avoids using LargeBuffer
I’m nut sure why you read the file twice and sent it to Response and separately to wifi directly.
I also don’t know what getBufferSize does.

@ Reinhard Ostermeier - Thanks a lot, I’ll see if that improves my results. When I tried sending it through the response in a loop before I ended up getting extra header data with each pass, so I opted to send it directly after a single response.

largebuffer and getBufferSize are just remnants from older iterations/attempts I’ll phase them out after I adapt the code you provided.

Well if Response send sends a header each time, then it might not work.
But I think sending the data directly would not work ether.
Check if Response has some more methods or properties.

@ Reinhard Ostermeier - There’s not much there; of course it has the standard methods like toString and GetHashCode. Beyond that there’s only send (which seems to have no overload and can only accept a single byte[]). Header-data which is of course used to set header data, and StatusCode used to return statuses.

Then I fear it’s not possible to send anything bigger than a single byte array as data with the response.
The maximum byte array you can allocate is about 760k, if I remember correctly.
But depending on memory fragmentation you might not be able to allocate it.

I’m sure there are a lot of HTTP web server apps in code share.
Have you checked?
You cold also provide files via FTP:
There is a improved FTP server from me in code share.
If you add the file links as FTP links in your web site, the difference should not be to big.

I can actually allocate ~3MB of data with largebuffer, without any problems, but that’s not exactly a hard-cap, so we’re running into a problem where the file will outgrow that (be it by 10KB or 10MB) and trying to download that crashes the system (more or less)

I’ve glanced over some of the HTTP code shares, but I haven’t really looked into the FTP option because based on the functions available to me I’m not really sure how to get it working. I’ll continue searching as before, I’m just disappointed that what should be a simple task is giving me so much grief

I just checked the RN171 Gadgeteer driver.
So it’s not a classic Socket Network.
It might even work to use the send method.
By this I would try to send the 1st part with
request.Response.Send(buffer)
and the following parts with
wifi_RN171.Send(buffer);

like this:

var dataRemaining = (int)file.Length;
 var buffer = new byte[10 * 1024]; // adjust size to your needs
if (dataRemaining < buffer.Length)
{
    buffer = new byte[dataRemaining];
}
int cnt = file.Read(buffer, 0, dataRemaining.Length);
request.Response.Send(buffer);
dataRemaining -= cnt;
while (dataRemaining > 0)
{
    if (dataRemaining < buffer.Length)
    {
        buffer = new byte[dataRemaining];
    }
    cnt = file.Read(buffer, 0, dataRemaining.Length);
    request.Response.Send(buffer); // assuming this overload with offset and size exists
    dataRemaining -= cnt;
}

Give it a try.
Can’t promise it will work.

@ Reinhard Ostermeier - This brings be back to where I’ve been for a while (albeit a lot cleaner) in that this [em]mostly[/em] works, the only problem with this method is that I always come up a few bytes short of the file size. This causes the download to fail completely and leaves my browser in download limbo (ex: 95% with no more data coming in)

Also worth sharing is that Fiddler fails to see these lost bytes as well, the only thing supporting the idea that all bytes are being sent is the code

the code Reinhart posted also sends the entire buffer even on the last few bytes of the file. the size of the send should be the cnt variable. sending the extra junk at the end of the file might be causing funny things to happen at receive end.

To avoid this I allocate a smaller buffer for the last chunk.
The RN171 seems not to have overloads for sending only a part of the buffer.

@ jstone05 - I do not see why there should be too less bytes sent!?
May be the HTTP protocol requires something else sent as well.

But …
this is how HttpRrespons.Send is implemented:

/// <summary>
    /// Sends a response.
    /// 
    /// </summary>
    /// <param name="document">The body of the response.</param>
    public void Send(byte[] document)
    {
      byte[] bytes = Encoding.UTF8.GetBytes(this.HeaderData.ToString());
      this._stream.Write(bytes, 0, bytes.Length);
      this._stream.Write(document, 0, document.Length);
    }

WiFi_RN171 internally uses a Serial class which is given to the HttpStream to send it’s data to. By this it should work to send the remaining data directly by the wifi Send method.

The Serial class has a Flush method.
But it looks like Flush is never called, and you can’t access it from external.

What exactly is the length of the file in your example? What is the original lenth of the buffer and how many Bytes are missing?
Furthermore i would try and see what happens if you send “/r/n” after the last Byte of the file.