I wanted to add firmware file upload to my device using http and W5100. What i saw is that the file i receive and save to SD is a little different from one I’m sending. When i run Diff on both the original and the result files i see that differences occure only in a few places. I have analyzed the TCP packets in Wireshark and see why. The file content is sent to FEZ using multiple TCP packets (segments). Normaly this shouldn’t matter because with HttpRequest we are using higher layer where those packets should be merged. The effect however is that somehow the packets overlap. See the attatched image showing comparision of the original and transfered file.
Did anyone face this problem or is able to reproduce it?
I have tested this using Fiddler, you can see the screenshots in the attachment. When i send a file upload request (using plain text to make it more readable) of Content-Length 2360 i am able to retrieve the file without any modification. After I add extra byte to the request i retrieve the file malformed. This is my temporary code that saves the file to SD.
protected bool FirmwareUpload(HttpListenerContext context)
{
var contentLength = context.Request.ContentLength64;
var contentType = context.Request.Headers.GetValues(HttpKnownHeaderNames.ContentType)[0];
var boundary = "--" + contentType.Split(';')[1].Split('=')[1] + "--";
Debug.Print(boundary);
using (var stream = context.Request.InputStream)
{
var headers = new WebHeaderCollection();
var line = ((InputNetworkStreamWrapper)stream).Read_HTTP_Line(1000);
contentLength -= line.Length + 2;
line = ((InputNetworkStreamWrapper)stream).Read_HTTP_Line(1000);
contentLength -= line.Length + 2;
headers.Add(line);
line = ((InputNetworkStreamWrapper)stream).Read_HTTP_Line(1000);
contentLength -= line.Length + 2;
headers.Add(line);
stream.ReadByte();
stream.ReadByte();
contentLength -= 4 + boundary.Length + 2;
var contentDisposition = headers.GetValues(HttpKnownHeaderNames.ContentDisposition)[0].Split(';');
var fileName = contentDisposition[2].Split('=')[1].Trim('\"');
Debug.Print(fileName);
var buffer = new byte[255];
using (var fileStream = File.Create(@ "\SD\webroot\" + fileName, 80))
{
for (var i = 0; i < contentLength; i += buffer.Length)
{
var count = System.Math.Min(buffer.Length, (int)(contentLength - i));
stream.Read(buffer, 0, count);
fileStream.Write(buffer, 0, count);
}
fileStream.Flush();
fileStream.Close();
}
Debug.Print("Finished");
HttpServer.ReturnMessage(context, "OK", HttpStatusCode.OK);
stream.Close();
}
return true;
}
@ ddurant - yes, http request (OSI application level) is sent using multiple tcp packets (OSI transport layer). While we operate on application level the request/response should be complete and in correct order. The problem here is for some reason the packets get mixed up and presented to application layer corrupted.
@ Joe - I’m using the W5100.Http source in my project from the start (i had to make it smaller in order to fit into my flash) so i started playing with it and I made some chages. Now my files are transported and saved correctly. I have uploaded the patch to codeplex. The problem is I don’t know why this works as your solution wasn’t much different. I don’t think you should include my patch unless you find time to investigate this issue. I changed this in _InputNetworkStreamWrapper.cs:
private int RefillInternalBuffer()
{
#if DEBUG
if (m_dataStart != m_dataEnd)
{
//Microsoft.SPOT.Debug.Print("Internal ERROR in InputNetworkStreamWrapper");
m_dataStart = m_dataEnd = 0;
}
#endif
// m_dataStart should be equal to m_dataEnd. Purge buffered data.
m_dataStart = m_dataEnd = 0;
// Read up to read_buffer_size, but less data can be read.
// This function does not try to block, so it reads available data or 1 byte at least.
int readCount = (int)m_Stream.Length;
if ( readCount > read_buffer_size )
{
readCount = read_buffer_size;
}
else if (readCount == 0)
{
readCount = 1;
}
m_dataEnd = m_Stream.Read(m_readBuffer, 0, readCount);
return m_dataEnd;
}
to this:
private int RefillInternalBuffer(int readCount = 0)
{
#if DEBUG
if (m_dataStart != m_dataEnd)
{
//Microsoft.SPOT.Debug.Print("Internal ERROR in InputNetworkStreamWrapper");
m_dataStart = m_dataEnd = 0;
}
#endif
// m_dataStart should be equal to m_dataEnd. Purge buffered data.
m_dataStart = m_dataEnd = 0;
// Read up to read_buffer_size, but less data can be read.
// This function does not try to block, so it reads available data or 1 byte at least.
if (readCount == 0)
readCount = (int)m_Stream.Length;
if (readCount > read_buffer_size)
readCount = read_buffer_size;
if (readCount > 0)
m_dataEnd = m_Stream.Read(m_readBuffer, 0, readCount);
return m_dataEnd;
}
This way the method is not waiting for 1 byte when no is available but for how many is needed. Don’t know why this matters.
@ WouterH - I think that topic and my problem are all about the same issue. I read it and it seems they ended up with a fix though… I also think that the problem is when calling Read on socket when there is no data available (the next TCP packet of the request has not been processed/received yet).
@ Architect - Good idea, I made a test on Cobra but it works out-of-the-box. Here is the source code:
public static class Program
{
static HttpListener _httpListener;
static void Main()
{
var network = NetworkInterface.GetAllNetworkInterfaces()[0];
network.EnableStaticIP("10.0.0.223", "255.255.0.0", "10.0.0.249");
_httpListener = new HttpListener("http", 80);
_httpListener.Start();
Debug.Print("Listening for requests...");
while (true)
{
try
{
var context = _httpListener.GetContext();
if (context == null)
continue;
Debug.Print(context.Request.HttpMethod + " " + context.Request.Url.OriginalString);
ReturnRequestAsResponse(context);
}
catch (Exception e)
{
Debug.Print("Failed process request. " + e.Message);
}
finally
{
Debug.GC(true);
}
}
}
static void ReturnRequestAsResponse(HttpListenerContext context)
{
var contentLength = context.Request.ContentLength64;
using (var stream = context.Request.InputStream)
{
var buffer = new byte[512];
context.Response.StatusCode = (int)HttpStatusCode.OK;
context.Response.ContentType = "text/plain; charset=utf-8";
using (var responseStream = context.Response.OutputStream)
{
while (contentLength > 0)
{
var dataToRead = System.Math.Min(buffer.Length, (int)contentLength);
var dataRead = stream.Read(buffer, 0, dataToRead);
if (dataRead > 0)
responseStream.Write(buffer, 0, dataRead);
contentLength -= dataRead;
}
stream.Close();
}
}
}
}
Just use Fiddler to e.g. POST some data to the device with a long payload. You should receive a response with the same payload. Can someone run this code on Panda + Connect shield ? You should receive a corrupted response. I will be very grateful.