We are glad to let you know that GHIElectronics.NETMF.W5100.Http source code is now open for everyone on codeplex.com
I don’t have time right now to debug this, but I bet the problem is here:
// create the request timeout timer. This will kill the operation if it takes longer than specified by the Timeout property.
// The underlying socket will be closed to end the web request
using (Timer tmr = new Timer(new TimerCallback(OnRequestTimeout), null, m_timeout, System.Threading.Timeout.Infinite))
{
// Processes response from server. Request stream should already be there.
This is in GetResponse in HttpWebRequest.cs. The last line above is a comment about an assumption that is NOT valid. It can take seconds for the server to respond. Many servers respond in less than a second and everything works, but if the server takes longer ParseHttpResponse will throw.
@ Frogmore
I don’t know.
It looks like the default value for this timer is 100 seconds… which should be plenty of time to respond.
Guys, you can compare the source code to the original Http code that comes with .NET Micro Framwework to see the differences. This might give you some clues.
I believe I have a solution to the problem. ParseHTTPResponse() is reading the response before it is there and so it misses it. It seems that inStream.Read_HTTP_Line() can return a string of zero size, which will confuse the code. I added a simple do while to make sure there is data before proceeding, which seems to fix the problem. I don’t have to wait anymore and I don’t get all the headers in the response, like I used to sometimes.
// Parse the request line.
string line;
do
{
line = inStream.Read_HTTP_Line(maxHTTPLineLength).Trim();
Debug.Print("Read some stuff: " + line.Length.ToString());
} while (line.Length == 0);
I was able to copy all the files and simply rebuild them in VS. I then bumped the minor version number in the assembly information so my other project would use the newer assembly (otherwise it used the one in the SDK). I did not figure out how to use the debugger with two projects active, so I did it the old fashioned way 
I don’t know that this is the best solution to the problem, but it looks like it will work for me.
It looks like providing the source code is not a bad way to solve some problems.
I am glad it could help you out ;D
that is in HttpWebRequest.cs around about line 1450 -1460 is where ParseHTTPResoponse lives. when I am posting to SmartEnergyGroups.com I typically have to read 15 zero length strings before the header is available.
I spent some time looking at the .Net Micro Framework code and it looks like there is a difference in the socket implementation that W5100 library is using.
In HttpWebRequest.cs, the method ParseHTTPResponse() calls Read_HTTP_Line() on the inStream to get the next line of the Response. This is the same in NETMF.
In _InputNetworkStreamWrapper.cs, the method Read_HTTP_Line() calls RefillInternalBuffer() to get more data in the buffer. This is the same in NETMF.
In _InputNetworkStreamWrapper.cs, the method RefillInternalBuffer() gets the length of the stream, but if the stream length is 0 will set the readCount to 1 and then call Read on the stream with a count of 1. This too is the same in NETMF.
In NetworkStream.cs, the method Read() does some checking to see what is available, but only adjusts the count if available is greater than 0. This is the same in NETMF.
Unfortunately, the code for Socket.cs is not provided, so I can’t tell what the GHI WIZnet5100 code is doing. The NETMF code does the following:
return NativeSocket.recv(this, buffer, offset, size, (int)socketFlags, m_recvTimeout);
Which looks to me like it is a blocking call, at least until the timeout. Based on what I am seeing the WIZnet5100 code is not blocking. The NETMF documentation on Socket.Receive() is not very descriptive, but the .NET documentation says this:
If no data is available for reading, the Receive method will block until data is available, unless a time-out value was set by using Socket.ReceiveTimeout. If the time-out value was exceeded, the Receive call will throw a SocketException. If you are in non-blocking mode, and there is no data available in the in the protocol stack buffer, the Receive method will complete immediately and throw a SocketException. You can use the Available property to determine if data is available for reading. When Available is non-zero, retry the receive operation.
If you are using a connection-oriented Socket, the Receive method will read as much data as is available, up to the size of the buffer. If the remote host shuts down the Socket connection with the Shutdown method, and all available data has been received, the Receive method will complete immediately and return zero bytes.
So, I believe the Socket.Receive() implementation in the WIZnet5100 library is probably not quite right.
If the code for GHIElectronics.NETMF.W5100 were available, I might be able to tell if it is blocking and using the timeout on the native call to the W5100 receive function.
When I was creating the new HttpServer class, I did look at the InputNetworkStreamWrapper of the PK. And I did see the same issue
F.e. this line:
if(0 == RefillInternalBuffer()) return 0;
In ReadInternal method of InputNetworkStreamWrapper. What about previous received bytes before timeout or abortion? They just return 0, throwing away all received data. It should at least return retVal and you might want to catch the Timeout exception.
Any chance of getting the source to GHIELECTRONICS.NETMF.W5100 to see if it is making a blocking or non-blocking call? It might or might not be obvious from that code, or it might also require the native code that actually sets up and interacts with the WIZnet 5100. It would be good to fix this in a way that is likely to work correctly for all (or at least most) conditions.
I guess they are always blocking. There is no socket option to set it and there are no methods to do something async AFAIK
@ Frogmore
I can check it for you but please tell me what C# method exactly you want it to be check.
Dear GHI,
As I interpret this discussion, it appears there is a flaw in the ghi stack.
My question is: When will a patched lib be made available?
My project has suffered tremendously from this issue.
Great to see motion!
No need for multiple replies. Please see other thread you replied to. NETMF 4.2 is top priority now.
Thanks for offering to check.
In NetworkStream.cs, there is a call to Socket.Receive(buffer, offset, count, SocketFlags.None), which should be in Socket.cs. This should be a blocking call, i.e. it will only return 0 (for bytes read) if there is a timeout or the connection is closed, before the timeout. In my sample, this call returns 0 for bytes read each time I call it (actually I call GetResponse(), which calls ParseHTTPResponse(), which calls Read_HTTP_Line() which calls RefillInternalBuffer(), which calls Read() on the network stream, which finally calls Receive() on the socket), until the server has actually responded instead of blocking and waiting for the response the first time it is called.
In the NETMF PK this call results in the following call:
return NativeSocket.recv(this, buffer, offset, size, (int)socketFlags, m_recvTimeout);
Which certainly looks like it would block, given the Timeout parameter and a quick look at the native code.
Looking at my socket based code which works as expected, it appears that the W5100 implementation of Socket.cs does block (sometimes).
// Reusable buffer for receiving chunks of the document.
Byte[] buffer = new Byte[1024];
// Accumulates the received page as it is built from the buffer.
String page = String.Empty;
// Wait up to 30 seconds for initial data to be available. Throws an exception if the connection is closed with no data sent.
DateTime timeoutAt = DateTime.Now.AddSeconds(30);
while (serverSocket.Available == 0 && DateTime.Now < timeoutAt)
{
System.Threading.Thread.Sleep(100);
}
// Poll for data until 30-second timeout. Returns true for data and connection closed.
while (serverSocket.Poll(30 * c_microsecondsPerSecond, SelectMode.SelectRead))
{
// If there are 0 bytes in the buffer, then the connection is closed, or we have timed out.
if (serverSocket.Available == 0) break;
// Zero all bytes in the re-usable buffer.
Array.Clear(buffer, 0, buffer.Length);
// Read a buffer-sized HTML chunk.
Int32 bytesRead = serverSocket.Receive(buffer);
// Append the chunk to the string.
page = page + new String(Encoding.UTF8.GetChars(buffer));
}
// Return the complete string.
return page;
Notice how this code sample (which I got from the Socket Client Sample in the SDK) first waits for up to 30 seconds (with Thread.Sleep() to allow other threads to run) for there to be data available, then waits for up to another 30 seconds for the read to complete, only then (when it knows there is a data available) does it perform the Receive() on the socket. None of that should really be necessary if the Receive() call blocked like I believe it is supposed to. And sure enough, when I remove the Socket.Available() and Socket.Poll() checks and just Receive() from the socket it waits for the data. I have verified this by timing the call to Receive(). If I have waited for the data to be there before calling Recieve() the call takes just a few ms to return, but if I have not waited before making the call the call takes over 100ms to return. So, it seems that sometimes Socket.Receive() is blocking like it should. This is both in the emulator and with W5100 library on a Panda II with a FEZ Connect.
However, when reading from a NetworkStream (on the W5100 only) it seems that the Socket.Recieve() is not blocking like it should. Looking at the SDK code from the PK, it looks like this could happen if there is an exception on the socket handler. I believe this is in Library_spot_net_native_Microsoft_SPOT_Net_SocketNative::SendRecvHelper
(from spot_net_native_Microsoft_SPOT_Net_SocketNative.cpp)
which is what seems to do the actual receive on the socket.
@ Frogmore: thank you for the thorough answer but it is a little bit confusing to me. Could you please give me the specific method, the cs file and the line number that I need to check? I will give you one word answer (the method is blocking or not blocking) that’s all I can help with 
Since I don’t have the code to Socket.cs, I can’t give you the line number. The method probably looks like this:
public int Receive(byte[] buffer, int offset, int size, SocketFlags socketFlags)
{
if (m_Handle == -1)
{
throw new ObjectDisposedException();
}
return NativeSocket.recv(this, buffer, offset, size, (int)socketFlags, m_recvTimeout);
}
Which certainly looks like it should block. But, just as certainly, I can tell you that in my case (and others) it is not. That means we need to go to the next level down, which I have even less of an idea which file it is in, since your names might be different from what the PK uses.
Just to make sure I was on the right track, I tried something else. In NetworkStream.cs I modified Read() so that before calling Receive() on the socket, it waits (up to timeout) for Socket.Available to be non-zero. When I did that, the rest of the W5100.Http code started working as expected. So, I believe there is a problem in the Socket.Receive() code. This could be an issue in the C# code, the native code, or even the WIZnet 5100 implementation. The next step would be to look at the C# code. My guess is that it looks alot like the PK code, which calls:
NativeSocket.recv(this, buffer, offset, size, (int)socketFlags, m_recvTimeout)
This is the code that could have an issue. It might be returning 0 bytes received when it should be waiting for the data to show up.
@ Frogmore: Do I understand correctly that with your patches to the lib, a working HttpRequest/Response is in hand?
@ Frogmore:
Here is the sourceode of recv:
public static int recv(int socket, byte[] buf, int offset, int count, SocketFlags socketFlags, int timeout_ms)
{
byte s = (byte) socket;
int ret = 0;
poll(s, SelectMode.SelectRead, timeout_ms);
if(count > W5100.getSn_RX_RSR((byte)socket))
count = W5100.getSn_RX_RSR((byte)socket);
if (count > 0)
{
W5100.RecvDataProc(s, buf,offset, count);
W5100.RegisterWrite(s, W5100.SocketRegisters.CR, (byte)CR_val.Sn_CR_RECV);
/* +20071122[chungs]:wait to process the command... */
while (W5100.RegisterRead(s, W5100.SocketRegisters.CR) != 0) ;
/* ------- */
ret = count;
}
return ret;
}
And here is the bug.
This line:
poll(s, SelectMode.SelectRead, timeout_ms);
Should be:
poll(s, SelectMode.SelectRead, timeout_ms*1000);
Poll is expecting a number in microseconds.
public bool Poll (
int microSeconds,
SelectMode mode
)
The reason this is flakey is that the timeout is generally too short, but if the web server is fast enough the response will be there before the timeout.