HTTP response streams > 20K possible w/ Hydra, ENC28?

Archive
1

I’ve been having trouble with moderately-sized file downloads on my Hydra + ENC28 platform. I’m running the 2013 R2 firmware but this issue has been present since I started with 4.1.

I generally don’t have trouble with XML and JSON content using the System.Net.HttpWebRequest class. My response streams are only a few KB for the XML and JSON use cases. The issue shows up intermittently with response streams 20K and larger.

Example Symptom:


Bytes Read Now: 4096 Total: 4096
Bytes Read Now: 1970 Total: 6066
    #### Exception System.Net.Sockets.SocketException - CLR_E_FAIL (1) ####
    #### Message: 
    #### Microsoft.SPOT.Net.SocketNative::recv [IP: 0000] ####
    #### System.Net.Sockets.Socket::Receive [IP: 0018] ####
    #### System.Net.Sockets.NetworkStream::Read [IP: 0062] ####
    #### System.Net.InputNetworkStreamWrapper::ReadInternal [IP: 00d7] ####
    #### System.Net.InputNetworkStreamWrapper::Read [IP: 000d] ####
    #### HttpClientSample.MyHttpClient::PrintHttpData [IP: 00a3] ####
    #### HttpClientSample.MyHttpClient::Main [IP: 0039] ####
    #### SocketException ErrorCode = 10060
    #### SocketException ErrorCode = 10060
A first chance exception of type 'System.Net.Sockets.SocketException' occurred in Microsoft.SPOT.Net.dll
    #### SocketException ErrorCode = 10060
An unhandled exception of type 'System.Net.Sockets.SocketException' occurred in Microsoft.SPOT.Net.dll

I have reproduced this symptom using the Microsoft NETMF example found in \My Documents\Microsoft .NET Micro Framework 4.3\Samples\HttpClient.

Is this a generic issue with the NETMF libraries? Unique to the Hydra+ENC28 combination or perhaps related to my environment and boards? How to discern? Can any fool build the NETMF libraries from scratch to debug further?

To reproduce in your environment:

  1. Take the Microsoft HttpClient example and open in Visual Studio 2008 or VS 2010. On my computer this was found in C:\Users\foo\Documents\Microsoft .NET Micro Framework 4.3\Samples\HttpClient

  2. Edit the properties for the HttpClient Project and set the Target Framework to “.NET Micro Framework 4.2”. Associate with your connected GHI Gadgeteer kit.

  3. Edit HttpClient.cs and add a line that calls PrintHttpData() with the following URL:
    htttp://netmf.codeplex.com/downloads/get/500745

i.e.


            PrintHttpData("http://netmf.codeplex.com/downloads/get/500745", null);
2

Any server but the Codeshare URL with download illustrates the issue.

In my testing I found that local servers (low latency) perform much better. Far-away servers (high latency) increase the probability of an error. Packet tracing revealed the ENC28 seems (my speculation) overrun with TCP ACKs, which it ignores.

In addition to this CLR_E_FAIL failure mode, we generally observe sub-par TCP performance on non-local HTTP servers. Probably related.

3

Thanks Andre for taking a look. I should have clarified my issue is the TCP socket always times out, when 20K or larger streams are encountered, irrespective of the timeout.

The timeout is adjustable as you know. Turn it up to 50 seconds (or 500) and you’ll get the same result. e.g. in HttpClient.cs find DownloadHttpData() and the line that says respStream.ReadTimeout = 5000. Change it to respStream.ReadTimeout = 50000 for a 50 second timeout.

I will pull in the core source as you suggested. I expect that is where the TCP state machine is? My hunch is that it doesn’t get the TCP ACK frame from the ENC28 chip but I may need to hook up the logic analyzer to prove/disprove that.

4

I Would try to set the timeout this way instead of what you mentioned as I observed that does nothing to the socket…


respStream.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReceiveTimeout,50000);

try it and let us know…
Cheers,
Jay.

5

JayJay thanks for taking a look. What assembly reference do I need to add for that to work? System.IO.Stream does not contain a definition for ‘SetSocketOption’.

6

That’s the listening Socket, you know the socket that accepts the request…

7

In this example, the System.Net.HttpWebRequest class is being used as a client for a web server. We aren’t listening for or accepting any connections. This example also doesn’t bind our own Socket. This is probably implemented in HttpWebRequest.

Do you have the SDK installed? If so, use Windows Explorer and browse to the .NET Micro Framework samples directory and double-click the HttpClient.sln

Incidentally, the response timeout setting does appear to work in my case.

8

Just curious but can one use async socket IO on .Net MF? If so then it might be interesting to write a test program using async reads to see if that works, my reasoning being that by definition theres no such thing a timeout in async so it might run through a different code path, possibly bypassing the code causing your issue .

B

9

No Async implementation in MF

10

Really? That’s frankly astonishing. Are there any kinds of event primitives (as in operating system like primitives: Mutex, Semaphore, Event etc)?

Does the framework support threading?

11

threading. yes
mutex yes with object lock
semaphore no
event yes

12

Hi.

I’ve had the same problem with my G120 + ENC. Especially when using the device behind a router.

I solved it by instead of using the HttpRequest and Reponse libraries, opening up a raw socket and read data from a networkstream.


						while (totalBytesRead < contentLength)
						{
							if (inputStream.Length > 0)
							{
								bytesRead = inputStream.Read(buffer, 0, buffer.Length);
								totalBytesRead += bytesRead;
								Output.WriteLine("BytesRead: " + bytesRead + " TotalBytesRead: " + totalBytesRead);
							}

						}

This has been very robust and I have no problemsdownloading files of around 500 kB.

13

I’ve pulled down the Porting Kit (PK) source and debugged as far as the Native Socket implementation.

It seems my NETMF environment goes into a persistent error condition after encountering a timeout on an HttpWebRequest.GetResponse().GetResponseStream().Read()

Subsequent calls (for different urls) to HttpWebRequest.GetResponse() seem to block in Socket.Connect() or Microsoft.SPOT.Net.SocketNative.poll().

What is my next step to debug? Has anyone run across this stack trace (see screen shot) before? It’s pretty easy to reproduce with the HttpClient sample application from the 4.2 NETMF SDK.

The key to reproduction is talking to servers on the Internet, not your LAN. When TCP segments are dropped and retransmission is needed, the NETMF TCP stack seems prone to lockup. I’ve got packet sniffer output where TCP segments are retransmitted but lost leading to the Stream read() timeout. The error recovery doesn’t seem robust.

If anyone has the native (C++) build environment up and running, I’d be very happy to provide you a reproducible test case.

14

Anybody with a Hydra, Cerberus or Cerbuino out there who is using the ENC28?

If so, would you see if this LWIP TCP lockup issue reproduces on your bench? I am planning to pay GHI and/or Microsoft to fix this, but I need to see if the issue is reproducible first.

https://drive.google.com/file/d/0B1gfDjx7MnijNWlwdjZHR1BveFU/edit?usp=sharing

I’ve seen it in the last 2012 GHI-assembled NETMF firmware, the 2013, 2013 R2 and 2013 R3 firmware releases.

15

yes I do see wired error message when i’m debugging using MFDeploy… something to do with a timeout…until it locks up…
try Debuggin with MFDeploy instead of VS it will give you more info…

Cheers,
Jay.

16

Hi,

I am using 2014 R2 Beta 3 (4.3.2.0)
and could not reproduce the problem. Is it happened 100% or just randomly?

Below is my program with the link http://netmf.codeplex.com/downloads/get/500745 as you expected.


using GHI.Networking;

using System;
using System.IO;
using System.Text;
using System.Threading;
using System.Net;
using System.Net.Sockets;
using System.Security;
using Microsoft.SPOT;
using Microsoft.SPOT.Hardware;
using Microsoft.SPOT.Net;
using Microsoft.SPOT.Net.NetworkInformation;

namespace Networking
{
    public class Program
    {
        static EthernetENC28J60 netif;
        public static void Main()
        {
            DoNetwork();
        }       
        static bool useDHCP = true; //9E-12-81-79-3B-E5
        private static void DoNetwork()
        {
            int a = 0;
            while (a == 0)
            {
                Thread.Sleep(100);
            }
            netif = new EthernetENC28J60(SPI.SPI_module.SPI1, (Cpu.Pin)(32 + 13), (Cpu.Pin)(32 + 8), (Cpu.Pin)(32 + 9)); // Hydra socket 3
            netif.PhysicalAddress = new byte[] { 0x9E, 0x12, 0x81,0x79,0x3B,0xE5};
            Debug.Print("Opening interface...");
            netif.Open();
            if (useDHCP == false)
            {
                Debug.Print("Set static IP...");
                netif.EnableStaticIP("x.x.x.x", "x.x.x.x", "x.x.x.x");
                netif.EnableStaticDns(new string[] { "x.x.x.x", "x.x.x.x" });

            }
            else
            {
                Debug.Print("Get DHCP IP...");
                //// DHCP IP
                if (netif.IsDhcpEnabled == false)
                {
                    netif.EnableDhcp();
                    netif.EnableDynamicDns();
                }
            }
            NetworkChange.NetworkAvailabilityChanged += new NetworkAvailabilityChangedEventHandler(NetworkChange_NetworkAvailabilityChanged);
            NetworkChange.NetworkAddressChanged += new NetworkAddressChangedEventHandler(NetworkChange_NetworkAddressChanged);
            int cnt = 0;
            while (isReady == false)
            {
                Debug.Print("Waiting for networ ready " + (cnt++));
                Thread.Sleep(250);
            }
            while (true)
            {
                PrintHttpData("http://netmf.codeplex.com/downloads/get/500745");
                Thread.Sleep(1000);
            }
        }
        static void NetworkChange_NetworkAvailabilityChanged(object sender, NetworkAvailabilityEventArgs e)
        {
            Debug.Print("Network has changed! ");
        }
        static bool isReady = false;
        static void NetworkChange_NetworkAddressChanged(object sender, EventArgs e)
        {
            Debug.Print("New address for the Network Interface ");
            Debug.Print("Is DhCp enabled: " + netif.NetworkInterface.IsDhcpEnabled);
            Debug.Print("Is DynamicDnsEnabled enabled: " + netif.NetworkInterface.IsDynamicDnsEnabled);
            Debug.Print("NetworkInterfaceType " + netif.NetworkInterface.NetworkInterfaceType);
            Debug.Print("Network settings:");
            Debug.Print("IP Address: " + netif.NetworkInterface.IPAddress);
            Debug.Print("Subnet Mask: " + netif.NetworkInterface.SubnetMask);
            Debug.Print("Default Gateway: " + netif.NetworkInterface.GatewayAddress);
            Debug.Print("Number of DNS servers:" + netif.NetworkInterface.DnsAddresses.Length);
            for (int i = 0; i < netif.NetworkInterface.DnsAddresses.Length; i++)
                Debug.Print("DNS Server " + i.ToString() + ":" + netif.NetworkInterface.DnsAddresses[i]);
            Debug.Print("------------------------------------------------------");
            if (netif.IPAddress != "0.0.0.0")
            {
                isReady = true;
            }
        }
        public static void PrintHttpData(string url)
        {
            // Create an HTTP Web request.
            HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
            request.KeepAlive = true;



            // Get a response from the server.
            WebResponse resp = null;
            request.Reset();
            try
            {
                resp = request.GetResponse();
            }
            catch (Exception e)
            {
                Debug.Print("Exception in HttpWebRequest.GetResponse(): "
                + e.ToString());
            }

            // Get the network response stream to read the page data.
            if (resp != null)
            {
                Stream respStream = resp.GetResponseStream();

                string page = null;
                byte[] byteData = new byte[4096];
                char[] charData = new char[4096];
                int bytesRead = 0;

                Decoder UTF8decoder = System.Text.Encoding.UTF8.GetDecoder();

                int totalBytes = 0;

                // allow 5 seconds for reading the stream
                respStream.ReadTimeout = 5000;

                // If we know the content length, read exactly that amount of
                // data; otherwise, read until there is nothing left to read.
                if (resp.ContentLength != -1)
                {
                    for (int dataRem = (int)resp.ContentLength; dataRem > 0; )
                    {
                        Thread.Sleep(500);
                        bytesRead = respStream.Read(byteData, 0, byteData.Length);

                        if (bytesRead == 0)
                        {
                            Debug.Print("Error: Received " +
                            (resp.ContentLength - dataRem) + " Out of " +
                            resp.ContentLength);
                            break;
                        }

                        dataRem -= bytesRead;

                        // Convert from bytes to chars, and add to the page
                        // string.
                        int byteUsed, charUsed;
                        bool completed = false;

                        totalBytes += bytesRead;

                        UTF8decoder.Convert(byteData, 0, bytesRead, charData, 0,
                        bytesRead, true, out byteUsed, out charUsed,
                        out completed);

                        page = page + new String(charData, 0, charUsed);

                        // Display the page download status.
                        Debug.Print("Bytes Read Now: " + bytesRead +
                        " Total: " + totalBytes);
                    }

                    page = new String(System.Text.Encoding.UTF8.GetChars(byteData));
                }
                else
                {
                    // Read until the end of the data is reached.
                    while (true)
                    {
                        // If the Read method times out, it throws an exception,
                        // which is expected for Keep-Alive streams because the
                        // connection isn't terminated.
                        try
                        {
                            Thread.Sleep(500);
                            bytesRead = respStream.Read(byteData, 0, byteData.Length);
                        }
                        catch (Exception)
                        {
                            bytesRead = 0;
                        }

                        // Zero bytes indicates the connection has been closed
                        // by the server.
                        if (bytesRead == 0)
                        {
                            break;
                        }

                        int byteUsed, charUsed;
                        bool completed = false;

                        totalBytes += bytesRead;

                        UTF8decoder.Convert(byteData, 0, bytesRead, charData, 0,
                        bytesRead, true, out byteUsed, out charUsed,
                        out completed);

                        page = page + new String(charData, 0, charUsed);

                        // Display page download status.
                        Debug.Print("Bytes Read Now: " + bytesRead +
                        " Total: " + totalBytes);
                    }

                    Debug.Print("Total bytes downloaded in message body : "
                    + totalBytes);
                }

                // Display the page results.
                Debug.Print(page);

                // Close the response stream.
                respStream.Dispose();
                resp.Close();
                resp.Dispose();
                request.Dispose();
            }

        }
        
    }
}

Please try it and share to me the result.
Thanks

17

Hi Andre.m

if we can, wow did you change it? please
:smiley:

18

thanks