Network problems with G120 and ENC28

Hello,

I have a huge problem with my G120. I am hosting a TCP server which stops working after ~4000 connections.

My solution is very large. I have created a short version of the network relevant code.

 public static class Program
    {
        private static readonly byte[] Buffer = new byte[1024];
        private static Socket _serverSocket;
        private static int _connections;

        public static void Main(string[] args)
        {
            var networkInterface = new EthernetENC28J60(SPI.SPI_module.SPI2, GHI.Pins.G120.P1_17, GHI.Pins.G120.P0_5, GHI.Pins.G120.P1_14);
            networkInterface.Open();

            networkInterface.EnableStaticIP("192.168.1.15", "255.255.255.0", "192.168.1.1");

            Program._serverSocket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
            Program._serverSocket.Bind(new IPEndPoint(IPAddress.Any, 80));
            Program._serverSocket.Listen(10);

            var workerThread = new Thread(Program.Listen);
            workerThread.Start();

            while(true)
            {
            	Debug.Print("Tick! " + Program._connections);
                Thread.Sleep(10000);
            }
        }

        private static void Listen()
        {
        	while (true)
            {
                try
                {
                    Socket clientSocket = Program._serverSocket.Accept();

                    Program._connections++;

                    using (clientSocket)
                    {
                        if (clientSocket.PollRead(3000))
                        {
                            if (clientSocket.Available > Program.Buffer.Length)
                            {
                                Debug.Print("Too much data!");
                            }
                            else
                            {
                                int requestLength = clientSocket.Receive(Program.Buffer);
                                if (requestLength > 0)
                                {
                                    // TODO...
                                }
                            }
                        }
                    }

                    Debug.Print("Client served...");
                }
                catch (Exception exception)
                {
                    Debug.Print(exception.Message);
                }
            }
        }
    }

The above code runs a few days and then stops working. The G120 is still working (I get the Debug messages) but I cannot connect to the device anymore. The connection count shows 3000-4000 successful connections.

The network server also stops working if I create a lot of async connections from my PC.

A few other details:

  • I am using the new SDK (NETMF and Gadgeteer Package 2014 R2).
  • I have updated the assembly references.
  • The firmware of the G120 is up to date.
  • I am using the .NET micro framework 4.3.

I hope you can help me to solve this issue.

Christian

Another interesting thing is that the same code runs without any problems on a FEZ Panda 2 with a FEZ Connect shield. I think this indicates that the network infrastructure cannot be the problem.

The hardware initialization is different but the loop which accepts the connections is the same.

What do you mean exactly?

Ahh OK.
I think I will never reach the limits because I have only one socket listening and only accept new clients after the previous one has disconnected.

Can be the backlog of 10 a problem?

I searched within this forum and found some related posts.

https://www.ghielectronics.com/community/forum/topic?id=12342&page=1
https://www.ghielectronics.com/community/forum/topic?id=9929

These two posts are describing exactly my problem. It seems that the bug is alredy fixed with a new SDK or isn’t it?

@ ChrisK - Are you sending any data to the device when you open a connection?

@ andre.m - Those network limits have changed slightly for 4.3

Edit: just checked the 4.3 LWIP options,they have changed. I guess this is why GHI removed the limits documentation, it would be nice if they rather just updated it.

@ John: Yes, I use the Chrome browser to send a HTTP request to the device. The full version of the code parses the request and sends data (response) back.

It seems to crash earlier if I hold the F5 key which sends a lot of requests and drops the old connections. In this case I get some SocketExceptions with error code 10050. These exceptions are catched by me but printed to the debug window. The IP of the remote endpoint is also broken after the SocketException has been thrown. The value is nonesense.

Running the Garbage Collector after each request seems to increase to possible number of working connections.

Another interesting thing is that the socket crashes after a few connections if the connection is routed through NAT.

May changes at the protocol headers during routing causing this problem?

A have a port redirection configured at my router which allows me to access the G120 over the internet.

I know wireshark but I am not an protocol expert. I can try to create a dump and post it if you want.

@ ChrisK - Can you add some code that reads that data and still shows the error reliably? Even if it just reads and discards it.

@ John: Here is a complete solution which has the described problem.

I have started the program with an attached debugger using VS2012. Then I opened a browser and hit F5 as fast as possible. The response was a valid JSON document but after 308 connections, the server socket dies. The main loop is still alive.

Another interesting thing is that the remote endpoint address is broken after the exception. I have added a screenshot for this.

[title]The code[/title]


using System;
using Microsoft.SPOT;

namespace SocketTest
{
    using System.Net;
    using System.Net.Sockets;
    using System.Text;
    using System.Threading;

    using GHI.Networking;

    using Microsoft.SPOT.Hardware;

    public class Program
    {
        private static EthernetENC28J60 _networkInterface;

        private static Thread _workerThread;

        private static Socket _serverSocket;

        private static readonly byte[] _receiveBuffer = new byte[1024];

        private static int _clientConnections;

        public static void Main()
        {
            Program.InitializeNetworkHardware();

            Program._serverSocket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
            Program._serverSocket.Bind(new IPEndPoint(IPAddress.Any, 80));
            Program._serverSocket.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.NoDelay, true);
            Program._serverSocket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true);
            Program._serverSocket.Listen(10);

            Program._workerThread = new Thread(Program.Listen);
            Program._workerThread.Start();
            
            while (true)
            {
                Debug.Print("I am alive (at least " + Program._clientConnections + " client connections)!");
                Thread.Sleep(5000);
            }
        }

        private static void InitializeNetworkHardware()
        {
            Debug.Print("Enabling ENC28 module.");
            Program._networkInterface = new EthernetENC28J60(SPI.SPI_module.SPI2, GHI.Pins.G120.P1_17, GHI.Pins.G120.P0_5, GHI.Pins.G120.P1_14);

            Debug.Print("Opening ENC28 module.");
            Program._networkInterface.Open();

            Debug.Print("Configuring ENC28 module.");
            Program._networkInterface.EnableStaticIP("192.168.1.15", "255.255.255.0", "192.168.1.1");

            // Wait 10 seconds to ensure that the network hardware is fully initialized. The first UDP broadcasts will
            // fail if we not wait!
            Debug.Print("Waiting for ENC28 module.");
            Thread.Sleep(10000);
            Debug.Print("ENC28 module initialized.");
        }

        private static void HandleClient(Socket client)
        {
            if (client.Available == 0)
            {
                if (!client.Poll(5000 * 1000, SelectMode.SelectRead))
                {
                    return;
                }

                if (client.Available == 0)
                {
                    return;
                }

                if (client.Available > Program._receiveBuffer.Length)
                {
                    Debug.Print("Received data exceeds the buffer size.");

                    return;
                }

                int requestLength = client.Receive(Program._receiveBuffer);

                Debug.Print("Received a request.");

                string message = Program.ParseAsciiString(Program._receiveBuffer, requestLength);

                // TODO: Do something with the message...

                if (!client.Poll(5000 * 1000, SelectMode.SelectWrite))
                {
                    Debug.Print("Could not send data to the client.");
                    return;
                }

                string body = "{\"clients\":" + Program._clientConnections + "}";

                client.Send(Encoding.UTF8.GetBytes("HTTP/1.1 200 OK\r\n"));
                client.Send(Encoding.UTF8.GetBytes("Access-Control-Allow-Origin: *\r\n"));
                client.Send(Encoding.UTF8.GetBytes("Connection: close\r\n"));
                client.Send(Encoding.UTF8.GetBytes("Content-Type: application/json\r\n"));
                client.Send(Encoding.UTF8.GetBytes("Content-Length: " + body.Length + "\r\n"));
                client.Send(Encoding.UTF8.GetBytes("\r\n" + body));

                Debug.Print("Sent response to client '" + client.RemoteEndPoint + "'.");
            }
        }

        private static string ParseAsciiString(byte[] buffer, int bufferLength)
        {
            var chars = new char[bufferLength];
            for (int index = 0; index < chars.Length; index++)
            {
                chars[index] = (char)buffer[index];
            }

            return new string(chars);
        }

        private static void Listen()
        {
            while (true)
            {
                try
                {
                    Program.WaitForClient();
                }
                catch (Exception exception)
                {
                    Debug.Print("Unhandled exception!" + exception.Message);
                }
            }
        }

        private static void WaitForClient()
        {
            Debug.Print("Waiting for a client...");
            Socket client = Program._serverSocket.Accept();

            if (client == null)
            {
                // For shure...
                return;
            }

            Program._clientConnections++;

            Debug.Print("Client '" + client.RemoteEndPoint + "' connected.");

            try
            {
                client.ReceiveTimeout = 5000;
                client.SendTimeout = 5000;
                client.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.NoDelay, true);

                Program.HandleClient(client);
            }
            catch (Exception exception)
            {
                Debug.Print("Client connection '" + client.RemoteEndPoint + "' failed. " + exception.Message);
            }
            finally
            {
                client.Close();
            }
        }
    }
}


[title]The output[/title]

@ ChrisK - Could you be running out of memory resources somewhere? Since you are hitting it “as fast as possible” the garbage collection might not be able to keep up. I don’t see how you could run out of memory on the G120 but there might be some other limit that is being reached.

If you slowed the requests down can you get over 308 connections? Does forcing a garbage collection ( Debug.GC(true) ) make any difference?

Just some thoughts, I am by no means an expert in these areas.

@ ChrisK - I ran your code and I think I found a small problem. You only send a response to the client if client.Available == 0, otherwise you send no response back to the browser.

When I took the client.Available == 0 check out and sent a response back regardless of available bytes I was able to get over 1000 connections.
There were pauses in the process, probably because of the debugger. I also would have to let the browser catch up to actually see the JSON reply update.

@ ChrisK - Sorry one more thing I changed before I changed the socket.Available == 0. I think the bigger problem is the multiple sends. I condensed the 6 client.send commands to one. When I used only 1 send I got no errors and over 1500 connections.

                
string body = "{\"clients\":" + Program._clientConnections + "}";
string send = "HTTP/1.1 200 OK\r\n"
                     + "Access-Control-Allow-Origin: *\r\n"
                    + "Connection: close\r\n"
                    + "Content-Type: application/json\r\n"
                    + "Content-Length: " + body.Length + "\r\n"
                    + "\r\n" + body;

client.Send(Encoding.UTF8.GetBytes(send), send.Length, SocketFlags.None);
Debug.Print("Sent response to client '" + client.RemoteEndPoint + "'.");

@ skeller: You are right. This is a bug. I have fixed it and added a Debug.GC(true) directly after the client socket.Close().

Sending in chunks is very important for me. The original solution is also compiled with .NET Micro Framework 4.1 for some FEZ Panda 2 using a FEZ Connect shield. The socket objects are wrapped using interfaces and this works fine on every Panda 2. All the time! Because of the memory limitations of the Panda 2 I need to send the data in chunks to avoid a large string and a large buffer byte array.

The original solution has 4 iOS web apps which polls a JSON document every 2 seconds. This works 1-2 days and works with nearly 3000 connections. Then the server socket stops accepting new connections.

Test result (holding F5 key down):

  • The server works until connection attempt 1048 without any problems.
  • Then the servers stops working for at least 2 minutes! The “I am alive…” debug messages are also missing!
  • I hit the “pause” button in Visual Studio 2013.
  • The call stack of the main thread stays at “Thread.Sleep(5000)”.
  • After pressing the “Continue” button everything works fine!?
  • I stopped testing after connection attempt 4006.

This looks good and I will now add the Debug.GC(true) to my other solution. Then I wait whether the server crahses again (or not).

But there are questions left:

  • Why is it working at my Panda 2 WITHOUT a garbage collection.
  • Why is the remote address of the socket broken after a SocketException while sending data.

[title]The updated code[/title]


using System;
using Microsoft.SPOT;

namespace SocketTest
{
    using System.Net;
    using System.Net.Sockets;
    using System.Text;
    using System.Threading;

    using GHI.Networking;

    using Microsoft.SPOT.Hardware;

    public class Program
    {
        private static EthernetENC28J60 _networkInterface;

        private static Thread _workerThread;

        private static Socket _serverSocket;

        private static readonly byte[] _receiveBuffer = new byte[1024];

        private static int _clientConnections;

        public static void Main()
        {
            Program.InitializeNetworkHardware();

            Program._serverSocket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
            Program._serverSocket.Bind(new IPEndPoint(IPAddress.Any, 80));
            Program._serverSocket.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.NoDelay, true);
            Program._serverSocket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true);
            Program._serverSocket.Listen(10);

            Program._workerThread = new Thread(Program.Listen);
            Program._workerThread.Start();
            
            while (true)
            {
                Debug.Print("I am alive (at least " + Program._clientConnections + " client connections)!");
                Thread.Sleep(5000);
            }
        }

        private static void InitializeNetworkHardware()
        {
            Debug.Print("Enabling ENC28 module.");
            Program._networkInterface = new EthernetENC28J60(SPI.SPI_module.SPI2, GHI.Pins.G120.P1_17, GHI.Pins.G120.P0_5, GHI.Pins.G120.P1_14);

            Debug.Print("Opening ENC28 module.");
            Program._networkInterface.Open();

            Debug.Print("Configuring ENC28 module.");
            Program._networkInterface.EnableStaticIP("192.168.1.15", "255.255.255.0", "192.168.1.1");

            // Wait 10 seconds to ensure that the network hardware is fully initialized. The first UDP broadcasts will
            // fail if we not wait!
            Debug.Print("Waiting for ENC28 module.");
            Thread.Sleep(10000);
            Debug.Print("ENC28 module initialized.");
        }

        private static void HandleClient(Socket client)
        {
            if (client.Available == 0)
            {
                if (!client.Poll(5000 * 1000, SelectMode.SelectRead))
                {
                    return;
                }

                if (client.Available == 0)
                {
                    return;
                }
            }

            if (client.Available > Program._receiveBuffer.Length)
            {
                Debug.Print("Received data exceeds the buffer size.");

                return;
            }

            int requestLength = client.Receive(Program._receiveBuffer);

            Debug.Print("Received a request.");

            string message = Program.ParseAsciiString(Program._receiveBuffer, requestLength);

            // TODO: Do something with the message...

            if (!client.Poll(5000 * 1000, SelectMode.SelectWrite))
            {
                Debug.Print("Could not send data to the client.");
                return;
            }

            string body = "{\"clients\":" + Program._clientConnections + "}";

            client.Send(Encoding.UTF8.GetBytes("HTTP/1.1 200 OK\r\n"));
            client.Send(Encoding.UTF8.GetBytes("Access-Control-Allow-Origin: *\r\n"));
            client.Send(Encoding.UTF8.GetBytes("Connection: close\r\n"));
            client.Send(Encoding.UTF8.GetBytes("Content-Type: application/json\r\n"));
            client.Send(Encoding.UTF8.GetBytes("Content-Length: " + body.Length + "\r\n"));
            client.Send(Encoding.UTF8.GetBytes("\r\n" + body));

            Debug.Print("Sent response to client '" + client.RemoteEndPoint + "'.");
        }

        private static string ParseAsciiString(byte[] buffer, int bufferLength)
        {
            var chars = new char[bufferLength];
            for (int index = 0; index < chars.Length; index++)
            {
                chars[index] = (char)buffer[index];
            }

            return new string(chars);
        }

        private static void Listen()
        {
            while (true)
            {
                try
                {
                    Program.WaitForClient();
                }
                catch (Exception exception)
                {
                    Debug.Print("Unhandled exception!" + exception.Message);
                }
            }
        }

        private static void WaitForClient()
        {
            Debug.Print("Waiting for a client...");
            Socket client = Program._serverSocket.Accept();

            if (client == null)
            {
                // For shure...
                return;
            }

            Program._clientConnections++;

            Debug.Print("Client '" + client.RemoteEndPoint + "' connected.");

            try
            {
                client.ReceiveTimeout = 5000;
                client.SendTimeout = 5000;
                client.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.NoDelay, true);

                Program.HandleClient(client);
            }
            catch (Exception exception)
            {
                Debug.Print("Client connection '" + client.RemoteEndPoint + "' failed. " + exception.Message);
            }
            finally
            {
                client.Close();
                Debug.GC(true);
            }
        }
    }
}


[quote=“andre.m”]

OK. I have expected <> or “” in this case.

@ ChrisK - My only guess on why it worked on the Panda and not on the G120 is that there is a difference in the way Socket.send() works between .netmf 4.1 and 4.3. Check links below for difference.

4.1: http://msdn.microsoft.com/en-us/library/ee436070.aspx

4.3: http://msdn.microsoft.com/en-us/library/w93yy28a(v=vs.102).aspx

Bad news! It doesn’t work.
I have tested the example code with 20 auto refreshing tabs (every second) and the server was accepting over 10,000 connections. But after updating my other solution it doesn’t work. The server socket stops listening after 246 connections.

Now I have added the UDP broadcast sender from the other solution to check whether this socket causes the problems.
I also increased the response size and added a small sleep before sending the response.

Then I have deployed the application and opened 5 tabs which are refreshing each second. The server socket accepts 347 sockets (without an attached debugger) at the first test run before hanging at the Accept-method.

At the second try the code runs without any problems and accepted 4,000 connections (without an attached debugger).

This is very strange.

Another interesing thing is that the server sockets hangs (at my other solution) at the Accept-method after 100 connections if the code is running in release mode.

It is possible to upload my full solution so other users can try it with that code instead of the short version?

[title]The new version[/title]


namespace SocketTest
{
    using System;
    using System.Net;
    using System.Net.Sockets;
    using System.Text;
    using System.Threading;

    using GHI.Networking;

    using Microsoft.SPOT;
    using Microsoft.SPOT.Hardware;

    public class Program
    {
        private static EthernetENC28J60 _networkInterface;

        private static Thread _workerThread;

        private static Socket _serverSocket;

        private static Socket _broadcastSocket;

        private static IPEndPoint _broadcastAddress = new IPEndPoint(IPAddress.Parse("255.255.255.255"), 40000);

        private static readonly byte[] _receiveBuffer = new byte[1024];

        private static int _clientConnections;

        public static void Main()
        {
            Program.InitializeNetworkHardware();

            Program._broadcastSocket = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
            Program._broadcastSocket.SetSocketOption(SocketOptionLevel.Udp, SocketOptionName.Broadcast, true);

            Program._serverSocket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
            Program._serverSocket.Bind(new IPEndPoint(IPAddress.Any, 80));
            Program._serverSocket.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.NoDelay, true);
            Program._serverSocket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true);
            Program._serverSocket.Listen(10);

            Program._workerThread = new Thread(Program.Listen);
            Program._workerThread.Start();
            
            while (true)
            {
                Program.PrintDebugMessage("I am alive (at least " + Program._clientConnections + " client connections)!");
                Thread.Sleep(5000);
            }
        }

        private static void PrintDebugMessage(string message)
        {
            Debug.Print(message);

            try
            {
                var buffer = Encoding.UTF8.GetBytes(message);
                Program._broadcastSocket.SendTo(buffer, Program._broadcastAddress);
            }
            catch (Exception exception)
            {
                Debug.Print("Could not broadcast message '" + message + "'. " + exception.Message);
            }
        }

        private static void InitializeNetworkHardware()
        {
            Debug.Print("Enabling ENC28 module.");
            Program._networkInterface = new EthernetENC28J60(SPI.SPI_module.SPI2, GHI.Pins.G120.P1_17, GHI.Pins.G120.P0_5, GHI.Pins.G120.P1_14);

            Debug.Print("Opening ENC28 module.");
            Program._networkInterface.Open();

            Debug.Print("Configuring ENC28 module.");
            Program._networkInterface.EnableStaticIP("192.168.1.15", "255.255.255.0", "192.168.1.1");

            // Wait 10 seconds to ensure that the network hardware is fully initialized. The first UDP broadcasts will
            // fail if we not wait!
            Debug.Print("Waiting for ENC28 module.");
            Thread.Sleep(10000);
            Debug.Print("ENC28 module initialized.");
        }

        private static void HandleClient(Socket client)
        {
            if (client.Available == 0)
            {
                if (!client.Poll(5000 * 1000, SelectMode.SelectRead))
                {
                    return;
                }

                if (client.Available == 0)
                {
                    return;
                }
            }

            if (client.Available > Program._receiveBuffer.Length)
            {
                Program.PrintDebugMessage("Received data exceeds the buffer size.");

                return;
            }

            int requestLength = client.Receive(Program._receiveBuffer);

            Program.PrintDebugMessage("Received a request.");

            string message = Program.ParseAsciiString(Program._receiveBuffer, requestLength);

            // TODO: Do something with the message...
            Thread.Sleep(300);

            if (!client.Poll(5000 * 1000, SelectMode.SelectWrite))
            {
                Program.PrintDebugMessage("Could not send data to the client.");
                return;
            }

            string body = "{\"clients\":" + Program._clientConnections + ",\"data\":\"" + Program.GetData() + "\"}";

            client.Send(Encoding.UTF8.GetBytes("HTTP/1.1 200 OK\r\n"));
            client.Send(Encoding.UTF8.GetBytes("Access-Control-Allow-Origin: *\r\n"));
            client.Send(Encoding.UTF8.GetBytes("Connection: close\r\n"));
            client.Send(Encoding.UTF8.GetBytes("Content-Type: application/json\r\n"));
            client.Send(Encoding.UTF8.GetBytes("Content-Length: " + body.Length + "\r\n"));
            client.Send(Encoding.UTF8.GetBytes("\r\n" + body));

            Program.PrintDebugMessage("Sent response to client '" + client.RemoteEndPoint + "'.");
        }

        private static string GetData()
        {
            string data = string.Empty;
            for (int i = 0; i < 1024; i++)
            {
                data += "x";
            }

            return data;
        }

        private static string ParseAsciiString(byte[] buffer, int bufferLength)
        {
            var chars = new char[bufferLength];
            for (int index = 0; index < chars.Length; index++)
            {
                chars[index] = (char)buffer[index];
            }

            return new string(chars);
        }

        private static void Listen()
        {
            while (true)
            {
                try
                {
                    Program.WaitForClient();
                }
                catch (Exception exception)
                {
                    Program.PrintDebugMessage("Unhandled exception!" + exception.Message);
                }
            }
        }

        private static void WaitForClient()
        {
            Program.PrintDebugMessage("Waiting for a client...");
            Socket client = Program._serverSocket.Accept();

            if (client == null)
            {
                // For shure...
                return;
            }

            Program._clientConnections++;

            Program.PrintDebugMessage("Client '" + client.RemoteEndPoint + "' connected.");

            try
            {
                client.ReceiveTimeout = 5000;
                client.SendTimeout = 5000;
                client.SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.NoDelay, true);

                Program.HandleClient(client);
            }
            catch (Exception exception)
            {
                Program.PrintDebugMessage("Client connection '" + client.RemoteEndPoint + "' failed. " + exception.Message);
            }
            finally
            {
                client.Close();
                Debug.GC(true);
            }
        }
    }
}


Hello eveybody,

I’m fighting the same battle since a while. I build a webserver which seems to be working, but after a while there is no response (pressing the browser refresh button fast again and again). Reducing the code explains, that the server.Accept() stops working.

I tried it with G120 and at least with FEZ Raptor, hoping it were a G120 issue, but it is still an issue (both with ENC28).

My code is about similar (excluding th udp). I’m very very very interested about the solution. Thus I will follow this thread and try the posted code on raptor until a solution found.

It must be possible to run such a simple socket server.

(I’m using the Panda as well but only for UDP client and server and in case of penetration it sometimes stops working as well).

(I already tried the gadgeteer webserver implementation with the issue in case of some requests)