Network stops working after several HTTP clients times out

TinyCLR OS
1

Hi,

I am having issues with running out of memory or possibly running out of sockets. I have a HTTPListener running on a Fez Portal via wifi. If the network connection is unstable the connecting client will sometimes timeout. This causes some memory on the Fez to not be released. Eventually the HTTPListener throws an exception and it is not possible to connect anymore.

Does anyone of you clever forummembers have some ideas on what I can do to prevent this?

Server code (Thread.Sleep(500) causes the client to time out):

        //Create a listener.
        HttpListener listener = new HttpListener("http", 80);

        listener.Start();
        Debug.WriteLine("Listening...");

        var clientRequestCount = 0;

        while (true)
        {
            try
            {
                //Note: The GetContext method blocks while waiting for a request.
                HttpListenerContext context = listener.GetContext();

                //Obtain a response object.
                HttpListenerResponse response = context.Response;

                Thread.Sleep(500);

                //Construct a response.
                System.GC.Collect();
                var freeRam = GHIElectronics.TinyCLR.Native.Memory.ManagedMemory.FreeBytes;

                var responseString = string.Format("<HTML><BODY> I am TinyCLR OS Server. " + "Client request count: {0}, FreeRAM: {1}</BODY></HTML>", ++clientRequestCount, freeRam);
                Debug.WriteLine(responseString);
                byte[] buffer = System.Text.Encoding.UTF8.GetBytes(responseString);

                //Get a response stream and write the response to it.
                response.ContentLength64 = buffer.Length;
                var output = response.OutputStream;
                output.Write(buffer, 0, buffer.Length);

                // When you close the response all the streams are closed and released.
                response.Close();
            }
            catch (Exception)
            {

            }
        }

        listener.Stop();

Client code:
// Create the URL
Uri address = new Uri(“http://192.168.1.105”);

            // Create request
            HttpWebRequest request = HttpWebRequest.Create(address) as HttpWebRequest;

            request.Method = "GET";
            request.Timeout = 100;
            request.KeepAlive = false;

            // Get response  
            string responseString;
            using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
            {
                using (var stream = new StreamReader(response.GetResponseStream()))
                {
                    responseString = stream.ReadToEnd();
                }
            }

Debug output, the free RAM decreases until an exception occurs and all clients are refused:

I am TinyCLR OS Server. Client request count: 261, FreeRAM: 323792 I am TinyCLR OS Server. Client request count: 262, FreeRAM: 323008 I am TinyCLR OS Server. Client request count: 263, FreeRAM: 323008 I am TinyCLR OS Server. Client request count: 264, FreeRAM: 323008 I am TinyCLR OS Server. Client request count: 265, FreeRAM: 322480 I am TinyCLR OS Server. Client request count: 266, FreeRAM: 322480 I am TinyCLR OS Server. Client request count: 267, FreeRAM: 321952 I am TinyCLR OS Server. Client request count: 268, FreeRAM: 321952 I am TinyCLR OS Server. Client request count: 269, FreeRAM: 321424 I am TinyCLR OS Server. Client request count: 270, FreeRAM: 321424 I am TinyCLR OS Server. Client request count: 271, FreeRAM: 321424 I am TinyCLR OS Server. Client request count: 272, FreeRAM: 320896 I am TinyCLR OS Server. Client request count: 273, FreeRAM: 320896 I am TinyCLR OS Server. Client request count: 274, FreeRAM: 320368 I am TinyCLR OS Server. Client request count: 275, FreeRAM: 320368 I am TinyCLR OS Server. Client request count: 276, FreeRAM: 319840 I am TinyCLR OS Server. Client request count: 277, FreeRAM: 319840 I am TinyCLR OS Server. Client request count: 278, FreeRAM: 319840 I am TinyCLR OS Server. Client request count: 279, FreeRAM: 319312 I am TinyCLR OS Server. Client request count: 280, FreeRAM: 319312 I am TinyCLR OS Server. Client request count: 281, FreeRAM: 318784 I am TinyCLR OS Server. Client request count: 282, FreeRAM: 318784 I am TinyCLR OS Server. Client request count: 283, FreeRAM: 318784 I am TinyCLR OS Server. Client request count: 284, FreeRAM: 318256 #### Exception System.InvalidOperationException - CLR_E_INVALID_OPERATION (5) #### #### Message: #### GHIElectronics.TinyCLR.Devices.Network.Provider.NetworkControllerApiWrapper::Accept [IP: 0000] #### #### System.Net.Sockets.Socket::Accept [IP: 0023] #### Exception thrown: 'System.InvalidOperationException' in GHIElectronics.TinyCLR.Devices.Network.dll The thread '' (0x5) has exited with code 0 (0x0).
2

So I think I have some clues on what is happening. The client times out after 100ms and tries to connect again. Since the server is waiting 500ms the client requests gets queued. When the HTTPListener throws the exception there are 29 queued requests:

image

Now the problem is how do I get the HTTPListener running again? I tried to Abort and Start, and event to create a brand new listener, but they do not want to restart. Do I need to reset something else?

3

Tried with Stop() and Close() yet?

4

Now I think I finally have found the root cause of this memory leak. :exploding_head: The HttpListenerContext does not release its resources if the Response has not been accessed.

Server code:

            //Create a listener.
            HttpListener listener = new HttpListener("http", 80);

            listener.Start();
            Debug.WriteLine("Listening...");

            while (true)
            {
                try
                {
                    //Note: The GetContext method blocks while waiting for a request.
                    var context = listener.GetContext();

                    var inputStream = context.Request.InputStream;
                    var bytes = new byte[inputStream.Length];
                    inputStream.Read(bytes, 0, (int)inputStream.Length);
                    var requestString = Encoding.UTF8.GetString(bytes);

                    GC.Collect();
                    GC.WaitForPendingFinalizers();
                    var freeRam = GHIElectronics.TinyCLR.Native.Memory.ManagedMemory.FreeBytes;
                    Debug.WriteLine(string.Format("FreeRam: {0}", freeRam));
                }
                catch (Exception)
                {

                }
            }

Debug output with decreasing free RAM that eventually will lead to an OutOfMemory exception and the Fez will die:

FreeRam: 24432
FreeRam: 23360
FreeRam: 22288
FreeRam: 21216
FreeRam: 20144
FreeRam: 19072
FreeRam: 18000
FreeRam: 16928
FreeRam: 15856
FreeRam: 14784
FreeRam: 13712
FreeRam: 12640
FreeRam: 11568
FreeRam: 10496
FreeRam: 9424
FreeRam: 8352
FreeRam: 7280
FreeRam: 6208
FreeRam: 5136
FreeRam: 4064
Failed allocation for 35 blocks, 560 bytes

    #### Exception System.OutOfMemoryException - CLR_E_OUT_OF_MEMORY (1) ####
    #### Message: 
    #### System.Int64::ToString [IP: 000e] ####
    #### System.String::Format [IP: 03a9] ####
    #### System.String::Format [IP: 0006] ####
    #### MemoryLeak.Program2::Main [IP: 0078] ####
Exception thrown: 'System.OutOfMemoryException' in mscorlib.dll

Failed allocation for 52 blocks, 832 bytes

Failed allocation for 30 blocks, 480 bytes

Failed allocation for 10 blocks, 160 bytes

I believe the reason for this memory leak is that HttpListenerContext does not get disposed of correctly. It does not implement IDisposable (neither does the full .NET version), but it also does not have a deconstructor.

By adding a deconstructor to the HttpListenerContext that calls Close, the memory gets released. The problem now is that the Close method throws an exception when it tries to make the socket linger. The exception is caught, but it takes alot of time to handle it. This in turn chockes the system. A way to get around this is to call Close on the Response object. This will release all memory without throwing an exception. I will file an Issue for this on your GitHub.

HttpListenerContext deconstructor:

        ~HttpListenerContext() {
            Response.Close();
        }

Debug output with the deconstructor implemented, stabalizing at 314080 free RAM:

Listening...
The thread '<No Name>' (0x3) has exited with code 0 (0x0).
FreeRam: 333776
FreeRam: 313648
FreeRam: 312032
FreeRam: 312096
FreeRam: 312384
FreeRam: 312384
FreeRam: 312384
FreeRam: 312384
FreeRam: 314080
FreeRam: 314080
FreeRam: 314080
FreeRam: 314080
FreeRam: 314080
FreeRam: 314080

Now to the reason why I ended up in this situation:

            //Create a listener.
            HttpListener listener = new HttpListener("http", 80);

            listener.Start();
            Debug.WriteLine("Listening...");

            var clientRequestCount = 0;

            while (true)
            {
                try
                {
                    //Note: The GetContext method blocks while waiting for a request.
                    var context = listener.GetContext();

                    // Get the request content
                    var inputStream = context.Request.InputStream;
                    var bytes = new byte[inputStream.Length];
                    inputStream.Read(bytes, 0, (int)inputStream.Length);
                    var requestString = Encoding.UTF8.GetString(bytes);

                    var receivedTemperature = (TemperatureReading)JsonConverter.DeserializeObject(requestString, typeof(TemperatureReading));

                    //Obtain a response object.
                    var response = context.Response;
                    response.ContentLength64 = 0;

                    //Construct a response.
                    System.GC.Collect();
                    var freeRam = GHIElectronics.TinyCLR.Native.Memory.ManagedMemory.FreeBytes;
                    var usedRam = GHIElectronics.TinyCLR.Native.Memory.ManagedMemory.UsedBytes;
                    var responseString = string.Format("<HTML><BODY> {0} - Start time: {1}, client request count: {2}, RSSI: {3}, FreeRAM: {4}, UsedRAM: {5}.", DateTime.Now, mStartTime, ++clientRequestCount, mNetworkManager.GetRssi(), freeRam, usedRam);
                    responseString += "</ BODY ></ HTML > ";
                    Debug.WriteLine(responseString);
                    var buffer = System.Text.Encoding.UTF8.GetBytes(responseString);

                    //Get a response stream and write the response to it.
                    response.ContentLength64 = buffer.Length;
                    using (var outputStream = response.OutputStream)
                    {
                        outputStream.Write(buffer, 0, buffer.Length);
                    }
                }
                catch (Exception)
                {

                }
            }

            listener.Stop();

I used the InputStream.Length when reading the content. When testing this with a PC client, everything worked perfect, but when I used another Fez Portal as a client, and with unstable network, the inputstream would not contain the whole message when I tried to read it. This made the JsonConverter throw an exception that was caught at the end of the loop. Since we did not access the Response, and the HttpListenerContext did not get disposed, the application leaked memory.

I should have used the ContentLength64 property to get the whole content:

             // Use the StreamReader to make sure we read the whole message
             using (var inputStream = new StreamReader(context.Request.InputStream))
             {
                var bytes = new char[context.Request.ContentLength64];
                inputStream.Read(bytes, 0, (int)context.Request.ContentLength64);
                var requestString = new string(bytes);
3 Likes
5

I just want to thank you for giving a detailed explanation of your solution - it’s the worst when someone has a difficult question then just posts “FIXED” without explaining what they found.

2 Likes
6

As I know this is not happened on GHI forum, usually we will ask “how did you fix it?”…

But yes, I usually get upset when on other forums and see the post has only “fixed” or “resolved” without explanation :)).

1 Like
7

Thank you for your kind words.