SSL bug / memory leak issue?

robert42 · March 30, 2016, 6:10am

Hi guys. I wonder if anyone can point me to a workaround for this apparent SSL bug in the new 2016 SDK?

After running the following test class for a long time(1hour +) , the SSL connection is no longer possible, and it continually throws an exception every time a subsequent request is made.

        public static void Main()
        {


           GHI.Networking.EthernetENC28J60 ethernet =  new GHI.Networking.EthernetENC28J60(SPI.SPI_module.SPI1,G120.P1_21,G120.P2_21, G120.P1_21, 1000);
           ethernet.EnableDhcp();
           ethernet.EnableDynamicDns();
           ethernet.Open();

           while (ethernet.IPAddress == "0.0.0.0") 
           {
               Thread.Sleep(1000);
           }
           
           Utility.SetLocalTime(new DateTime(2016, 03, 30, 10, 58, 01)); // set the time so that SSL works correctly

            for (; ; )
            {
                try
                {
                    string url = "https://httpbin.org/"; // this can be any https site
                    using (var req = System.Net.HttpWebRequest.Create(url))
                    {
                        req.Method = "Get";

                        using (var res = req.GetResponse())
                        {
                            Debug.Print("HTTP Response length: "
                              + res.ContentLength.ToString());
                            using (StreamReader reader = new StreamReader(res.GetResponseStream()))
                            {
                                while (reader.Read() > 0)
                                { }
                            }
                        }

                  }
                }
                catch (Exception e)
                {
                    Debug.Print("Fail");
                }

            }

The behavior is much worse when trying to use the Cellular module than with Ethernet, but it occurs in both cases.
below is the exception: ( one thing I noticed is that each time the garbage collector runs, the memory goes up, so I wonder if its a memory leak?). Initially, it executes fine for a while, but then it gets worse and worse until eventually all I get is exceptions.

HTTP Response length: 166
GC: 8msec 7119216 bytes used, 220452 bytes available
Type 0F (STRING ): 2136 bytes
Type 11 (CLASS ): 16140 bytes
Type 12 (VALUETYPE ): 552 bytes
Type 13 (SZARRAY ): 7884 bytes
Type 01 (BOOLEAN ): 24 bytes
Type 03 (U1 ): 4692 bytes
Type 04 (CHAR ): 624 bytes
Type 07 (I4 ): 60 bytes
Type 0F (STRING ): 24 bytes
Type 11 (CLASS ): 2376 bytes
Type 12 (VALUETYPE ): 84 bytes
Type 15 (FREEBLOCK ): 220452 bytes
Type 16 (CACHEDBLOCK ): 720 bytes
Type 17 (ASSEMBLY ): 33084 bytes
Type 18 (WEAKCLASS ): 48 bytes
Type 19 (REFLECTION ): 168 bytes
Type 1B (DELEGATE_HEAD ): 360 bytes
Type 1D (OBJECT_TO_EVENT ): 96 bytes
Type 1E (BINARY_BLOB_HEAD ): 7050108 bytes
Type 1F (THREAD ): 1536 bytes
Type 20 (SUBTHREAD ): 192 bytes
Type 21 (STACK_FRAME ): 1860 bytes
Type 22 (TIMER_HEAD ): 72 bytes
Type 27 (FINALIZER_HEAD ): 864 bytes
Type 31 (IO_PORT ): 108 bytes
Type 34 (APPDOMAIN_HEAD ): 72 bytes
Type 36 (APPDOMAIN_ASSEMBLY ): 3216 bytes
HTTP Response length: 166
GC: 7msec 7130508 bytes used, 209160 bytes available
Type 0F (STRING ): 1884 bytes
Type 11 (CLASS ): 12108 bytes
Type 12 (VALUETYPE ): 552 bytes
Type 13 (SZARRAY ): 5196 bytes
Type 01 (BOOLEAN ): 24 bytes
Type 03 (U1 ): 2004 bytes
Type 04 (CHAR ): 624 bytes
Type 07 (I4 ): 60 bytes
Type 0F (STRING ): 24 bytes
Type 11 (CLASS ): 2376 bytes
Type 12 (VALUETYPE ): 84 bytes
Type 15 (FREEBLOCK ): 209160 bytes
Type 16 (CACHEDBLOCK ): 96 bytes
Type 17 (ASSEMBLY ): 33084 bytes
Type 18 (WEAKCLASS ): 48 bytes
Type 19 (REFLECTION ): 168 bytes
Type 1B (DELEGATE_HEAD ): 360 bytes
Type 1D (OBJECT_TO_EVENT ): 96 bytes
Type 1E (BINARY_BLOB_HEAD ): 7069524 bytes
Type 1F (THREAD ): 1536 bytes
Type 20 (SUBTHREAD ): 192 bytes
Type 21 (STACK_FRAME ): 1860 bytes
Type 22 (TIMER_HEAD ): 72 bytes
Type 27 (FINALIZER_HEAD ): 336 bytes
Type 31 (IO_PORT ): 108 bytes
Type 34 (APPDOMAIN_HEAD ): 72 bytes
Type 36 (APPDOMAIN_ASSEMBLY ): 3216 bytes
HTTP Response length: 166
HTTP Response length: 166
GC: 7msec 7206024 bytes used, 133644 bytes available
Type 0F (STRING ): 1920 bytes
Type 11 (CLASS ): 12684 bytes
Type 12 (VALUETYPE ): 552 bytes
Type 13 (SZARRAY ): 5580 bytes
Type 01 (BOOLEAN ): 24 bytes
Type 03 (U1 ): 2388 bytes
Type 04 (CHAR ): 624 bytes
Type 07 (I4 ): 60 bytes
Type 0F (STRING ): 24 bytes
Type 11 (CLASS ): 2376 bytes
Type 12 (VALUETYPE ): 84 bytes
Type 15 (FREEBLOCK ): 133644 bytes
Type 16 (CACHEDBLOCK ): 192 bytes
Type 17 (ASSEMBLY ): 33084 bytes
Type 18 (WEAKCLASS ): 48 bytes
Type 19 (REFLECTION ): 168 bytes
Type 1B (DELEGATE_HEAD ): 360 bytes
Type 1D (OBJECT_TO_EVENT ): 96 bytes
Type 1E (BINARY_BLOB_HEAD ): 7143876 bytes
Type 1F (THREAD ): 1536 bytes
Type 20 (SUBTHREAD ): 192 bytes
Type 21 (STACK_FRAME ): 1860 bytes
Type 22 (TIMER_HEAD ): 72 bytes
Type 27 (FINALIZER_HEAD ): 408 bytes
Type 31 (IO_PORT ): 108 bytes
Type 34 (APPDOMAIN_HEAD ): 72 bytes
Type 36 (APPDOMAIN_ASSEMBLY ): 3216 bytes
Failed allocation for 1480 blocks, 17760 bytes

#### Exception System.Net.Sockets.SocketException - CLR_E_FAIL (1) ####
#### Message: 
#### Microsoft.SPOT.Net.Security.SslNative::SecureConnect [IP: 0000] ####
#### Microsoft.SPOT.Net.Security.SslStream::Authenticate [IP: 0060] ####
#### Microsoft.SPOT.Net.Security.SslStream::AuthenticateAsClient [IP: 000c] ####
#### System.Net.HttpWebRequest::EstablishConnection [IP: 0247] ####
#### System.Net.HttpWebRequest::SubmitRequest [IP: 0019] ####
#### System.Net.HttpWebRequest::GetResponse [IP: 000c] ####
#### Ethernet.Program::Main [IP: 007e] ####
#### SocketException ErrorCode = 1

GC: performing heap compaction…
#### SocketException ErrorCode = 1
A first chance exception of type ‘System.Net.Sockets.SocketException’ occurred in Microsoft.SPOT.Net.Security.dll
#### SocketException ErrorCode = 1
#### SocketException ErrorCode = 1
#### SocketException ErrorCode = 1
#### SocketException ErrorCode = 1
A first chance exception of type ‘System.Net.Sockets.SocketException’ occurred in System.Net.Security.dll
#### SocketException ErrorCode = 1
#### SocketException ErrorCode = 1
#### Exception System.Net.WebException - 0x00000000 (1) ####
#### Message:
#### System.Net.HttpWebRequest::GetResponse [IP: 00c8] ####
#### Ethernet.Program::Main [IP: 007e] ####
A first chance exception of type ‘System.Net.WebException’ occurred in System.Http.dll
Fail
GC: 6msec 7229568 bytes used, 110100 bytes available
Type 0F (STRING ): 1884 bytes
Type 11 (CLASS ): 12108 bytes
Type 12 (VALUETYPE ): 552 bytes
Type 13 (SZARRAY ): 5196 bytes
Type 01 (BOOLEAN ): 24 bytes
Type 03 (U1 ): 2004 bytes
Type 04 (CHAR ): 624 bytes
Type 07 (I4 ): 60 bytes
Type 0F (STRING ): 24 bytes
Type 11 (CLASS ): 2376 bytes
Type 12 (VALUETYPE ): 84 bytes
Type 15 (FREEBLOCK ): 110100 bytes
Type 17 (ASSEMBLY ): 33084 bytes
Type 18 (WEAKCLASS ): 48 bytes
Type 19 (REFLECTION ): 168 bytes
Type 1B (DELEGATE_HEAD ): 360 bytes
Type 1D (OBJECT_TO_EVENT ): 96 bytes
Type 1E (BINARY_BLOB_HEAD ): 7168656 bytes
Type 1F (THREAD ): 1536 bytes
Type 20 (SUBTHREAD ): 192 bytes
Type 21 (STACK_FRAME ): 1860 bytes
Type 22 (TIMER_HEAD ): 72 bytes
Type 27 (FINALIZER_HEAD ): 360 bytes
Type 31 (IO_PORT ): 108 bytes
Type 34 (APPDOMAIN_HEAD ): 72 bytes
Type 36 (APPDOMAIN_ASSEMBLY ): 3216 bytes
Failed allocation for 1467 blocks, 17604 bytes

#### Exception System.Net.Sockets.SocketException - CLR_E_FAIL (1) ####
#### Message: 
#### Microsoft.SPOT.Net.Security.SslNative::SecureConnect [IP: 0000] ####
#### Microsoft.SPOT.Net.Security.SslStream::Authenticate [IP: 0060] ####
#### Microsoft.SPOT.Net.Security.SslStream::AuthenticateAsClient [IP: 000c] ####
#### System.Net.HttpWebRequest::EstablishConnection [IP: 0247] ####
#### System.Net.HttpWebRequest::SubmitRequest [IP: 0019] ####
#### System.Net.HttpWebRequest::GetResponse [IP: 000c] ####
#### Ethernet.Program::Main [IP: 007e] ####
#### SocketException ErrorCode = 1

GC: performing heap compaction…
#### SocketException ErrorCode = 1
A first chance exception of type ‘System.Net.Sockets.SocketException’ occurred in Microsoft.SPOT.Net.Security.dll
#### SocketException ErrorCode = 1
#### SocketException ErrorCode = 1
#### SocketException ErrorCode = 1
#### SocketException ErrorCode = 1
A first chance exception of type ‘System.Net.Sockets.SocketException’ occurred in System.Net.Security.dll
#### SocketException ErrorCode = 1
#### SocketException ErrorCode = 1
#### Exception System.Net.WebException - 0x00000000 (1) ####
#### Message:
#### System.Net.HttpWebRequest::GetResponse [IP: 00c8] ####
#### Ethernet.Program::Main [IP: 007e] ####
A first chance exception of type ‘System.Net.WebException’ occurred in System.Http.dll
Fail

robert42 · March 30, 2016, 6:57am

If it helps, I connected MFdeploy, which gives more detailed debug:

LWIP Assertion “tcp_pcb_purge: pcb->state == SYN_RCVD but tcp_listen_pcbs is NULL” failed at line 336 in D:\Repos\NETMF_Firmware\DeviceCode\pal\lwip\lwip\src\core\tcp.c
GC: performing heap compaction…
GC: 6msec 7311708 bytes used, 27960 bytes available
Type 0F (STRING ): 1884 bytes
Type 11 (CLASS ): 12108 bytes
Type 12 (VALUETYPE ): 552 bytes
Type 13 (SZARRAY ): 5196 bytes
Type 01 (BOOLEAN ): 24 bytes
Type 03 (U1 ): 2004 bytes
Type 04 (CHAR ): 624 bytes
Type 07 (I4 ): 60 bytes
Type 0F (STRING ): 24 bytes
Type 11 (CLASS ): 2376 bytes
Type 12 (VALUETYPE ): 84 bytes
Type 15 (FREEBLOCK ): 27960 bytes
Type 17 (ASSEMBLY ): 33084 bytes
Type 18 (WEAKCLASS ): 48 bytes
Type 19 (REFLECTION ): 168 bytes
Type 1B (DELEGATE_HEAD ): 360 bytes
Type 1D (OBJECT_TO_EVENT ): 96 bytes
Type 1E (BINARY_BLOB_HEAD ): 7251228 bytes
Type 1F (THREAD ): 1152 bytes
Type 20 (SUBTHREAD ): 144 bytes
Type 21 (STACK_FRAME ): 1860 bytes
Type 22 (TIMER_HEAD ): 72 bytes
Type 27 (FINALIZER_HEAD ): 360 bytes
Type 31 (IO_PORT ): 108 bytes
Type 34 (APPDOMAIN_HEAD ): 72 bytes
Type 36 (APPDOMAIN_ASSEMBLY ): 3216 bytes
Failed allocation for 130 blocks, 1560 bytes

Err: 20, 166, 65, D:\Repos\NETMF_Firmware\DeviceCode\pal\OpenSSL\OpenSSL_1_0_0\ssl\ssl_ciph.cpp (1313)
Err: 20, 169, 161, D:\Repos\NETMF_Firmware\DeviceCode\pal\OpenSSL\OpenSSL_1_0_0\ssl\ssl_lib.cpp (1604)
#### Exception System.Net.Sockets.SocketException - CLR_E_FAIL (1) ####
#### Message:
#### Microsoft.SPOT.Net.Security.SslNative::SecureClientInit [IP: 0000] ####
#### Microsoft.SPOT.Net.Security.SslStream::Authenticate [IP: 0051] ####
#### Microsoft.SPOT.Net.Security.SslStream::AuthenticaUser Cancel…
teAsClient [IP: 000c] ####
#### System.Net.HttpWebRequest::EstablishConnection [IP: 0247] ####
#### System.Net.HttpWebRequest::SubmitRequest [IP: 0019] ####
#### System.Net.HttpWebRequest::GetResponse [IP: 000c] ####
#### Ethernet.Program::Main [IP: 007e] ####
#### SocketException ErrorCode = -1
#### SocketException ErrorCode = -1
#### Exception System.Net.WebException - 0x00000000 (1) ####
#### Message:
#### System.Net.HttpWebRequest::GetResponse [IP: 00c8] ###

robert42 · March 31, 2016, 1:51am

Thank you, Andre for the response.
I thought that they had worked out the SSL bugs in the new SDK, or at least found workarounds other than rebooting.

Hopefully in the next release, this will be resolved.

Gus_Issa · March 31, 2016, 6:39am

Your both statements are false and misleading.
GHI does fixes in the netmf core and there is no confirmation this is fixed in 4.4. This is a tough bug that GHI and Microsoft are yet to locate its cause.

njbuch · March 31, 2016, 7:28am

@ Gus - Hmm, I guess you are busy at //build because you might want to revise your above…

RobvanSchelven · March 31, 2016, 11:58am

How does this statement relates to this answer?
https://www.ghielectronics.com/community/forum/topic?id=22320 reply#3 ?

Gus_Issa · March 31, 2016, 12:37pm

It is typically not recommended to change the core code so all netmf devices behave similarly. And we instruct our team not to change the core but there many occasions where we did when it was absolutely necessary. The better way of course is in pushing the changes to the core but that is also not always an option.

As for the ssl bug, GHI and Microsoft had spent time working on but we yet to have a hard answer. We are still together working on tracking it down.

mtylerjr · March 31, 2016, 11:41pm

Nothing entertains me like engineers using their autistic social skills to pick apart the language of other engineers!
:

Dave_McLaughlin · April 1, 2016, 2:03am

But it’s just not the same without godefroi being part of it :whistle:

runs and hides

mtylerjr · April 3, 2016, 6:49pm

I think I was overly caffeinated on Friday when I posted that. I apologize to andre.m if I offended him.

Gus_Issa · April 8, 2016, 12:29pm

We might have the fix https://www.ghielectronics.com/community/forum/topic?id=22997