G400 Ethernet/Sockets causing watchdog reset (new issue)

AlexTZA · March 3, 2017, 7:15am

This is a new bug/behaviour found since I first raised a related issue in the following thread:

[url]https://www.ghielectronics.com/community/forum/topic?id=22770&page=1[/url]

The device running against the 2016 R1 release SDK and latest firmware.

The issue we are facing is we have a number of G400-based systems deployed in the field. However, we record the number of uncontrolled system restarts that are occurring on the devices and it is significant. This is of concern because the device uses an SD card and corruption of the file system on this card will require a field trip and a tedious device removal/replacement process. We have included design elements, such as the use of super caps, to avoid an uncontrolled closure of the SD file system.

After some testing, it appears the issue is related to the Ethernet and/or Sockets, but I suspect Sockets due to the previous issue mentioned in the thread above and because of the environment the devices work in where the network connectivity at the device side or path to the server is variable. The issue can be replicated by using the test code of the above linked thread (or re-posted below) under the following conditions:

The server must be running and have a working network connection
The Ethernet cable on the G400-device side must be removed during operation. The issue manifests fairly immediately/readily by pulling the network cable out of the device after it has connected to the server and then replacing it after about 30 seconds.

With this setup I have validated the test code will always cause the device to crash/reset when it attempts to open a new Socket connection to the reachable server. If the server is physically not connected to the network or the server application is not running, the device does not crash, so it seems like the fact that if the Ethernet is disconnected when a Socket connection is open, something is going wrong in the Socket/network stack upon connection of the new Socket.

I have validated the code (SocketEx) still works around the following scenarios where the Ethernet connection on the device side remains permanently connected and:

Server not connected to network
Server connected to network but service not running
Server intermittently connected to network and service running

I have also validated the following scenarios work where the Ethernet connection on the device side is disconnected periodically:

Server not connected to network
Server connected to network but service not running

The following is the MFDeploy debug trace:

Connecting to G400_Nexus…Connected
Starting comms…
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…

---------------------------->Network cable plugged in

Connecting to [192.168.0.46]…
SocketEx: Attempting to asynchronously open a socket connection…
SocketEx: Socket connection attempt complete
Connected to [192.168.0.46]!
Closing connection to [192.168.0.46]
SocketEx: The socket is not scheduled to close and is not busy connecting so calling base socket close.
Connecting to [192.168.0.46]…
SocketEx: Attempting to asynchronously open a socket connection…
SocketEx: Socket connection attempt complete
Connected to [192.168.0.46]!
Closing connection to [192.168.0.46]
SocketEx: The socket is not scheduled to close and is not busy connecting so calling base socket close.
Connecting to [192.168.0.46]…
SocketEx: Attempting to asynchronously open a socket connection…
SocketEx: Socket connection attempt complete
Connected to [192.168.0.46]!
Closing connection to [192.168.0.46]
SocketEx: The socket is not scheduled to close and is not busy connecting so calling base socket close.
Connecting to [192.168.0.46]…
SocketEx: Attempting to asynchronously open a socket connection…
SocketEx: Socket connection attempt complete
Connected to [192.168.0.46]!
Closing connection to [192.168.0.46]
SocketEx: The socket is not scheduled to close and is not busy connecting so calling base socket close.
Connecting to [192.168.0.46]…
SocketEx: Attempting to asynchronously open a socket connection…
SocketEx: Socket connection attempt complete
Connected to [192.168.0.46]!
Closing connection to [192.168.0.46]
SocketEx: The socket is not scheduled to close and is not busy connecting so calling base socket close.
Connecting to [192.168.0.46]…
SocketEx: Attempting to asynchronously open a socket connection…
SocketEx: Socket connection attempt complete
Connected to [192.168.0.46]!
Closing connection to [192.168.0.46]
SocketEx: The socket is not scheduled to close and is not busy connecting so calling base socket close.
Connecting to [192.168.0.46]…
SocketEx: Attempting to asynchronously open a socket connection…
SocketEx: Socket connection attempt complete
Connected to [192.168.0.46]!
Closing connection to [192.168.0.46]
SocketEx: The socket is not scheduled to close and is not busy connecting so calling base socket close.
Connecting to [192.168.0.46]…
SocketEx: Attempting to asynchronously open a socket connection…
SocketEx: Socket connection attempt complete
Connected to [192.168.0.46]!
Closing connection to [192.168.0.46]
SocketEx: The socket is not scheduled to close and is not busy connecting so calling base socket close.
Connecting to [192.168.0.46]…
SocketEx: Attempting to asynchronously open a socket connection…

---------------------------->Network cable disconnected

#### Exception System.Net.Sockets.SocketException - CLR_E_FAIL (13) ####
#### Message: 
#### Microsoft.SPOT.Net.SocketNative::poll [IP: 0000] ####
#### System.Net.Sockets.Socket::Poll [IP: 0011] ####
#### System.Net.Sockets.Socket::Connect [IP: 0029] ####
#### Testing.SocketEx::ConnectThread [IP: 0016] ####
#### SocketException ErrorCode = 10050

SocketEx: Socket connection attempt complete
#### Exception System.Net.Sockets.SocketException - CLR_E_FAIL (13) ####
#### Message:
#### Microsoft.SPOT.Net.SocketNative::poll [IP: 0000] ####
#### System.Net.Sockets.Socket::Poll [IP: 0011] ####
#### Testing.SocketEx::ConnectThread [IP: 0077] ####
#### SocketException ErrorCode = 10050
Failed to connect to [192.168.0.46]!
SocketEx: The socket is not scheduled to close and is not busy connecting so calling base socket close.
LWIP Assertion “PCB must be deallocated outside this function” failed at line 650 in D:\Repos\NETMF_Firmware\DeviceCode\pal\lwip\lwip\src\api\api_msg.c
LWIP Assertion “recvmbox must be deallocated before calling this function” failed at line 651 in D:\Repos\NETMF_Firmware\DeviceCode\pal\lwip\lwip\src\api\api_msg.c
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…
Network not available…

---------------------------->Network cable reconnected

Connecting to [192.168.0.46]…
SocketEx: Attempting to asynchronously open a socket connection…
SocketEx: Socket connection attempt complete
Connected to [192.168.0.46]!
LWIP Assertion “conn->state == NETCONN_CONNECT” failed at line 967 in D:\Repos\NETMF_Firmware\DeviceCode\pal\lwip\lwip\src\api\api_msg.c
LWIP Assertion “(conn->current_msg != NULL) || conn->in_non_blocking_connect” failed at line 968 in D:\Repos\NETMF_Firmware\DeviceCode\pal\lwip\lwip\src\api\api_msg.c
Closing connection to [192.168.0.46]
SocketEx: The socket is not scheduled to close and is not busy connecting so calling base socket close.
LWIP Assertion “PCB must be deallocated outside this function” failed at line 650 in D:\Repos\NETMF_Firmware\DeviceCode\pal\lwip\lwip\src\api\api_msg.c
LWIP Assertion “recvmbox must be deallocated before calling this function” failed at line 651 in D:\Repos\NETMF_Firmware\DeviceCode\pal\lwip\lwip\src\api\api_msg.c

----------------------------> Device hangs and then reboots

AlexTZA · March 3, 2017, 7:17am

Device code used (server is as per link above):

using System;
using System.Threading;
using System.Net.Sockets;
using System.Net;
using System.Reflection;
using System.Collections;

using Microsoft.SPOT;
using Microsoft.SPOT.Hardware;
using Microsoft.SPOT.Net.NetworkInformation;

using GHI.Networking;
using GHI.Pins;


namespace Testing
{
    public class TestsSimple
    {
        private static TimeSpan TWO_SECONDS = new TimeSpan(0, 0, 2);
        private static bool _commsRunning = false;
        private static IPEndPoint endpoint = new IPEndPoint(new IPAddress(new byte[] { 192, 168, 0, 46 }), 21000);
        private static BaseInterface _ethernet = null;
        private static SocketEx _socket;
         
        
        public static void Main()
        {  
            Thread commsThread = null;
            int stopCounter = 0;

            Random random = new Random();
            int stopThreshold = 0;

            while (true)
            {
                if (stopCounter == 0)
                {
                    Debug.Print("Creating Ethernet connection...");
                    _ethernet = Ethernet_Connect();
                 
                    Debug.Print("Starting comms...");
                    _commsRunning = true;
                    commsThread = new Thread(CommsThread);
                    commsThread.Start();

                    stopThreshold = random.Next(60);
                }

                Thread.Sleep(5000);

                stopCounter++;

                if (stopCounter >= stopThreshold)
                {
                    //test scenario where the Ethernet interface is stopped/started
                    Debug.Print("Stopping comms...");
                    _commsRunning = false;

                    if (!ThreadEx.JoinAbort(commsThread, 5000))
                    {
                        Debug.Print("WARNING: Comms thread was aborted!");
                    }

                    commsThread = null;

                    Debug.Print("Closing Ethernet...");
                    //close the ethernet interface
                    _ethernet.Close();
                    _ethernet.Dispose();                 

                    stopCounter = 0;
                }               
            }
        }  

        private static void CommsThread()
        {
            while (_commsRunning)
            {
                if (_ethernet != null && _ethernet.NetworkAvailable && _ethernet.Opened)
                {
                    _socket = new SocketEx(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
           
                    Debug.Print("Connecting to [" + endpoint.Address.ToString() + "]...");

                    //block/wait for a connection
                    //_connectionComplete.WaitOne();

                    if (_socket.Connect(endpoint))
                    {                      
                        Debug.Print("Connected to [" + endpoint.Address.ToString() + "]!");

                        DateTime start = DateTime.Now;

                        //socket already connected so close after a while
                        while (_commsRunning && DateTime.Now - start < TWO_SECONDS)
                        {
                            Thread.Sleep(100);
                        }

                        Debug.Print("Closing connection to [" + endpoint.Address.ToString() + "]");
                    }
                    else
                    {                      
                        Debug.Print("Failed to connect to [" + endpoint.Address.ToString() + "]!");
                        DateTime start = DateTime.Now;

                        while (_commsRunning && DateTime.Now - start < TWO_SECONDS)
                        {
                            Thread.Sleep(100);
                        }
                    }

                    _socket.Close();
                    _socket = null;
                }
                else
                {
                    Debug.Print("Network not available...");
                    Thread.Sleep(1000);

                    if (_socket != null)
                    {
                        _socket.Close();
                        _socket = null;
                    }
                }
            }     
    
            if (_socket != null)
            {                
                _socket.Close();
            }
        }        

        private static BaseInterface Ethernet_Connect()
        {
            BaseInterface ethernet = null;

            bool useDHCP = false;
            bool useSPI = false;

            ///Initialise Ethernet interface///////////////////////////////////////////////////////////////////////////

            bool initialised = false;

            if (useSPI)
            {
                ethernet = new EthernetENC28J60(SPI.SPI_module.SPI2, GHI.Pins.G400.PC22, GHI.Pins.G400.PC31, GHI.Pins.G400.PA5, 4000);//SPI.SPI_module.SPI2, GHI.Pins.G400.PB5, GHI.Pins.G400.PB0, GHI.Pins.G400.PA7);
                initialised = true;
            }
            else
            {
                //the following is a hack/workaround for the Builtin Ethernet to work in Debug mode
                //as per https://www.ghielectronics.com/community/forum/topic?id=22487

                long constructorTime = 0;
                int retryCount = 0;

                while (constructorTime == 0 && retryCount < 5)
                {
                    DateTime now = DateTime.Now;

                    ethernet = new GHI.Networking.EthernetBuiltIn();

                    constructorTime = ((DateTime.Now - now).Ticks / TimeSpan.TicksPerMillisecond);

                    //Debug.Print("Ethernet constructor time (ms) = " + constructorTime);

                    if (constructorTime == 0)
                    {
                        ethernet.Dispose();
                        ethernet = null;
                        retryCount++;
                    }
                }

                if (retryCount < 5)
                {
                    initialised = true;
                }
                else
                {
                    Debug.Print("ERROR: Working Ethernet interface object could not be created!");
                }
            }

            Debug.Assert(initialised, "Failed to initialise Ethernet");

            ethernet.Open();

            //open ethernet 

            if (useDHCP)
            {
                ethernet.EnableDhcp();
                ethernet.EnableDynamicDns();

                while (ethernet.IPAddress == "0.0.0.0")
                {
                    Thread.Sleep(500);
                }
            }
            else
            {
                ethernet.EnableStaticIP("192.168.0.141", "255.255.255.0", "192.168.0.1");
                ethernet.EnableStaticDns(new string[] { "192.168.0.200" });
            }            

            Debug.Assert(ethernet.IPAddress != "0.0.0.0");

            return ethernet;
        }
    }



    public class SocketEx : Socket
    {
        #region Events
        public delegate void ConnectComplete(SocketEx sender, ConnectionEventArgs args);
        public event ConnectComplete EndConnect = delegate { };
        #endregion

        #region Fields
        private bool _connected = false;
        private bool _isClosed = false;
        private object _closeLock = new object();
        private Thread _connectThread;
        private IPEndPoint _endPoint;
        private AutoResetEvent _connectionComplete;
        private int _lastSocketErrorCode;

        #endregion

        #region Properties

        /// <summary>
        /// Gets the last socket error code.
        /// </summary>
        /// <value>
        /// The last socket error code.
        /// </value>
        public int LastSocketErrorCode
        {
            get { return _lastSocketErrorCode; }
        }

        /// <summary>
        /// Gets the end point.
        /// </summary>
        /// <value>
        /// The end point.
        /// </value>
        public IPEndPoint EndPoint
        {
            get { return _endPoint; }
        }

        /// <summary>
        /// Gets a value indicating whether this <see cref="SocketEx"/> is connected.
        /// </summary>
        /// <value>
        ///   <c>true</c> if connected; otherwise, <c>false</c>.
        /// </value>
        public bool Connected
        {
            get
            {
                lock (_closeLock)
                {
                    return _connected;
                }
            }
        }

        /// <summary>
        /// Gets a value indicating whether a connection is pending.
        /// </summary>
        /// <value>
        ///   <c>true</c> if [connection pending]; otherwise, <c>false</c>.
        /// </value>
        private bool ConnectionPending
        {
            get { return (!_connected && _connectThread != null && _connectThread.IsAlive); }
        }
        #endregion

        #region Constructors
        /// <summary>
        /// Initializes a new instance of the <see cref="SocketEx"/> class.
        /// </summary>
        /// <param name="addressFamily">The address family.</param>
        /// <param name="socketType">Type of the socket.</param>
        /// <param name="protocolType">Type of the protocol.</param>
        public SocketEx(AddressFamily addressFamily, SocketType socketType, ProtocolType protocolType)
            : base(addressFamily, socketType, protocolType)
        {
            //SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.Linger, -2);
            if (protocolType == ProtocolType.Tcp)
            {
                SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.DontLinger, true);
                SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.NoDelay, true);
            }
            //SetSocketOption(SocketOptionLevel.Tcp, SocketOptionName.KeepAlive, true);    

            _connectionComplete = new AutoResetEvent(false);
        }

        #endregion

        #region Methods
        /// <summary>
        /// Closes this instance. If this is called while a connect on the socket is in progress, the socket will not abort the connect operation 
        /// thread but let it fail in its own time to avoid crashing the entire .net micro framework.
        /// </summary>
        public new void Close()
        {
            //release any pending blocking/waiting connect operations
            _connectionComplete.Set();

            lock (_closeLock)
            {
                if (!_isClosed && (_connectThread == null || !_connectThread.IsAlive))
                {
 #if DEBUG
                    Debug.Print("SocketEx: The socket is not scheduled to close and is not busy connecting so calling base socket close.");
 #endif
                    //if the socket is not already closed and the connect thread was never started or the connect thread has completed, do a Close
                    base.Close();
                }
                else
                {
 #if DEBUG
                    Debug.Print("SocketEx: Scheduled close on a connecting socket...");
 #endif
                }

                _connected = false;
                _isClosed = true;
            }
        }

        /// <summary>
        /// Starts an asynchronous connect
        /// </summary>
        /// <param name="endPoint">The end point.</param>
        /// <returns></returns>
        public bool BeginConnect(IPEndPoint endPoint)
        {
            lock (_closeLock)
            {
                if (!ConnectionPending)
                {
                    //if there is no connection already pending, start the connection thread
                    _endPoint = endPoint;

                    _connectThread = null;
                    _connectThread = new Thread(ConnectThread);
                    _connectThread.Start();

                    return true;
                }
                else
                {
                    return false;
                }
            }
        }

        /// <summary>
        /// Does a blocking connect that supports other threads calling Close during the Connect
        /// </summary>
        /// <param name="endPoint">The end point.</param>
        /// <returns></returns>
        public bool Connect(IPEndPoint endPoint)
        {
            if (BeginConnect(endPoint))
            {
                _connectionComplete.WaitOne();
            }

            return _connected;
        }

        private void ConnectThread()
        {
 #if DEBUG
            Debug.Print("SocketEx: Attempting to asynchronously open a socket connection...");
 #endif
            try
            {
                _connected = false;

                //in .NETMF 4.3:
                //- the connect will fail immediately if the remote IP is available but the endpoint server is not running
                //- the connect will timeout after 45 sec or so if the endpoint IP is not available, and will succeed if the endpoint is suddenly made available                
                base.Connect(_endPoint);
            }
            catch (SocketException f)
            {
                //usually occurs if the local network cable is disconnected or the underlying ethernet connection is not initialised
                _lastSocketErrorCode = f.ErrorCode;
            }
 #if DEBUG
            Debug.Print("SocketEx: Socket connection attempt complete");
 #endif

            //do a lazy close if necessary as this thread may still be running after a Close was called on this object
            lock (_closeLock)
            {
                //was Close previously called on this socket?
                if (_isClosed)
                {
 #if DEBUG
                    Debug.Print("SocketEx: Doing lazy close of a socket that was trying to connect");
 #endif
                    //the socket was closed but the thread was still active when that happened so call Close now to clean this socket up properly
                    base.Close();
                }
                else
                {
                    //the following is a check-fix for GHI SDK 2016 R1 PR1
                    //where the connection succeeds when the remote endpoint is not actually available
                    //and the contained remote end point data is garbage.
                    if (RemoteEndPoint.ToString() == _endPoint.ToString())
                    {
                        try
                        {
                            if (Poll(20, SelectMode.SelectWrite))
                            {
                                _connected = true;
                            }
                        }
                        catch (SocketException)
                        {
                        }
                    }

                    //call aynch end event
                    EndConnect(this, new ConnectionEventArgs(_connected, _lastSocketErrorCode));
                }
            }

            _connectionComplete.Set();
        }

        public new int Send(byte[] buffer)
        {
            int sent = 0;
            try
            {
                sent = base.Send(buffer);
            }
            catch (SocketException)
            {
                _connected = false;
            }

            return sent;
        }

        public new int Send(byte[] buffer, SocketFlags socketFlags)
        {
            int sent = 0;
            try
            {
                sent = base.Send(buffer, socketFlags);
            }
            catch (SocketException)
            {
                _connected = false;
            }

            return sent;
        }

        public new int Send(byte[] buffer, int size, SocketFlags socketFlags)
        {
            int sent = 0;
            try
            {
                sent = base.Send(buffer, size, socketFlags);
            }
            catch (SocketException)
            {
                _connected = false;
            }

            return sent;
        }

        public new int Send(byte[] buffer, int offset, int size, SocketFlags socketFlags)
        {
            int sent = 0;
            try
            {
                sent = base.Send(buffer, offset, size, socketFlags);
            }
            catch (SocketException)
            {
                _connected = false;
            }

            return sent;
        }
        #endregion
    }

    public class ConnectionEventArgs : EventArgs
    {
        /// <summary>
        /// Gets the volume information.
        /// </summary>
        /// <value>
        /// The volume information.
        /// </value>
        /// 
        public bool Connected { get; internal set; }
        public int SocketErrorCode { get; internal set; }

        /// <summary>
        /// Initializes a new instance of the <see cref="VolumeEventArgs"/> class.
        /// </summary>
        /// <param name="volumeInfo">The volume information.</param>
        public ConnectionEventArgs(bool connected, int socketErrorCode)
            : base()
        {
            Connected = connected;
            SocketErrorCode = socketErrorCode;
        }
    }


    public static class ThreadEx
    {

        public static bool JoinAbort(Thread[] threads, int timeToWaitMilliseconds)
        {
            bool result = false;
            TimeSpan timeout = new TimeSpan(0, 0, 0, 0, timeToWaitMilliseconds);
            DateTime startTime = DateTime.Now;
            bool keepWaiting = true;

            if (threads != null)
            {
                while (keepWaiting)
                {
                    keepWaiting = false;

                    foreach (Thread thread in threads)
                    {
                        if (thread != null && thread.IsAlive && thread.ThreadState != ThreadState.Stopped && DateTime.Now - startTime < timeout)
                        {
                            keepWaiting = true;
                            break;
                        }
                    }

                    if (keepWaiting)
                    {
                        Thread.Sleep(100);
                    }
                }

                result = true;

                foreach (Thread thread in threads)
                {
                    if (thread != null && thread.IsAlive)
                    {
                        thread.Abort();
                        result = false;
                    }
                }
            }

            return result;
        }

        public static bool JoinAbort(Thread thread, int timeToWaitMilliseconds)
        {
            bool result = false;
            TimeSpan timeout = new TimeSpan(0, 0, 0, 0, timeToWaitMilliseconds);
            DateTime startTime = DateTime.Now;

            if (thread != null)
            {
                while ((thread.IsAlive) && (thread.ThreadState != ThreadState.Stopped) && (DateTime.Now - startTime < timeout))
                {
                    Thread.Sleep(100);
                }

                if (thread.IsAlive)
                {
                    thread.Abort();
                }
                else
                {
                    result = true;
                }

                thread = null;
            }

            return result;
        }
    }
}

Mr_John_Smith · March 3, 2017, 7:58am

So in short are you trying to open a socket when the ethernet is not connected?

Plug in -> Open Socket -> Unplug -> Open Socket -> Exception?

AlexTZA · March 3, 2017, 8:35am

Not quite - to summarize the sequence of events that cause the issue are more like:

Plug in → Open Socket ->Connected->Unplug->Close Socket->Wait for network to become available->Plug in->Open socket->Crash

Brett · March 3, 2017, 3:42pm

your subject line talks about Watchdog but your description of the problem has none of that. What’s the relevance of that?

AlexTZA · March 4, 2017, 11:51pm

Apologies if the Watchdog link is not clear - the device reboots after the crash I presume due to the Watchdog not being reset internally by the GHI libraries, as mentioned before in the linked topic. My understanding is if I don’t explicitly enable the G400 Watchdog, the GHI libraries handle the watchdog reset internally anyways.

What I’m getting at is the resulting behaviour is the same (i.e. device crash with watchdog reset).

AlexTZA · March 6, 2017, 5:53am

After some additional scrutiny, it appears the issue is related to the SocketException being thrown on Socket.Connect due the network becoming unavailable. A few experiments and testing later, it appears if one does not close the Socket immediately after such a SocketException has occurred, the LWIP Assertion errors seen after the SocketException in MFDeploy no longer occur and the device no longer crashes.

Hence the workaround: I updated the SocketEx class ConnectThread method in the code posted earlier as follows to handle this case. The key point is if a SocketException is thrown during Connect, the Thread will do a lazy close of the underlying socket and then the .net Micro elves remain happy.


        private void ConnectThread()
        {   
 #if DEBUG
            Debug.Print("SocketEx: Attempting to asynchronously open a socket connection...");
 #endif
            try
            {
                _connected = false;

                //in .NETMF 4.3:
                //- the connect will fail immediately if the remote IP is available but the endpoint server is not running
                //- the connect will timeout after 45 sec or so if the endpoint IP is not available, and will succeed if the endpoint is suddenly made available                
                base.Connect(_endPoint);

 #if DEBUG
                Debug.Print("SocketEx: Socket.Connect complete");
 #endif
            }
            catch (SocketException f)
            {
 #if DEBUG
                Debug.Print("SocketEx: Socket.Connect threw exception...");
 #endif
                //usually occurs if the local network cable is disconnected or the underlying ethernet connection is not initialised
                _lastSocketErrorCode = f.ErrorCode;
                             
                //call aynch end event
                EndConnect(this, new ConnectionEventArgs(_connected, _lastSocketErrorCode));

                //let any threads waiting for connect to pass - this attempt failed
                _connectionComplete.Set();

                //wait longer than the connect timeout period before allowing the socket to be closed (will be a lazy close if the user calls Close in the meantime)
                Thread.Sleep(60000);        
            }
           
             //do a lazy close if necessary as this thread may still be running after a Close was called on this object
            lock (_closeLock)
            {                
                //was Close previously called on this socket?
                if (_isClosed)
                {
 #if DEBUG
                    Debug.Print("SocketEx: Doing lazy close of a socket that was trying to connect");
 #endif
                    //the socket was closed but the thread was still active when that happened so call Close now to clean this socket up properly
                    base.Close();
                }
                else
                {
                    //the following is a check-fix for GHI SDK 2016 R1 PR1
                    //where the connection succeeds when the remote endpoint is not actually available
                    //and the contained remote end point data is garbage.
                    if (RemoteEndPoint.ToString() == _endPoint.ToString())
                    {
                        try
                        {
                            if (Poll(20, SelectMode.SelectWrite))
                            {
                                _connected = true;                                   
                            }
                        }
                        catch (SocketException f)
                        {
                            Debug.Print("SocketEx: Socket connection attempt failed at poll validation with exception...");
                            //usually occurs if the local network cable is disconnected or the underlying ethernet connection is not initialised
                            _lastSocketErrorCode = f.ErrorCode;
                        }
                    }

                    //call aynch end event
                    EndConnect(this, new ConnectionEventArgs(_connected, _lastSocketErrorCode));

                    _connectionComplete.Set();
                }
            }  
        }