Hydra FW crashes with

Hello,
The topic with similar issue was already submitted (http://www.ghielectronics.com/community/forum/topic?id=8557) however the solution not supplied.
We have multiple Hydra devices (~20) using for HW and FW testing purposes. The devices run with Ethernet FW.
The network configured on device initializing stage.
Our users experienced with problem that after number of resets / power re-connects Hydra device stopping to response with “USB device not recognized” message and yellow bang in Device Manager.
The recovery can be done using FW update only.
I tried to investigate the problem with very simple code (below) running on Hydra and client that checks that Hydra is alive over network. If Hydra is alive the clients restarts it using MFDeploy tool.
I found that after ~ 10-15 restarts the problem reproduced and Hydra FW indeed crashes.
The code compiled and deployed in Release mode with USB serial device used for debugging messages connect to Hydra socket 6.
Hydra code:


using System;
using System.Net;
using System.Net.Sockets;
using System.Text;
using System.Threading;
using Gadgeteer.Interfaces;
using Microsoft.SPOT;
using Microsoft.SPOT.Net.NetworkInformation;

namespace HydraTest
{
    /// <summary>
    ///     Main entry point for the execution program
    /// </summary>
    public partial class Program
    {
        #region Private members

        private bool _initialized;
        private NetworkInterface _networkInterface;
        private System.Net.Sockets.Socket _server;


        #endregion

        #region Control events and init

        // This method is run when the mainboard is powered up or reset.   
        private void ProgramStarted()
        {
            lED7R.TurnLightOn(1);
            lED7R.TurnLightOn(2);
            if (usbSerial1 != null)
            {
                usbSerial1.Configure(38400, Serial.SerialParity.None, Serial.SerialStopBits.One, 8);
                usbSerial1.SerialLine.Open();
            }
            Start();

        }

        private void Start()
        {
            if (_initialized)
                return;
            DebugPrint("Entering to initialize network");
            Thread.Sleep(500);
            InitializeNetwork();
            _initialized = true;
            Thread.Sleep(1000);
            DebugPrint("Network initialized");
            //usbSerial1.SerialLine.Write("Network initialized" + "EndMessage");
            Thread.Sleep(1000);
            lED7R.TurnLightOn(7);
        }


        #endregion

        #region Network stack

        private void InitializeNetwork()
        {
            DebugPrint("Initializing network...");
            try
            {
                DebugPrint("Getting interfaces");
                NetworkInterface[] interfaces = NetworkInterface.GetAllNetworkInterfaces();
                DebugPrint("Interfaces received");
                if (interfaces != null && interfaces.Length > 0)
                {
                    DebugPrint("Got " + interfaces.Length + " network interfaces...");
    
                    _networkInterface = interfaces[0];
    
                    DebugPrint("Try initializing network with Static configurations...");
                    _networkInterface.EnableStaticIP("10.12.227.200", "255.255.255.0", "10.12.227.3");

                    DebugPrint("Network initialized. IP Address: " + _networkInterface.IPAddress);
                    _server = new System.Net.Sockets.Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
                    _server.Bind(new IPEndPoint(IPAddress.Parse(_networkInterface.IPAddress), 81));
                    DebugPrint("Bound...");
                    _server.Listen(3);
                    Thread.Sleep(1000);
                    DebugPrint("Listening...");
                    return;
                }
                DebugPrint("No network interface founded...");
            }
            catch (Exception )
            {
                DebugPrint("Error");
                //e.LogMe("InitializeNetwork");
            }
        }


        #endregion




        #region Helpers
        /// <summary>
        ///     Prints message to debug and COM
        /// </summary>
        /// <param name="message"></param>
        public void DebugPrint(string message)
        {
            if (message == null || message == string.Empty)
                return;
            var stringBuilder = new StringBuilder();
            stringBuilder.Append(message);
            Debug.Print(stringBuilder.ToString());
            stringBuilder.Append("EndMessage");
            var bytes = Encoding.UTF8.GetBytes(stringBuilder.ToString());
            if (usbSerial1 != null)
                usbSerial1.SerialLine.Write(bytes, 0, bytes.Length);
        }

        #endregion
    }
}

Client code


using System;
using System.Diagnostics;
using System.Net;
using System.Net.Sockets;
using System.Threading;

namespace SocketHdyraClient
{
    class Program
    {
        private const int TOTAL_TRIES = 50;
        static void Main(string[] args)
        {
            for (int i = 0; i < 50; i++)
            {
                Console.WriteLine("Stressing Hydra attempt {0} from {1}",i,TOTAL_TRIES);
                try
                {
                    if (!ConnectSocket())
                    {
                        Console.WriteLine("Failed to connect");
                        return;
                    }
                    Console.WriteLine("Restarting Hydra");
                    var p = Process.Start(@ "C:\Program Files (x86)\Microsoft .NET Micro Framework\v4.2\Tools\MFDeploy.exe",
                                  "Reboot /I:USB:FEZ Hydra_Gadgeteer");
                    p.WaitForExit();

                }
                catch (Exception e)
                {
                    Console.WriteLine("Unexpected exception:{0}",e.Message);
                    return;
                }
                Console.WriteLine("Entering sleep for 5 seconds");
                for (int j = 0; j < 5; j++)
                {
                    Console.Write(".");
                    Thread.Sleep(TimeSpan.FromSeconds(1));
                }
            }
        }


        private static bool ConnectSocket()
        {
            var socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
            var ep = new IPEndPoint(IPAddress.Parse("10.12.227.200"), 81);
            var result = socket.BeginConnect(ep, null, null);

            bool success = result.AsyncWaitHandle.WaitOne(TimeSpan.FromSeconds(10), true);
            if (success)
            {
                socket.EndConnect(result);
            }
            else
            {
                socket.Close();
                return false;
            }

            return true;

        }


    }
}


DeviceInfo:
HAL build info: 4.2.0.0, Microsoft Copyright (C) Microsoft Corporation. All rig
OEM Product codes (vendor, model, SKU): 255, 0, 65535
Serial Numbers (module, system):
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
Solution Build Info: 4.2.5.0, Copyright (C) GHI Electronics, LLC
AppDomains:
default, id=1
Assemblies:
mscorlib,4.2.0.0
Microsoft.SPOT.Native,4.2.0.0
Microsoft.SPOT.Hardware,4.2.0.0
Microsoft.SPOT.Graphics,4.2.0.0
Microsoft.SPOT.TinyCore,4.2.0.0
Microsoft.SPOT.IO,4.2.0.0
System.IO,4.2.0.0
Microsoft.SPOT.Hardware.Usb,4.2.0.0
Microsoft.SPOT.Hardware.SerialPort,4.2.0.0
Microsoft.SPOT.Touch,4.2.0.0
Microsoft.SPOT.Ink,4.2.0.0
Microsoft.SPOT.Hardware.PWM,4.2.0.1
Microsoft.SPOT.Hardware.OneWire,4.2.0.0
System.Xml,4.2.0.0
Microsoft.SPOT.Time,4.2.0.0
Microsoft.SPOT.Net,4.2.0.0
System,4.2.0.0
Microsoft.SPOT.Net.Security,4.2.0.0
System.Net.Security,4.2.0.0

Hydra, client and COM listener projects can be mailed for debugging or reproduction purposes.

According to COM port prints the code fails on:


NetworkInterface[] interfaces = NetworkInterface.GetAllNetworkInterfaces() ;

or


 if (interfaces != null && interfaces.Length > 0)

lines

I have a similar problem that the Hydra just bricks after a few power cycles. It will run for weeks as long as it doesn’t lose power, but one or two reboots and it bricks until I reload the firmware. They keep telling me it is probably a power issue and that the Hydra is too hungry to rely on just USB power. I’m going to try and switch back to a Cerberus if I can get around the memory limits.

Even if it is a power issue it shouldn’t result in corrupted firmware in my opinion. Something for GHI to look at.

does this happen when you load firmware? Also, please d no start multiple thread with the same question as we are not sure which thread to respond to now :slight_smile:

Redoing the firmware always fixes it for me and it doesn’t recognize until I do. I also have to use the SD method to get back into the bootloader.

I agree that it “shouldn’t” corrupt it, but that is what seems to happen. And I have it happen on both Hydra’s I have. But, when I have them running they seem to have no issue and my current monitor never shows them going above 450mA and the supply I use supports much higher.

I do find I have a bit better luck if I abort the running app before unplugging, I added a button with an interrupt that kills it, though that shouldn’t be necessary and it isn’t 100% effective.

does your app write to EWR? Does it update network settings?

Can you make hydra fail with simple blink LED app?

For me:

Yes I can make it fail with just a blink LED app. I can make it fail with no app, and I can do it to both Hydra’s I own.

My app that is running on Hydra is using the network and the code sets a static IP. No to writing to EWR because I am not sure what EWR is, but I don’t write to anything except the network.

Hello,

  1. The latest response for the thread I mentioned was 8 months ago.
  2. The code above does nothing, but only initializes the NIC with static IP. This is enough to make to FW corruption. We are using more than 20 Hydras and experiencing the same issue on all of them

So to be sure, all I have to do is put the latest firmware on hydra, load a blink LED program and then use power pack to power the board on and off and after few tries it will stop booting?

This seems impossible as there is nothing that writes to flash so what would be corrupting it?!

Would you please try the above again and confirm?

Can anyone else confirm they see this with same exact steps above?

Gus,
Does NIC setting to static IP writes to EWR?
If not It does even less than you ask from PintSize.Me and the issue reproduced.
I can configrm that we have this issue on more 20 Hydras we are working with.

@ Alexr111 - yes, writing network data, like, set ststic ip, dhcp, or else, will write to flash.

Please try it with a blink LED console app, nothing else.

What SPI Flash chip is used on Hydra? I don’t have mine at the moment to check.

Gus, I can confirm this. Of all my boards I use the Hydra the most and I definately have to reflash after a few reboots, I have never actually counted but 10 to 15 sounds about right. Btw. I use the non-Ethernet firmware.

I believe you guys but this is not helping in narrowing down the problem. We need to now if this ever happen with no write to flash. Meaning, you are not writing network config in your code, not LCD config, not even deploying to your device.

In other words, back to the question above, does it happen if you only blink LED and reboot few times. Even if you think it happened in the past, please try it now.

Hopefully we can get this sorted out with your help.

I’m stressing right now with LED blinking only passed 20 iterations.
I will keep you update

and you are NOT deploying 20 times, you are only hitting reset on the app the was previously deployed?

Blinking only code passed 130 iterations (the previous version failed after 15-20 reboots) , looks like the reboot is not an issue!

Gus,
Let me know if you need me to do any additional tests.

@ Gus - I will run a test this weekend, but I can tell the following in the meantime.

  1. Deploying multiple times with updated code, eventually VS reports that it cannot connect to the device after which attempts to connect to the board with MFDeploy fails (This was before I was using the new GHI Tool)

  2. The board hangs (My fault, bad RLP etc.) so I unplug the USB cable from the DP module then redeploy after a few times doing this, when I plug the USB in the windows driver load fails with a unidentified device. This looks more like the Windows driver crashes, because I sometimes have to reboot my machine to recover from this situation.

  3. Same as 2 except that reflashing the Hydra resolves the issue.

I never raised a concern around this because most of the time I am loading RLP code or a custom firmware onto the board so I always assume that it is just something I have done until proven otherwise. But since others are experiencing similar issues, maybe it is not entirely my bad code :).

Like I said, I will try do a detailed test this weekend and hopefully be able to provide more precise details around the particular scenarios I have seen.