Main Site Documentation

Need a working HTTPS WebClient


#1

Has anyone managed to get https working with the webclient?
I can get http to work but otherwise no look.

I’ve tried:
https://www.ghielectronics.com/community/codeshare/entry/614

http://netmftoolbox.codeplex.com/SourceControl/latest#Samples/Visual C#/Web client/Program.cs

I’m using a Fez Spider with an ethernet_JD11 module.

If anyone has any help I would greatly appreciate it.

#edit second link not showing
netmftoolbox.codeplex.com/SourceControl/latest#Samples/Visual C#/Web client/Program.cs


#2

Thanks for the sample code, but I’m still struggling with https

using the sample code, it gets an exception when trying to get a response in PrintHttpData()

The exception is as follows:
#### Exception System.NotSupportedException - CLR_E_NOT_SUPPORTED (1) ####
#### Message:
#### Microsoft.SPOT.Net.Security.SslNative::SecureClientInit [IP: 0000] ####
#### Microsoft.SPOT.Net.Security.SslStream::Authenticate [IP: 0051] ####
#### Microsoft.SPOT.Net.Security.SslStream::AuthenticateAsClient [IP: 000c] ####
#### System.Net.HttpWebRequest::EstablishConnection [IP: 023d] ####
#### System.Net.HttpWebRequest::SubmitRequest [IP: 0043] ####
#### System.Net.HttpWebRequest::GetResponse [IP: 000c] ####
#### PrototypeBasestation.wc::getRequest [IP: 0026] ####
#### PrototypeBasestation.Program::timer_Tick [IP: 0017] ####
#### Gadgeteer.Timer::dt_Tick [IP: 0018] ####
#### Microsoft.SPOT.DispatcherTimer::FireTick [IP: 0010] ####
#### Microsoft.SPOT.Dispatcher::PushFrameImpl [IP: 0054] ####
#### Microsoft.SPOT.Dispatcher::PushFrame [IP: 001a] ####
#### Microsoft.SPOT.Dispatcher::Run [IP: 0006] ####
#### Gadgeteer.Program::Run [IP: 0020] ####
A first chance exception of type ‘System.NotSupportedException’ occurred in Microsoft.SPOT.Net.Security.dll
A first chance exception of type ‘System.NotSupportedException’ occurred in System.Net.Security.dll
#### Exception System.Net.WebException - 0x00000000 (1) ####
#### Message:
#### System.Net.HttpWebRequest::GetResponse [IP: 00d3] ####
#### PrototypeBasestation.wc::getRequest [IP: 0026] ####
#### PrototypeBasestation.Program::timer_Tick [IP: 0017] ####
#### Gadgeteer.Timer::dt_Tick [IP: 0018] ####
#### Microsoft.SPOT.DispatcherTimer::FireTick [IP: 0010] ####
#### Microsoft.SPOT.Dispatcher::PushFrameImpl [IP: 0054] ####
#### Microsoft.SPOT.Dispatcher::PushFrame [IP: 001a] ####
#### Microsoft.SPOT.Dispatcher::Run [IP: 0006] ####
#### Gadgeteer.Program::Run [IP: 0020] ####
A first chance exception of type ‘System.Net.WebException’ occurred in System.Http.dll


// Root CA Certificate needed to validate HTTPS servers.
            byte[] ca = Resource1.GetBytes(
                Resource1.BinaryResources.VerisignCA);

            X509Certificate[] caCerts =
                new X509Certificate[] { new X509Certificate(ca) };

            // Initialize the default webproxy to be used by all 
            // HttpWebRequests.
            // Change the proxy address to fit your environment.
            HttpWebRequest.DefaultWebProxy = new WebProxy();
                //new WebProxy("itgproxy.dns.microsoft.com", true);

            // Test SSL connection with no certificate verification
            PrintHttpData("https://www.google.com/accounts/ManageAccount/", null);

            // Read from secure webpages by using the Verisign Root CA 
            // certificate that is stored in the Resource1.resx file.
            PrintHttpData("https://www.google.com/accounts/ManageAccount/",
                caCerts);
...

public static void PrintHttpData(string url, X509Certificate[] caCerts)
        {
            // Create an HTTP Web request.
            HttpWebRequest request =
                HttpWebRequest.Create(url) as HttpWebRequest;

            // Assign the certificates. The value must not be null if the
            // connection is HTTPS.
            request.HttpsAuthentCerts = caCerts;

            // Set request.KeepAlive to use a persistent connection. 
            request.KeepAlive = true;

            // Get a response from the server.
            WebResponse resp = null;

            try
            {
                resp = request.GetResponse(); //HTTPS exception occurs here
            }
            catch (Exception e)
            {
                Debug.Print("Exception in HttpWebRequest.GetResponse(): " +
                    e.ToString());
            }
            ...
        }


#3

My board is a Fez Spider with .NetMF v4.2

I’m not sure how to do that.


#4

I don’t have fez config, but I do have MFDeploy.
When I try and update ssl its responds:

“Failure - Device {0} is invalid or not responding”


#5

Starting to get somewhere.
I uninstalled all the old drivers and installed the latest sdk.
i used fez config to update the firmware and update the ssl.

I now get a webpage response from https with no certificate.

but i get an exception from https with a certificate:

#### Exception System.Net.Sockets.SocketException - CLR_E_FAIL (1) ####
#### Message: 
#### Microsoft.SPOT.Net.Security.SslNative::SecureConnect [IP: 0000] ####
#### Microsoft.SPOT.Net.Security.SslStream::Authenticate [IP: 0060] ####
#### Microsoft.SPOT.Net.Security.SslStream::AuthenticateAsClient [IP: 000c] ####
#### System.Net.HttpWebRequest::EstablishConnection [IP: 023d] ####
#### System.Net.HttpWebRequest::SubmitRequest [IP: 0043] ####
#### System.Net.HttpWebRequest::GetResponse [IP: 000c] ####
#### PrototypeBasestation.WebClient::getRequest [IP: 003e] ####
#### PrototypeBasestation.Program::timer_Tick [IP: 0034] ####
#### Gadgeteer.Timer::dt_Tick [IP: 0018] ####
#### Microsoft.SPOT.DispatcherTimer::FireTick [IP: 0010] ####
#### Microsoft.SPOT.Dispatcher::PushFrameImpl [IP: 0054] ####
#### Microsoft.SPOT.Dispatcher::PushFrame [IP: 001a] ####
#### Microsoft.SPOT.Dispatcher::Run [IP: 0006] ####
#### Gadgeteer.Program::Run [IP: 0020] ####
#### SocketException ErrorCode = 1
#### SocketException ErrorCode = 1

A first chance exception of type ‘System.Net.Sockets.SocketException’ occurred in Microsoft.SPOT.Net.Security.dll
#### SocketException ErrorCode = 1
#### SocketException ErrorCode = 1
#### SocketException ErrorCode = 1
#### SocketException ErrorCode = 1
A first chance exception of type ‘System.Net.Sockets.SocketException’ occurred in System.Net.Security.dll
#### SocketException ErrorCode = 1
#### SocketException ErrorCode = 1
#### Exception System.Net.WebException - 0x00000000 (1) ####
#### Message:
#### System.Net.HttpWebRequest::GetResponse [IP: 00c8] ####
#### PrototypeBasestation.WebClient::getRequest [IP: 003e] ####
#### PrototypeBasestation.Program::timer_Tick [IP: 0034] ####
#### Gadgeteer.Timer::dt_Tick [IP: 0018] ####
#### Microsoft.SPOT.DispatcherTimer::FireTick [IP: 0010] ####
#### Microsoft.SPOT.Dispatcher::PushFrameImpl [IP: 0054] ####
#### Microsoft.SPOT.Dispatcher::PushFrame [IP: 001a] ####
#### Microsoft.SPOT.Dispatcher::Run [IP: 0006] ####
#### Gadgeteer.Program::Run [IP: 0020] ####
A first chance exception of type ‘System.Net.WebException’ occurred in System.Http.dll


#6

This type of issue can pop up when using an invalid SSL Cert. The certification they include in the examples expired some months ago if I recall, so you will need to use a certificate other than what is included in Microsoft’s SSL example.


#7

Okay,
So I tried importing some roots certificate from http://www.symantec.com/page.jsp?id=roots and didn’t get anywhere.

Since it’s Google in general which I’m wishing to connect to;
I tried using an exported base64format.cer from Google https.
More specifically “GeoTrust Global CA”

I still seem to be getting the same error.

Is there any further instruction you can give me of how to update the certificate?

Thank you.


#8

I don’t really understand what you’re attempting to do. I’m pretty sure that this is how this works: to access a HTTPS source from your device, you don’t need a cert on your device, you would need to verify the cert the server presented was valid… and if you want to offer an HTTPS connection from a browser to your device you need a valid cert.


#9

ssl isn’t something I’ve done before.

Looking at the sample code provided in
C:\Users{username}\Documents\Microsoft .NET Micro Framework 4.2\Samples\HttpClient

it has a verisignCA.cer in the resources

at the moment I’m trying to do


// Root CA Certificate needed to validate HTTPS servers.
byte[] ca = Resources.GetBytes(Resources.BinaryResources.VerisignCA);
X509Certificate[] caCerts = new X509Certificate[] { new X509Certificate(ca) };
...

PrintHttpData("https://www.google.com/accounts/ManageAccount/", caCerts);

It fails most likely with the exception I gave previously because the cert is out of date.
I’m now trying to update that cert so that it is valid.

then hopefully it won’t crash when it gets to this part of the code

 
PrintHttpData(string url, X509Certificate[] caCerts )
{
...
   try
   {
      resp = request.GetResponse(); //crashes here with https 
   }
   catch (Exception e)
   {
      Debug.Print(e.Message + "\n");
      Debug.Print("Exception in HttpWebRequest.GetResponse(): " + e.ToString());
    }
...


#10

Please can some explain how to do this correctly for a fez spider as I can’t seem to get it right.
The SSL Certs seem to work fine when you test the sample with the emulator, but not with my device.


#11

Call me on it if I’m wrong but, to me that looks like the sample code with the networking code for an ethernet module included.

I don’t have a problem with my ethernet module, I can connect to the network and internet fine with it. Never the less, I went along and took the sample code and added code for the ethernet module.
I’ve tried it with the verison in the GHI.Premium libraries and the one in Gadgeteer.Modules.GHIElectronics.Ethernet_JD11

Both times I come out with the same result which is the following exception:

#### Exception System.Net.Sockets.SocketException - CLR_E_FAIL (1) ####
#### Message: 
#### Microsoft.SPOT.Net.Security.SslNative::SecureConnect [IP: 0000] ####
#### Microsoft.SPOT.Net.Security.SslStream::Authenticate [IP: 0060] ####
#### Microsoft.SPOT.Net.Security.SslStream::AuthenticateAsClient [IP: 000c] ####
#### System.Net.HttpWebRequest::EstablishConnection [IP: 023d] ####
#### System.Net.HttpWebRequest::SubmitRequest [IP: 0019] ####
#### System.Net.HttpWebRequest::GetResponse [IP: 000c] ####
#### GadgeteerApp1.Program::ProgramStarted [IP: 0089] ####
#### GadgeteerApp1.Program::Main [IP: 0015] ####
#### SocketException ErrorCode = 1
#### SocketException ErrorCode = 1

A first chance exception of type ‘System.Net.Sockets.SocketException’ occurred in Microsoft.SPOT.Net.Security.dll
#### SocketException ErrorCode = 1
#### SocketException ErrorCode = 1
#### SocketException ErrorCode = 1
#### SocketException ErrorCode = 1
A first chance exception of type ‘System.Net.Sockets.SocketException’ occurred in System.Net.Security.dll
#### SocketException ErrorCode = 1
#### SocketException ErrorCode = 1
#### Exception System.Net.WebException - 0x00000000 (1) ####
#### Message:
#### System.Net.HttpWebRequest::GetResponse [IP: 00c8] ####
#### GadgeteerApp1.Program::ProgramStarted [IP: 0089] ####
#### GadgeteerApp1.Program::Main [IP: 0015] ####
A first chance exception of type ‘System.Net.WebException’ occurred in System.Http.dll
Exception in HttpWebRequest.GetResponse(): System.Net.WebException

The code looks


using System;
using System.IO;
using System.Net;
using System.Text;
using System.Threading;
using Microsoft.SPOT;
using Microsoft.SPOT.Net.Security;
using System.Security.Cryptography.X509Certificates;
using GHI.Premium.Net;
using System.Collections;

using Gadgeteer.Networking;
using GT = Gadgeteer;
using GTM = Gadgeteer.Modules;

namespace HttpClientSample
{
    public class MyHttpClient
    {

        /// <summary>
        /// Retrieves a page from a Web server, using a simple GET request.
        /// </summary>
        public static void Main()
        {
            EthernetBuiltIn ethernet = new EthernetBuiltIn();
            
            ethernet.Open();
            ethernet.NetworkInterface.EnableDhcp();
            ethernet.NetworkInterface.EnableDynamicDns();

            NetworkInterfaceExtension.AssignNetworkingStackTo(ethernet);
            Microsoft.SPOT.Hardware.Utility.SetLocalTime(new DateTime(2013, 11, 14));

            // Wait for DHCP (on LWIP devices)
            while (true)
            {
                IPAddress ip = IPAddress.GetDefaultLocalAddress();

                if (ip != IPAddress.Any) break;

                Thread.Sleep(1000);
            }
            // Root CA Certificate needed to validate HTTPS servers.
            byte[] ca = Resource1.GetBytes(
                Resource1.BinaryResources.VerisignCA);

            X509Certificate[] caCerts = 
                new X509Certificate[] { new X509Certificate(ca) };

            // Initialize the default webproxy to be used by all 
            // HttpWebRequests.
            // Change the proxy address to fit your environment.
            HttpWebRequest.DefaultWebProxy = null;

            // Print the HTTP data from each of the following pages.
            PrintHttpData("http://autos.msn.com/default.aspx", null);
            PrintHttpData("http://www.nytimes.com/", null);
            PrintHttpData("https://www.google.com/accounts/ManageAccount/", null);

            // Read from secure webpages by using the Verisign Root CA 
            // certificate that is stored in the Resource1.resx file.
           
            //This crashes with the above exception
              PrintHttpData("https://www.google.com/accounts/ManageAccount/", 
              caCerts);
              
        }
        

        /// <summary>
        /// Prints the HTTP Web page from the given URL and status data while 
        /// receiving the page.
        /// </summary>
        /// <param name="url">The URL of the page to print.</param>
        /// <param name="caCerts">The root CA certificates that are required for 
        /// validating a secure website (HTTPS).</param>
        public static void PrintHttpData(string url, X509Certificate[] caCerts)
        {
            // Create an HTTP Web request.
            HttpWebRequest request =
                HttpWebRequest.Create(url) as HttpWebRequest;

            // Assign the certificates. The value must not be null if the
            // connection is HTTPS.
            request.HttpsAuthentCerts = caCerts;

            // Set request.KeepAlive to use a persistent connection. 
            request.KeepAlive = true;

            // Get a response from the server.
            WebResponse resp = null;

            try
            {
                resp = request.GetResponse(); //This is where the exception is caused
            }
            catch (Exception e)
            {
                Debug.Print("Exception in HttpWebRequest.GetResponse(): " +
                    e.ToString());
            }

            // Get the network response stream to read the page data.
            if (resp != null)
            {
                Stream respStream = resp.GetResponseStream();
                string page = null;
                byte[] byteData = new byte[4096];
                char[] charData = new char[4096];
                int bytesRead = 0;
                Decoder UTF8decoder = System.Text.Encoding.UTF8.GetDecoder();
                int totalBytes = 0;

                // allow 5 seconds for reading the stream
                respStream.ReadTimeout = 5000;

                // If we know the content length, read exactly that amount of 
                // data; otherwise, read until there is nothing left to read.
                if (resp.ContentLength != -1)
                {
                    for (int dataRem = (int)resp.ContentLength; dataRem > 0; )
                    {
                        Thread.Sleep(500);
                        bytesRead =
                            respStream.Read(byteData, 0, byteData.Length);
                        if (bytesRead == 0)
                        {
                            Debug.Print("Error: Received " +
                                (resp.ContentLength - dataRem) + " Out of " +
                                resp.ContentLength);
                            break;
                        }
                        dataRem -= bytesRead;

                        // Convert from bytes to chars, and add to the page 
                        // string.
                        int byteUsed, charUsed;
                        bool completed = false;
                        totalBytes += bytesRead;
                        UTF8decoder.Convert(byteData, 0, bytesRead, charData, 0,
                            bytesRead, true, out byteUsed, out charUsed,
                            out completed);
                        page = page + new String(charData, 0, charUsed);

                        // Display the page download status.
                        Debug.Print("Bytes Read Now: " + bytesRead +
                            " Total: " + totalBytes);
                    }

                    page = new String(
                        System.Text.Encoding.UTF8.GetChars(byteData));
                }
                else
                {
                    // Read until the end of the data is reached.
                    while (true)
                    {
                        // If the Read method times out, it throws an exception, 
                        // which is expected for Keep-Alive streams because the 
                        // connection isn't terminated.
                        try
                        {
                            Thread.Sleep(500);
                            bytesRead =
                                respStream.Read(byteData, 0, byteData.Length);
                        }
                        catch (Exception)
                        {
                            bytesRead = 0;
                        }

                        // Zero bytes indicates the connection has been closed 
                        // by the server.
                        if (bytesRead == 0)
                        {
                            break;
                        }

                        int byteUsed, charUsed;
                        bool completed = false;
                        totalBytes += bytesRead;
                        UTF8decoder.Convert(byteData, 0, bytesRead, charData, 0,
                            bytesRead, true, out byteUsed, out charUsed,
                            out completed);
                        page = page + new String(charData, 0, charUsed);

                        // Display page download status.
                        Debug.Print("Bytes Read Now: " + bytesRead +
                            " Total: " + totalBytes);
                    }

                    Debug.Print("Total bytes downloaded in message body : "
                        + totalBytes);
                }

                // Display the page results.
                Debug.Print(page);

                // Close the response stream.  For Keep-Alive streams, the 
                // stream will remain open and will be pushed into the unused 
                // stream list.
                resp.Close();
            }
        }
        
    }
}

I’ve been stuck at this point so long now I feel it may be a hardware error, as I said before it runs fine on the emulator (as long as you don’t include the GHI.Premium part).


#12

This has made an infinite loop now.

Then the memory leak kicks in.

#### Exception System.OutOfMemoryException - CLR_E_OUT_OF_MEMORY (1) ####
#### Message: 
#### System.Net.IPEndPoint::.ctor [IP: 0004] ####
#### System.Net.IPEndPoint::Create [IP: 0048] ####
#### System.Net.Sockets.Socket::GetEndPoint [IP: 003e] ####
#### System.Net.Sockets.Socket::get_RemoteEndPoint [IP: 0005] ####
#### Microsoft.SPOT.Net.Security.SslStream::.ctor [IP: 0006] ####
#### System.Net.HttpWebRequest::EstablishConnection [IP: 021b] ####
#### System.Net.HttpWebRequest::SubmitRequest [IP: 0019] ####
#### System.Net.HttpWebRequest::GetResponse [IP: 000c] ####
#### HttpClientSample.MyHttpClient::Main [IP: 007a] ####

GC: 35msec 7327596 bytes used, 12072 bytes available
Type 0F (STRING ): 1428 bytes
Type 11 (CLASS ): 8628 bytes
Type 12 (VALUETYPE ): 1476 bytes
Type 13 (SZARRAY ): 4704 bytes
Type 03 (U1 ): 1704 bytes
Type 04 (CHAR ): 444 bytes
Type 07 (I4 ): 36 bytes
Type 0F (STRING ): 72 bytes
Type 11 (CLASS ): 2364 bytes
Type 12 (VALUETYPE ): 84 bytes
Type 15 (FREEBLOCK ): 12072 bytes
Type 17 (ASSEMBLY ): 34116 bytes
Type 18 (WEAKCLASS ): 48 bytes
Type 19 (REFLECTION ): 192 bytes
Type 1B (DELEGATE_HEAD ): 360 bytes
Type 1D (OBJECT_TO_EVENT ): 72 bytes
Type 1E (BINARY_BLOB_HEAD ): 7268184 bytes
Type 1F (THREAD ): 1152 bytes
Type 20 (SUBTHREAD ): 144 bytes
Type 21 (STACK_FRAME ): 2640 bytes
Type 22 (TIMER_HEAD ): 72 bytes
Type 27 (FINALIZER_HEAD ): 192 bytes
Type 28 (MEMORY_STREAM_HEAD ): 36 bytes
Type 29 (MEMORY_STREAM_DATA ): 396 bytes
Type 31 (IO_PORT ): 72 bytes
Type 34 (APPDOMAIN_HEAD ): 72 bytes
Type 36 (APPDOMAIN_ASSEMBLY ): 3612 bytes
Failed allocation for 39 blocks, 468 bytes


#13

seems like there is an error here

[quote]PrintHttpData(“https://www.google.com/accounts/ManageAccount/”, null);


#14

@ eddie_garmon - Not being funny but that exception came from

PrintHttpData(“https://www.google.com/accounts/ManageAccount/”, caCerts);

NOT

PrintHttpData(“https://www.google.com/accounts/ManageAccount/”, null);

however if you mean the string is wrong, then you’re right the website is now something more like this
https://www.google.com/settings/account
but for the considerations of getting to just work in the first place it’s perfectly legal as Google still sends a valid webpage back.

More to the point there is quite a big memory leak when doing
PrintHttpData(someURL, caCerts);
Which is mentioned here https://netmf.codeplex.com/workitem/2005

The guy pointed out the exact line of code where the problem is, and as far as I know the problem isn’t actually solved yet.

For anyone else who is looking for a solution to this I would currently suggest writing your own gateway server to go and get said webpage.
If you want to do httpRequests or httpsRequests(with no auth) then you are fine with the sample code.


#15

I have had zero luck getting an https url passing no certs… as what I was pointing to. not the url itself


#16

Oh okay, I had to update my firmware to the latest version before I got that to work; which is 4.2.11.1

Note: if you haven’t, you need to update both TinyCLR and TinyBooter using fez config. It was a real bugger to update as well(for me).

If that’s not your problem throw some code at me and I will try help.


#17

@ JPenny1993 - Help us improve our products: please create an entry in our tracker and detail the problems/difficulties you encountered. https://www.ghielectronics.com/tracker/entry/create


#18

@ JPenny1993 - You could try this tutorial for creating your own certificate:

http://wiki.tinyclr.com/index.php?title=Create_a_self-signed_x509_certificate


#19

I’m having the same kind of problem with my Hydra, running v4.2.6.1.

It only happens when I try to call HTTPS.
I’ll get a socket exception in
System.Net.InputNetworkStreamWrapper::Read_HTTP_Line
System.Net.HttpWebRequest::ParseHTTPResponse
System.Net.HttpWebRequest::GetResponse

Everything works fine with normal HTTP, but unfortunately that isn’t an option.

I’m pretty new with gadgeteer, but I’ve been following the examples and I’ve been having no luck.


#20

just to make sure… it works fine in the emulator, but not running on the actual device.