Https data stream issue

Hello all,
I am trying to send some data using data stream to DigiCloud. I managed to write this function:


try
{
	byte[] data = UTF8Encoding.UTF8.GetBytes("{\r\n\"  \"stream_id\": \"G120M_test\",\r\n   \"stream_type\": \"STRING\",\r\n  \"value\": \"" + DateTime.Now.ToString() + "\"\r\n}\r\n");

	Uri url = new Uri("https://devicecloud.digi.com/ws/v1/streams/history");
	HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
	Debug.Print("Adres URL: " + url.OriginalString);

	request.Method = "POST";

	String username = "username";
	String password = "password";
	String encoded = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(username + ":" + password));
	request.Headers.Add("Authorization", "Basic " + encoded);
	request.ContentLength = data.Length;
	request.ContentType = "application/json";
	request.Accept = "application/json";

	Debug.Print("HttpWebRequest: " + request.GetRequestStream().Length);

	Stream dataStream = request.GetRequestStream();
	dataStream.Write(data, 0, data.Length);
	dataStream.Close();

	WebResponse response = request.GetResponse();
	request.Dispose();

	Debug.Print(((HttpWebResponse)response).StatusDescription);
	Stream respData = response.GetResponseStream();
	StreamReader reader = new StreamReader(dataStream);

	string responseFromServer = reader.ReadToEnd();
	Debug.Print(responseFromServer);
	reader.Close();
	dataStream.Close();
	response.Close();        
}

catch (Exception e)
{
	byte[] buffer = Encoding.UTF8.GetBytes("Error:" + e);
	UART.Write(buffer, 0, buffer.Length);

	Debug.Print("Error:" + e);
}

I had searched over the Internet and found solution to update SSL feed. However, that didn’t work for me. I’m still getting exceptions:

[quote] #### Exception System.Net.Sockets.SocketException - CLR_E_FAIL (1) ####
#### Message:
#### Microsoft.SPOT.Net.Security.SslNative::SecureRead [IP: 0000] ####
#### Microsoft.SPOT.Net.Security.SslStream::Read [IP: 0040] ####
#### System.Net.InputNetworkStreamWrapper::RefillInternalBuffer [IP: 0038] ####
#### System.Net.InputNetworkStreamWrapper::Read_HTTP_Line [IP: 004b] ####
#### System.Net.HttpWebRequest::ParseHTTPResponse [IP: 002e] ####
#### System.Net.HttpWebRequest::GetResponse [IP: 0035] ####
#### Drzemlik_brama_G120E.DeviceCloud::.ctor [IP: 00c8] ####
#### Drzemlik_brama_G120E.Program::Main [IP: 0061] ####
#### SocketException ErrorCode = -1
#### SocketException ErrorCode = -1
A first chance exception of type ‘System.Net.Sockets.SocketException’ occurred in Microsoft.SPOT.Net.Security.dll
#### SocketException ErrorCode = -1
#### SocketException ErrorCode = -1
#### Exception System.Net.WebException - 0x00000000 (1) ####
#### Message:
#### System.Net.HttpWebRequest::GetResponse [IP: 00c8] ####
#### Drzemlik_brama_G120E.DeviceCloud::.ctor [IP: 00c8] ####
#### Drzemlik_brama_G120E.Program::Main [IP: 0061] ####
A first chance exception of type ‘System.Net.WebException’ occurred in System.Http.dll
Error:System.Net.WebException[/quote]

Did somebody have similiar problem and please could help me?

Regards

1 Like

Could be cert related. Have you tried creating a request to a known valid secure location?

1 Like

I could mangage to send data to the cloud from NETMF devices using https but I did not succeed in every case. Perhaps you can get some hints from the following CodeShare entries:
https://www.ghielectronics.com/community/codeshare/entry/1095
https://www.ghielectronics.com/community/forum/topic?id=24376&page=1#msg225838
https://www.ghielectronics.com/community/codeshare/entry/1082

If the server supports only the newer encrytion protocols you will not succeed with NETMF
https://www.ghielectronics.com/community/forum/topic?id=23295

Hopefully GHI’s TinyCLR will support TLS 1.2 soon.

Thank you for your responses.

I don’t think so, I have generated code by DigiCloud to put data on the stream. The code is in C# but uses classes, which aren’t available on .NET MF. Here is the code, there aren’t any additional certificates and it’s working on a PC Console application like a charm:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
using System.IO;
using System.Web;

/**
 * This is a stub class with a main method to run an Device Cloud web service.
 */
    class WebServiceRequest
    {
    	/**
	    * Run the web service request
	    */
        static void Main(string[] args)
        {
            try
            {
            	// Create url to the Device Cloud server for a given web service request
                Uri url = new Uri("https://devicecloud.digi.com/ws/v1/streams/history");
                HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);

                request.Method = "POST";
                // replace with your username/password
                String user = "YourUsername";
                String pwd = "YourPassword";

                CredentialCache credCache = new CredentialCache();
                credCache.Add(url, "Basic", new NetworkCredential(user, pwd));

                request.Credentials = credCache;
                
                    request.ContentType = "application/json";
                    request.Accept = "application/json";
                    StreamWriter writer = new StreamWriter(request.GetRequestStream());
                   
                  writer.Write("{\r\n");
writer.Write("  \"stream_id\": \"myStream\",\r\n");
writer.Write("  \"stream_type\": \"DOUBLE\",\r\n");
writer.Write("  \"value\": \"42\"\r\n");
writer.Write("}\r\n");
                    writer.Close();
               
				   //Get response
                   WebResponse serverResponse = (WebResponse)request.GetResponse();
                    Stream output = serverResponse.GetResponseStream();

                    Encoding encode = System.Text.Encoding.GetEncoding("utf-8");
                    StreamReader readStream = new StreamReader(output, encode);
                    Char[] read = new Char[256];

                    // Read 256 characters at a time.    
                    int count = readStream.Read(read, 0, 256);

                    while (count > 0)
                    {
                        // Dump the 256 characters on a string and display the string onto the console.
                        String str = new String(read, 0, count);
                        Console.Write(str);
                        count = readStream.Read(read, 0, 256);
                    }

                    Console.WriteLine("");
                    // Release the resources of stream object.
                    readStream.Close();
                    serverResponse.Close();

            }
            catch (Exception e)
            {	
            	// Print any exceptions that occur
                Console.Write(e.StackTrace);
            }

        }
    }

.NET MF is missing CredentialCache class so my problem will be probably related to authorization?

[quote=“RoSchmi”]I could mangage to send data to the cloud from NETMF devices using https but I did not succeed in every case. Perhaps you can get some hints from the following CodeShare entries:
https://www.ghielectronics.com/community/codeshare/entry/1095
https://www.ghielectronics.com/community/forum/topic?id=24376&page=1#msg225838
https://www.ghielectronics.com/community/codeshare/entry/1082

If the server supports only the newer encrytion protocols you will not succeed with NETMF
https://www.ghielectronics.com/community/forum/topic?id=23295

Hopefully GHI’s TinyCLR will support TLS 1.2 soon.[/quote]

Thanks for there links, I will look deeper into them in a moment. I checked the certificate, here is the result: https://www.ssllabs.com/ssltest/analyze.html?d=devicecloud.digi.com. Will it work with .NET MF on G120E board?

I made few tests and what’s interesting, after adding all headers I check the length of the request using request.GetRequestStream().Length and it says zero. Is it normal?
Also, when I remove the ‘s’ from ‘https’ in URL I receive “Bad Request” response. When I additionally remove any credentials (I tried two methods) I get “Unauthorized” response, so I can assume that board is sending something to the stream, but security fails. Am I right?
1)

 
2) 
```cs
String encoded = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(username + ":" + password)); 
request.Headers.Add("Authorization", "Basic " + encoded);

Regards

1 Like

On a PC Windows holds a list of trusted Root Certificates. On a NETMF device there isn’t such a list.
So you must include the root certificate of the server you want to access via https.
Here you can find a tutorial how to get the certificate

Ok I exported the certificate and added it to BinaryResources. I try to add it using this code:


byte[] cert = System.Text.Encoding.UTF8.GetBytes(Resources.BinaryResources.certificate.ToString());
request.HttpsAuthentCerts = new X509Certificate[] { new X509Certificate(cert) };

I get these exceptions.

 ####
 #### System.Security.Cryptography.X509Certificates.X509Certificate::.ctor [IP: 0023] ####
 #### System.Security.Cryptography.X509Certificates.X509Certificate::.ctor [IP: 0008] ####

What I’m doing wrong?

Edit: Before using the certificate I’m setting succesfully the time and date.

Edit 2:
I’ve managed to add the certificate. I test it reading the expiration date and it’s ok. Here is the code:

 cert = Resources.GetBytes(Resources.BinaryResources.digi);
request.HttpsAuthentCerts = new X509Certificate[] { new X509Certificate(cert) };

I’m still getting exceptions like few posts before:

    #### Exception System.Net.Sockets.SocketException - CLR_E_FAIL (1) ####
    #### Message: 
    #### Microsoft.SPOT.Net.Security.SslNative::SecureConnect [IP: 0000] ####
    #### Microsoft.SPOT.Net.Security.SslStream::Authenticate [IP: 0060] ####
    #### Microsoft.SPOT.Net.Security.SslStream::AuthenticateAsClient [IP: 000c] ####
    #### System.Net.HttpWebRequest::EstablishConnection [IP: 0247] ####
    #### System.Net.HttpWebRequest::SubmitRequest [IP: 0019] ####
    #### System.Net.HttpWebRequest::GetRequestStream [IP: 0008] ####
    #### Drzemlik_brama_G120E.Program::Main [IP: 00af] ####
    #### SocketException ErrorCode = 1
    #### SocketException ErrorCode = 1
A first chance exception of type 'System.Net.Sockets.SocketException' occurred in Microsoft.SPOT.Net.Security.dll
    #### SocketException ErrorCode = 1
    #### SocketException ErrorCode = 1
    #### SocketException ErrorCode = 1
    #### SocketException ErrorCode = 1
A first chance exception of type 'System.Net.Sockets.SocketException' occurred in System.Net.Security.dll
    #### SocketException ErrorCode = 1
    #### SocketException ErrorCode = 1
    #### SocketException ErrorCode = 1
    #### SocketException ErrorCode = 1
A first chance exception of type 'System.Net.Sockets.SocketException' occurred in System.Http.dll
    #### SocketException ErrorCode = 1
    #### SocketException ErrorCode = 1
Error:System.Net.Sockets.SocketException
The thread '<No Name>' (0x1) has exited with code 0 (0x0).

What is the proper way to add credentials to the HTTP request? Adding a header “Authorization” or by using Credential property?

I think that you didn’t add the certificate as a resource to your project. It is not enough to copy it into the Resources folder.
Doubbleclick on “Resources.resx” and select Add Resource.
Then you can use the certificate in a manner like this:


public static byte[] caGoogle = Resources.GetBytes(Resources.BinaryResources.GeoTrust_Global_CA);
public static X509Certificate[] caCerts = new X509Certificate[] { new X509Certificate(caGoogle)};

Hi,
I added it in way you described and the certificate could be read succesfully. Maybe you post your reply before I edited my previous question?

Edit: I added a picture with my resources.

Yes, I posted before I saw your reply.
Perhaps it helps to use Fiddler as a Proxy to see the differences between the requests of the PC application and the NETMF application.
In some of my CodeShare entries you can see how Fiddler can be used.

I hooked up Wireshark but I even don’t have any packets going to the Cloud from my FEZ Spider… I searched through the forum and found information about memory leaks. Do you think my problem is related?
I will try to use sockets like you did in [url]https://www.ghielectronics.com/community/codeshare/entry/1095[/url].
Thanks for your help, if you had any further advice I will be grateful.

I think Wireshark will not help in this case. It only captures packets on the Ethernet port of your PC. The NETMF device does not use this Ethernet port. With Fiddler you can use your PC as a proxy and the request from your NETMF device are redirected and go via the Ethernet port of your PC to the Internet. So you can see what happens. Getting familiar with Fiddler is worth the effort.
I think that your problem has nothing to do with memory leaks.
I could successfully use POST requests with NETMF devices when no content was sent back from the Server. When concent was sent back I had exceptions too. I tried with sockets and sslStream but I did not come to a final solution.

I think I don’t understand my problem. What type of certificate do I need in my application? A certificate which I downloaded from (in my case) DigiCloud site or a “root certificate”? What exactly a “root certificate” is?
I would also like to ask about the SSL Seed. I found this advice searching trough the Internet to use .NET Micro Framework Deployment Tool. Do I need to make SSL Seed after every program upload?

Edit: I definitely have problem with GetRequestStream() method, this is where my Exceptions are getting from:


request.ContentLength: 100
    #### Exception System.Net.Sockets.SocketException - CLR_E_FAIL (1) ####
    #### Message: 
    #### Microsoft.SPOT.Net.Security.SslNative::SecureConnect [IP: 0000] ####
    #### Microsoft.SPOT.Net.Security.SslStream::Authenticate [IP: 0060] ####
    #### Microsoft.SPOT.Net.Security.SslStream::AuthenticateAsClient [IP: 000c] ####
    #### System.Net.HttpWebRequest::EstablishConnection [IP: 0247] ####
    #### System.Net.HttpWebRequest::SubmitRequest [IP: 0019] ####
    #### System.Net.HttpWebRequest::GetRequestStream [IP: 0008] ####
    #### Drzemlik_brama_G120E.Program::Main [IP: 00af] ####
    #### SocketException ErrorCode = 1
A first chance exception of type 'System.Net.Sockets.SocketException' occurred in Microsoft.SPOT.Net.Security.dll

Is it even possible to make a POST request and get a reponse using HTTPS on .NET MF?

The root certificate is the root of the certificates chain.
For htttps://www.digi.com/products/cloud/digi-device-cloud
it should be the certificate: Starfield Class 2 Certification Authority
this root certificate must be included in your application.
The ssl seed must not be set every time you deploy an application.
It must be set new if you use new firmware and may be it can be destroyed when a program crashes.
Perhaps you can show the code you are actually using.

Are you sure? The exact url where I try connect to has certificate “Go Daddy Class 2 Certification Authority”, like on picture I include below.

Actual code (I tried three different ways of adding credentials):

try
{
	/* Data to be sent */
	byte[] data = UTF8Encoding.UTF8.GetBytes("{\r\n\"  \"stream_id\": \"G120M_test\",\r\n   \"stream_type\": \"STRING\",\r\n  \"value\": \"" + DateTime.Now.ToString() + "\"\r\n}\r\n");

	/* Request URL */
	Uri url = new Uri("https://devicecloud.digi.com/ws/v1/streams/history");
	HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);

	/* Request Method */
	request.Method = "POST";

	/* Request Credentials */
	String username = "";
	String password = "";
	String encoded = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(username + ":" + password));
	request.Headers.Add("Authorization", "Basic " + encoded);

	//request.Credentials = new NetworkCredential(username, password, AuthenticationType.Basic);
	//request.Credentials = new NetworkCredential(username, password);

	/* Request Certificate */
	byte[] cert = Resources.GetBytes(Resources.BinaryResources.godaddy);
	request.HttpsAuthentCerts = new X509Certificate[] { new X509Certificate(cert) };
	Debug.Print("Certificate Issuer: " + request.HttpsAuthentCerts[0].Issuer);
	Debug.Print("Certificate Exp. Date: " + request.HttpsAuthentCerts[0].GetExpirationDate().ToString());

	/* Request other settings */
	request.KeepAlive = false;
	request.ReadWriteTimeout = 1000;
	request.ContentLength = data.Length;
	request.ContentType = "application/json";
	request.Accept = "application/json";

	/* Prepare Stream - here the program hangs */
	Stream dataStream = request.GetRequestStream();

	/* Write data to Stream and close */
	dataStream.Write(data, 0, data.Length);
	dataStream.Close();

	/* Get response */
	WebResponse response = request.GetResponse();
	request.Dispose();

	Debug.Print(((HttpWebResponse)response).StatusDescription);
	Stream respData = response.GetResponseStream();
	StreamReader reader = new StreamReader(dataStream);

	string responseFromServer = reader.ReadToEnd();
	// Display the content.
	Debug.Print(responseFromServer);
	// Clean up the streams.
	reader.Close();
	dataStream.Close();
	response.Close();
	
	buffer = Encoding.UTF8.GetBytes("Wyslano");
	UART.Write(buffer, 0, buffer.Length);               
	
}
catch (Exception e)
{
	buffer = Encoding.UTF8.GetBytes("Error:" + e.Message.ToString());
	UART.Write(buffer, 0, buffer.Length);

	Debug.Print("Error:" + e.Message.ToString());
}

No, I’m not shure, probable you have the right one. I never worked with the Digi Cloud.

This is strange, I tried a GET example without a certificate and it worked well. Then I tried to figure out what is wrong with POST example and it was probably the way of inserting data (I changed it to one line, but it should be divided). Here I post working code:

/* Data to be sent */
String temp = "{\r\n";
temp += "  \"stream_id\": \"G120M_test\",\r\n";
temp += "  \"stream_type\": \"STRING\",\r\n";
temp += "  \"value\": \"09.04.2017 15:46\"\r\n";
temp += "}\r\n";
byte[] data = UTF8Encoding.UTF8.GetBytes(temp);

/* Request URL */
Uri url = new Uri("https://devicecloud.digi.com/ws/v1/streams/history");
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);

/* Request Method */
request.Method = "POST";

/* Request Credentials */
String username = "";
String password = "";
request.Credentials = new NetworkCredential(username, password, AuthenticationType.Basic);

/* Request other settings */
request.KeepAlive = false;
request.ContentLength = data.Length;
request.ContentType = "application/json";
request.Accept = "application/json";

/* Prepare Stream */
Stream dataStream;
dataStream = request.GetRequestStream();

/* Write data to Stream and close */
dataStream.Write(data, 0, data.Length);
dataStream.Close();

/* Get response */
WebResponse response = request.GetResponse();                
Debug.Print("Response status: " + ((HttpWebResponse)response).StatusDescription);
Stream respData = response.GetResponseStream();
StreamReader reader = new StreamReader(respData);

/* Read response */
char[] read = new char[256];
int count = reader.Read(read, 0, 256);
while (count > 0)
{
	// Dump the 256 characters on a string and display the string onto the console.
	String str = new String(read, 0, count);
	Debug.Print(str);
	count = reader.Read(read, 0, 256);
}

/* Clean up the streams */
reader.Close();
dataStream.Close();
response.Close();
request.Dispose();

RoSchmi - thank you very much for your help and responses!

The Post Request seems to work as well.


static void myButton_ButtonPressed(ButtonNETMF sender, ButtonNETMF.ButtonState state)
        {
            

            byte[] data = UTF8Encoding.UTF8.GetBytes("{\r\n\"  \"stream_id\": \"G120M_test\",\r\n   \"stream_type\": \"STRING\",\r\n  \"value\": \"" + DateTime.Now.ToString() + "\"\r\n}\r\n");
            /* Request URL */
            Uri url = new Uri("https://devicecloud.digi.com/ws/v1/streams/history");
            HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);

            /* Request Method */
            request.Method = "POST";

            /* Request Credentials */
            String username = "RoSchmi";
            String password = "Ro$597328543806535";
            String encoded = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(username + ":" + password));
            request.Headers.Add("Authorization", "Basic " + encoded);

            //request.Credentials = new NetworkCredential(username, password, AuthenticationType.Basic);
            //request.Credentials = new NetworkCredential(username, password);

            /* Request Certificate */
            byte[] cert = Resources.GetBytes(Resources.BinaryResources.GoDaddy);
            request.HttpsAuthentCerts = new X509Certificate[] { new X509Certificate(cert) };
            Debug.Print("Certificate Issuer: " + request.HttpsAuthentCerts[0].Issuer);
            Debug.Print("Certificate Exp. Date: " + request.HttpsAuthentCerts[0].GetExpirationDate().ToString());

            
            string responseBody = "";
            HttpStatusCode responseStatusCode = HttpStatusCode.Ambiguous;
            string _date = DateTime.Now.ToString();

            using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
            {
                if (response != null)
                {
                    Debug.Print("Response != null");
                    if (response.Headers.Count > 0)
                    {
                        try
                        {
                            _date = response.GetResponseHeader("Date");
                        }
                        catch { }
                    }
                    responseStatusCode = response.StatusCode;

                    using (Stream dataStream = response.GetResponseStream())
                    {
                        //dataStream.ReadTimeout = 10000;
                        using (StreamReader reader = new StreamReader(dataStream))
                        {
                            responseBody = reader.ReadToEnd();
                            reader.Close();
                        }
                    }
                    
                    //Report all incomming data to the debug
                    Debug.Print(responseBody);

                    if (response.StatusCode == HttpStatusCode.Forbidden)
                    {
                            Debug.Print("Problem with signature. Check next debug statement for stack");
                    }
                    response.Close();
                    if (responseBody == null)
                    {
                        Debug.Print("No body content");
                    }

                    Debug.Print("Response Status Code: " + response.StatusCode);
                }
                
                else
                {
                    Debug.Print("Response is null");
                }
            }
        }

Edit: Didn’t read that you got it working.

I don’t know why it wasn’t working earlier. I updated TinyBooter during development of my code, maybe that was real reason?

Added the code for writing to the request stream from your example to my code and got Status code 201 as well. :clap: :clap:

Edit: Yes, perhaps this was the reason.

Don’t forget to reset your board periodically when using https (because of the memory leak).

Can I do it using a function? Or it has to be done by external resetting?
My device should be running continuously for many months…

Edit: some kind of Watchdog?