An http(s) error - stumped by a lack of diagnostic info

I want to air this among friends, so I am making these comments in Insiders…

There are a thousand things that can go wrong in the network stack, but I’m having extra trouble today because of a lack of diagnostic information. Consider the stack trace below. This fails whether or not I offer a cert (which, if I read the library code correctly, it should still work without one because without a cert, validation becomes optional, but the channel is still secured). In any case, using the exported root CA cert doesn’t help matters.

This is just one example. In another case, the stack traces for a dead server, blocked port, and an http url to an https nginx redirect all reported the same stack traces. It was only through use of a managed switch, wireshark, and server-side logs that I was able to debug it.

Notice also in the stack trace that ‘message’ is always empty. I had a misspelled domain name and the only way I figured that out was that a Dns call was the top thing in the stack trace. A message string could have told me that more clearly.

I will build a debug copy of the library, and put the pdbs where VS can find them, and I’ll use a managed switch and wireshark and probably get this figured out soon enough (though hints warmly welcomed). But those are steps the average user probably is not going to be able to do. My core message here is that the network stack is pretty inscrutable when things don’t go perfectly.

I see the thread where the fellow was having trouble with AWS and MQTT and his problem was a poorly formed/unresolvable dns name. If the exceptions had “host not found” or something similar, he might have worked it out on his own.

So, I need help with this stack, but I also think that in spite of the cost to program size, the net stack exceptions need some reason text in the “message” field.

 #### Exception System.InvalidOperationException - CLR_E_INVALID_OPERATION (4) ####
    #### Message: 
    #### GHIElectronics.TinyCLR.Devices.Network.Provider.NetworkControllerApiWrapper::AuthenticateAsClient [IP: 0000] ####
    #### System.Net.Security.SslStream::AuthenticateAsClient [IP: 0016] ####
    #### System.Net.Security.SslStream::AuthenticateAsClient [IP: 0009] ####
    #### System.Net.Security.SslStream::AuthenticateAsClient [IP: 0008] ####
    #### System.Net.Security.SslStream::AuthenticateAsClient [IP: 0006] ####
    #### System.Net.HttpWebRequest::SubmitRequest [IP: 002b] ####
    #### System.Net.HttpWebRequest::GetResponse [IP: 0014] ####
    #### PervasiveDigital.Galaxy.UX.Agents.SyncAgent::Get [IP: 0035] ####
    #### PervasiveDigital.Galaxy.UX.Agents.SyncAgent::GetDirectoryListing [IP: 0006] ####
    #### PervasiveDigital.Galaxy.UX.Agents.SyncAgent::Sync [IP: 004e] ####
    #### PervasiveDigital.Galaxy.UX.Agents.SyncAgent::Process [IP: 0036] ####
    #### PervasiveDigital.Galaxy.Core.NodeEngine::Run_Internal [IP: 00b9] ####
Exception thrown: 'System.InvalidOperationException' in GHIElectronics.TinyCLR.Devices.Network.dll
    #### Exception System.Net.WebException - 0x00000000 (4) ####
    #### Message: 
    #### System.Net.HttpWebRequest::GetResponse [IP: 0113] ####
    #### PervasiveDigital.Galaxy.UX.Agents.SyncAgent::Get [IP: 0035] ####
    #### PervasiveDigital.Galaxy.UX.Agents.SyncAgent::GetDirectoryListing [IP: 0006] ####
    #### PervasiveDigital.Galaxy.UX.Agents.SyncAgent::Sync [IP: 004e] ####
    #### PervasiveDigital.Galaxy.UX.Agents.SyncAgent::Process [IP: 0036] ####
    #### PervasiveDigital.Galaxy.Core.NodeEngine::Run_Internal [IP: 00b9] ####
Exception thrown: 'System.Net.WebException' in GHIElectronics.TinyCLR.Networking.Http.dll
1 Like

More info:

This call into native code at SslStream.cs line 37:

this.sslHandle = this.ni.AuthenticateAsClient(this._socket.m_Handle, targetHost, caCertificate, clientCertificate, sslProtocols, sslVerification);

Is being made with a valid handle,
a targetHost of “mcalsyn01.pervasive.digital”
a caCertificate of ‘null’
a clientCertificate of ‘null’
sslProtocols == SslProtocols.None
sslVerification == SslVerification.Optional

and the server is offering an SNI wildcard cert of *.pervasive.digital

Does the GHI TLS stack support SNI wildcard certs? (I suspect so, since they are heavily used by Azure and AWS)

I also tried setting HttpsAuthentCerts to both the Root CA binary DER file, and the *.pervasive.digital public-key DER file. That gave a failure of client authentication, which is also weird, because I didn’t offer a client cert.

Any help/tips would be greatly appreciated.

Can you narrow it down to a small program we can try to test the certificate?

Hmmmm. So I whipped up a simple test program that hits a public *.pervasive.digital web site, and that works fine with or without the cert. It’s only failing against my internal nginx-based server, which should be offering up the same cert. I can give you the test program, but it looks like without the nginx setup, that’s not going to do any good, so I will fiddle around with it a bit more here first.

1 Like

TL/DR: Your server must provide the public key certs for all of the certs in the trust chain.

Got this figured out. The problem is when TinyCLR (and probably other TLS stacks) inter-operate with sites that don’t public-key certs for the full trust chain.

I have an nginx config that routes some requests to my Edge server code; some to keycloak; some to RabbitMQ, and some to static files.

But the crux of the problem is that in order for TinyCLR to work, your certificate bundle has to include the entire chain of certificates. Browsers won’t complain, because they either already have or can download the missing parts of the trust chain, but TinyCLR does not.

So the bundle.crt file must include your site’s private key (server.key below) and the certificate bundle has to include your site cert, plus all the certs in your trust chain.

What certs are those? Well, if your cert provided didn’t give them to you already in text form, the easiest way to find that is to click on the lock in Chrome and click on Certificate, then click on “Certificate Path” and then click on the item above your site cert:

Then click on View Certificate and go to the details tab and use Copy To File to save that cert to a file in base64 X.509 format:.

Repeat that process for the each item up the chain in the certifification path, concatenate those files in any order along with the X.509 for your site certificate and put those in your bundle.crt.

And fwiw, here is the relevant part of my nginx config:

server {
        listen          80;
        listen          [::]:80;
        server_name     devbox.pervasive.digital localhost;
        return 301 https://devbox.pervasive.digital$request_uri;
    }

    server {
        listen      443 ssl;
        listen      [::]:443 ssl;
        server_name  devbox.pervasive.digital localhost;
        ssl_certificate /bin/nginx-1.17.6/ssl/default/ssl-bundle.crt;
        ssl_certificate_key /bin/nginx-1.17.6/ssl/default/server.key;
        ssl_prefer_server_ciphers on;
    ...the rest of my nginx gorp...
1 Like

Bottom line: At very least, every throw of WebException needs to include something better than string.Empty as a message

Better: Same for all of the networking exceptions

Still better: More diagnostic hints from the native code TLS stack, because errors there are completely opaque.

1 Like

Did you not sleep all night or up very early?

No sleep when I end the day with broken code :rofl:

Actually, I am usually at it by 5 or 6am, but for some reason I woke up even earlier today. Now that I know my code isn’t broken, there’s a nap in my future.

2 Likes

Here it is https://github.com/ghi-electronics/TinyCLR-Libraries/issues/508

1 Like