I am using MQTT with Azure IoT over a cellular modem using PPP. Overall it is working pretty well. Occasionally I get errors and MQTT dies when I try to check the signal strength of the cellular connection periodically.
To check the signal strength I do the following:
networkController.Suspend();
//send +++ to modem to enter command mode
//wait for 'OK' from modem
//get signal strength and connection status
// issue "ATO\r\n" to re-enable PPP
//Wait for CONNECTED from the modem
networkController.Resume()
During that time I suspend sending any new traffic over MQTT or network.
However, I still think the MQTT library is attempting to send traffic while the modem is in command mode causing socket errors.
Before I dig into the MQTT library to come up with some way to suspend traffic while doing the signal check does anyone have any other suggestions or ways to solve this?
This is one of the reasons I would like to see multiple serial UART support with the USB host port so that we can have a data connection at the same as being able to check for signal, sms etc.
@skeller - While I have no help on your direct issue with PPP, MQTT, such is of deep interest as to:
âHow does PPP workâ?
We use direct AT modem commands to direct the modem to do this-n-that.
Multitudes of AT commands and connecting logic are what we call a âModem Driverâ
Of course, this âModem Driverâ does not provide an âethernetâ type of context, provided by PPP.
Belaying the details of a PPP/Ethernet contextâŚ
What is the âCore Logicâ within the modem that handles all the " Whatâs, Whenâs, Howâs " logic to allow âAlways Available Connectivityâ.?
By activating PPP, does a âModem Internalâ driver handle all facets of maintaining an âAlways Available Connectionâ.?
Was there a âSuper Guruâ engineer (at say Telit) that was able to take into account all geographies and all dynamics of placing the modem âAnywhereâ.?
âŚNot a bang against PPP, just donât know what is at the âInternal Logical (driver) Heartâ.?
After writing, I realize âEthernetâ is not the proper description of what I meant.!
The meaning was:
Setup a System.Net.Sockets.Socket as a listener socket.
Bind the Listener socket to the PPP interface.
Request a connection, get a connection and transact data.
@mhardy, PPP is an old protocol that goes back to the dial-up days that allows the modem to be put in somewhat a âtransparentâ mode so that data bytes can move freely over the connection. It is the âData link layerâ, so it is one layer above the actual physical connection. The TCP/IP protocol still runs on top of the PPP connection and in this case is handled by TinyCLR. The engineering behind it, on the cellular side, is pretty complicated and fortunately, the âSuper Guruâ engineers standardized it all and have made it work fairly seamlessly. The modem does request a data connection from the cell provider based on the security information provided on the SIM and the APN settings. So there is always a chance that the modem canât establish a connection if there is no service or the SIM card is bad, among other things. It is also possible that the connection drops and would need to be re-established, just like if someone unplugged an ethernet cable.
As @Dave_McLaughlin pointed out using USB versus a standard UART interface allows for the status of the modem and connection to be polled over a secondary data connection without breaking into the PPP connection.
I think my solution will be to disconnect the MQTT connection, do the signal check, and reconnect with the Clean Session = false. That should allow all the original subscriptions to carry over and a âLast Will and Testamentâ message isnât generated. It is not what I wanted but with time be an issue on this project I think it is the safest.
@skeller
Understood on the roots of PPP.
Yes - âThe engineering behind it, on the cellular side, is pretty complicatedâ
As to âfairlyâ, as stated - âand fortunately, the âSuper Guruâ engineers standardized it all and have made it work fairly seamlessly.â
Given that (good) SIM (well inserted), APN settings are all correct, (good-to fair RSSI, service available), all good-to-go, i.e. initial and on-going connectivity has been established for say months or yearsâŚIn other words, only external (of no control) connections get dropped âŚâand need to be re-established, just like if someone unplugged an ethernet cable.â
In your experience, have you found âfairlyâ equates to approximately 1min (max) of âNo Connectivityâ, before the PPP/Modem driver can re-established a connectionâŚ?
I have not used PPP, on the Telit chipset, which is at the âBelly-of-the-beastâ of our modem.
1min (max) of âNo Connectivityâ is what our AT based modem driver roughly delivers.
@Dave_McLaughlin
Yep, not having to break from data mode to command mode back to data mode -back-n-forth, would be a great feature.!
Such dialog is greatâŚ!
Hard to find folks that have walked many-a-mile down such a path(s)âŚ
@mhardy, not sure I completely understand your question but I will try to answer.
The initial PPP connection establishes within 5 or 10 seconds of the connection request. The errors I was referring to in the original post only happened occasionally. Every 60 seconds I would break into the PPP connection with â+++â to enter command mode so I could check the signal strength. (sometimes it would take 5 or 10 seconds and/or additional â+++â to enter command mode.) Then re-init the PPP connection. For the most part, everything just picked up where it left off with no problems. But on occasions, while in Command Mode MQTT would crash with various socket and SSL errors. Even with those errors, returning from my signal check, the PPP connection was still active, it was just MQTT that died.
For now, I have moved the signal check out to every 5 minutes and I close the MQTT connection. It will increase my data usage slightly since it has to re-establish the TLS socket and the MQTT connection but it is necessary.
I have not tested yet how to recover from a dropped signal or bad signal. I donât have an easy way to test that since disconnecting the antenna could damage the modem. I am connected to an external antenna because I am inside a metal building. Any small antenna that I could shield to simulate a low signal canât even establish the initial connection. I will have to resolve that soon since that is a real-world situation. I will probably move development to my house to do that part.
As said, we donât use PPP, MQTT or Azure IoT.
âŚNot that any of these paths are incorrectâŚthey just did not exist back-in-the-day.
If I could offer of little advice:
Whatever you are building needs to be developed / debugged / tested âŚin âallâ environment(s) where final product will be deployed. If such is not done, endless-loop of (no-idea-what-the-heck) support, etc.
Never let the amount of data consumed limit reliability.
Customers just want stuff that works and will pay if it doesâŚ
Yes, MQTT is separated and it doesnât know what PPP is, suspend, resumeâŚ
MQTT has a property âKeepAliveTimeoutâ (time unit is second), can you please try to increase that one and make sure your suspend period is smaller that that value
if still, as usually, give us a small, simple project to see if we can help.
@Dat_Tran, Thank for jumping in. The PPP connection is in command mode for maybe 5 seconds, it all depends on how long it takes to respond to the â+++â. Sometimes it takes 10+ seconds. (After a 10 sec delay, if no response to the first â+++â I send a second â+++â and that gets its attention.)
My MQTT Keepalive is at 300 and I am sending a message about every 30seconds leading up to the break in the PPP and the signal check so I donât think it is trying to do a keepalive ping. I think it might be in the middle of a message acknowledge when I break the connection. I could wait a few seconds after the last publish before doing a signal check but it would be a complete guess as to how long to wait since I donât know how long it will take for the broker to respond to the publish. Depending on the cellular network it could be pretty variable.
I will go with the safe route for now and just close the connection. For future software revâs I can try to get more creative.
When you say; âI am sending a message about every 30secondsâ
What is the source? âŚAzure IoT or your device.?
Device to Azure IoT should not be a problem, i.e. the modem will increase dBm to connect to the tower.
Azure IoT to device is the tough problem, i.e. the modem sitâs at default dBm (power)âŚlistening.
Are you running over LTE.?
If you are, which version, carrier, etc.?
Every 30sec is extreme for keeping the connection âaliveâ.
With understanding you are in dev mode âŚall fine, but when rolling to production, messages every 30sec may kill you on data plan limits, overages, etc.?
A âtrueâ (udp) ping to the device costs very little in data.
A message, other than âpingââŚassume MQTT at basic is TCP, will cost much more in data.
Just IMO⌠trying to coalesce all forum members empirical discovery(s) as to cell connectivity.!
Cell documentation will say a session remains alive on some timer somewhere for (seem to remember) 120min.
NEVER believe suchâŚ!
Such metric was derived from âabsolutely perfect anechoic chamberâ testing.
One will never find the same conditions in the ârealâ world.
Paraphrasing the original question as âWhy does MQTT still send, even when I suspend my calls to it?â, Dat correctly pointed out the keep-alive interval. The reason is that within that keep-alive interval, the client (and maybe the server) are sending âpingsâ back and forth even if you donât send any packets or have any subscription traffic. Even if you stop making calls, pings will still happen and subscriptions will still result in inbound traffic.
You can slow the client->server pings with the keep-alive interval, but that only makes an inconvenient ping less probably, and doesnât stop any inbound messages (unless you unsubscribe).
Multi-channel serial would solve the issue, but in the meantime, youâre just taking a chance that a ping interval or inbound message wonât occur while you have suspended the network. Even if you create a long timeout for keep-alive, how can you be certain that it wonât still occur during that outage window? Closing the MQTT connection is the only way to be certain.
@mcalsyn, you are 100% correct. Increasing delays and keepalives only reduces the chances of the problem, it doesnât eliminate it. Short of re-writing the MQTT library to âpauseâ all communications while there is a break in the PPP connection, disconnecting is the best option.
I would rather wait for the multi-channel USB connection than re-write MQTT libraries.
I also support @mcalsyn on the MQTT issue.!
The same is very true with cell, best to totally disconnect a session and restart.
Use brute force, just do such every so often regardless as to what code may try and understand of âBeing Connectedâ, especially with server to device (device always command able) communications.
