Which Azure product is ideal for this?

I have a commercial project to capture weather sensor data and to transmit the data via GPRS or SMS. For SMS is easy but for GPRS I need a web based service that is easy to use and would be easy to scale up for more units.

I need it to be able to handle more than 1 unit, possibly up to 1000’s of them for some clients.

I’ve looked on Azure but not sure which would offer the solution I need. I want to be able to see graphed data as well as a table view with raw data. Also need the ability to download the data for external processing.

I’ve played around and can send data to a Mango Automation SCADA server but that has too limited graphing features and Azure looks like it would be the best way forward. I’d like to have a nice interface for this.

So, which Azure product would be ideal for this?

When using Azure this question comes down to at what frequency are you sending data up to it. Event Hubs can handle millions of events a second with AMQP, but it has a bit of a cost associated to it, whereas Mobile Services are cheaper but given it uses http(s) its not well suited to rapid connections. For example we were running a sleep/noise experiment here and data was sent x times a second sort of thing so it used Event Hubs, whereas another experiment was monitoring temperatures in a BioMass reaction and was a reading every 10 minutes, so it used mobile services.

Now nothings says you can’t switch as customer demand changes either.

To show the data PowerBI is killer good no matter what you use to capture it.

@ Duke Nukem - This will be sending every 60 seconds. The data doesn’t change much and they are not interested so much in wind speed which changes more often but they only want to gust information.

I am on Azure now and looking at the IoT hub and it looks like it might be suitable. Now all I need to do is find the API information on how to send the data to it.

I wouldn’t rule out just an Azure Database (SQL) and a simple Web API or API Services. I still think SQL is the easiest of all to work with when reporting is the end goal but I’ll admit that my hands-on experience with the new IoT services is nill. The price is a non-starter for my needs. There is a long line of tools you can generate your reports/exports with when SQL is your backend.

I agree with Duke that PowerBI is an excellent tool for web dashboards. However, the biggest problem with PowerBI is that it has zero export capabilities currently. The only way to get data/graphs from the screen is via screen capture. I still endorse PowerBI but keep in mind you’ll have to find something else for exports.

Most sensor data isn’t relational, and SQL server has a steep cost profile, low scale limits and high storage cost per row that make it less attractive to me for recording data. Azure has deprecated support for SSRS, so using that for reporting requires spinning up a virtual server and most of the other reporting options work with all the other storage platforms anyway.

DocumentDB is more interesting that storage tables for searching, but it isn’t built for rapid repetitive access (it throttles fairly quickly - and you pay for higher throttle ceilings) and only scales to 10Gb per collection (with additional costs calculated per collection). To me, it’s a great solution for storing lots of back-end service data, but not for sensor data.

ServiceBus is way too expensive (something like 10x the cost of EventHubs as you scale up). See: [url]azureservicebus - Looking for clarity on Event Hubs vs Topics in Azure Service Bus - Stack Overflow and EventHubs are one-way. I like having two-way communication so that I can trigger updates and config changes. And neither servicebus nor eventhubs are storage mechanisms. You still need to choose a back end storage mechanism, and you may need to create a front-end service if your endpoints can’t handle the complex servicebus interactions or do SSL.

The IoT preview gives you the scale of EventHubs (or better) plus two-way connectivity, friendlier front-end protocols, and free open-source apis for netmf, .net, mono and more, plus a bunch of goodies for device management. Incoming data scaleability is excellent (best among all the options) and the cost profile seems good, but the big caveat is that it is still in preview, so predicting the final pricing is anybody’s guess. And the vary nature of ‘preview’ means that going to production with it is a calculated gamble. They can make breaking changes that will send you scrambling.

DataLake is a complete unknown to me - it is touted as a hadoop-enabled big-data solution with both relational and tabular storage, but I have no experience with the costs or performance.

Don’t overlook the option to create your own TCP protocol and then writing directly to Azure storage tables. You have more control over the cost profile that way, can make a very lean protocol fit to your needs, and there are several options for silently scaling that up or down as needed with a very shallow cost profile.

I use and like Azure Table Storage. I have an Azure Website (WebAPI) that receives my data and then pushes it into Table Storage. At 7 cents per GB (US), it is very inexpensive.

1 Like

I agree with @ ianlee74 and @ terrence. For simple needs, an ASP.NET Web API endpoint pointing to either Azure SQL, Table Storage, DocumentDB, or whatever you like (heck, you could use MySQL, too, as there’s a free tier available for ClearDB…just don’t count on high throughput on that one), and you can do everything you need for simple IoT data collection with little or no cost.

Really the only complaint I have about Azure with respect to IoT is that there are far too many options to choose from. :slight_smile:

2 Likes

Ditto that!

I’d second that too. My first time trying to set this up and it’s a nightmare with all of the options. Some of the videos on here and elsewhere for using Azure are no longer fully valid as the interface has changed and all the options are different.

I am still playing with the Data+Storage Table option but not 100% sure how to set this up. Still delving into the documentation and looking for tutorials etc.

I have used Azure extensively in a number of projects and used most of the ‘offerings’ within the service. It was confusing a few years ago and they have rolled out more services recently which somewhat cloud the ‘best practices’. It is a great service overall though, I am a real devotee.

For cheap and easy storage without the need for any server side operations or aggregation, I would go for Azure Table Storage. There is a simple REST based interface for this and the basics (inserts, updates etc) are very easy to implement. Most of the published articles and blog posts refer to the Azure SDK which is obviously not for NetMF.

You can use AtomPub or Json as the payload for the entities but I think Json would be easiest on this platform. Be very careful with my point about server side operations as you really cannot do anything. A simple count of the rows of data is very difficult for instance.

There are a few tricks around storing data for effective queries and I have learnt a lot about how to deal with DateTIme / Time series data effectively.

If you want to proceed that way and need any guidance, feel free to ping me and I will do what I can to help

@ Dave McLaughlin - Did you ever decide on a Azure product?

I am currently using Azure for a product. I am primary using a Web server and SQL Server as well as some cloud services for some data processing. The Web server handles the user interface and my data comes in from my devices using a POST to the web server using JSON. It currently works okay for now but it doesn’t seem too efficient, primarily when going over cellular networks. JSON adds so much overhead when transferring data. I have experimented with Service Bus in the past and still might go back to it. Before I do I am considering just using a cloud service and roll my own TCP binary protocol.

Azure IoT with Event Hubs uses AMQP which I understand requires TLS/SSL which keeps you on more powerful devices. One of my biggest road blocks is finding secure and encrypted solutions on small devices.

I am still playing around with it. I have setup the IoT HUB and will try out the tables if I can figure out how it all works.

I am used to SCADA systems where I can choose something like Modbus or DNP3 to send data to the server. Nice simple and light payloads. JSON seems a bit too heavy on data and as I will be using GSM I’d like something with a lighter payload. Like you I may go the way of using a simpler home grown TCP option and just send binary encoded data. Fast and efficient.

I am also tinkering with Mango Automation too which allows a simple HTTP data connection as well as Modbus and other SCADA protocols. It’s lack of a graphical front end is the limiting fact and you need a server to run it on. I am testing with my own server for now and trying to work out how to use the API for creating my own website from it.

@ Dave McLaughlin - if using AMQP, the body of the message can be any format you like and doesn’t have to be json. Although as json is a good choice for downstream systems you could encode with messagepack or bson.