IoT Hub and Azure Time Series Insights

Azure Time Series Insights is a new service that makes it very easy to store and visualize time series data. In this blog post, we will create a dashboard that looks like the one below (click to enlarge):

image

The dashboard has four sections:

  • Query1: a heat map of events per device; in this case there are 20 devices sending data every 2 seconds
  • Query2: a line graph with random “temperature” data
  • Query3: a line graph with both “temperature” and “humidity” data
  • Query4: a line graph with “humidity” data

The events are sent to an IoT Hub using the following JSON shape: {temperature: x, humidity: y} where x and y are randomized floating point numbers, generated by an IoT device simulator.

Step 1: Create IoT Hub

Install Azure CLI 2.0, and then use az login to login. Use az account list to list your subscriptions and use az account set –subscription name_or_id to set the default subscription. Next, issue the following commands to create a resource group and an IoT Hub (set location to your preference):

az group create --name resource_group_name --location westeurope
az iot hub create --sku F1 --name iot_hub_name --resource-group resource_group_name

As a best practice, create a separate consumer group on the Events endpoint. In the Azure Portal, in the properties of the IoT Hub, click Endpoints. Then click Events and add a consumer group underneath $Default. Click Save.

Record the Connection String – primary key setting of the device or  iothubowner Shared access policy. Click Shared Access Policies, and device to find this connection string. It will be in the form of:

HostName=iot_hub_name.azure-devices.net;SharedAccessKeyName=keyname;SharedAccessKey=b5dARuGPhL6wdgHboUIhEC6LlcFalIjfEdh4aXYa1WI=

You will need this connection string later to configure the IoT Simulator.

Step 2: Create Time Series Insights Environment

In the Azure Portal, click the green + and navigate to Internet of Things. Click Time Series Insights and follow the on-screen instructions. You will end up with:

image

I selected one unit of the S1 tier which is more than enough for this example.

Step 3: Set Data Access Policy

Even though you created the Time Series Insights Environment, you still need to grant yourself access to the data. Click Data Access Policies and add your user or group and a role of Contributor.

image

Step 4: Add Event Source

We will add the IoT Hub we created earlier as an event source. Click Event Sources and then click Add. Give the event source a name and set the source to IoT Hub. Then select an IoT Hub from your available subscriptions and do not forget to set the consumer group to the one you created in step 1. If your event data has a timestamp, you can enter the timestamp property name. If you do not specify the timestamp, the event enqueue time set by the IoT Hub will be used.

Note that Azure Time Series Insights also supports Event Hubs as an event source.

Step 5: Configure the IoT simulator

Head over to https://github.com/gbaeke/iot-simulator/releases/tag/v0.3 and download iot-simulator.exe to a folder of your choice. In the same folder add a file called config.json with the following contents:

{
     "Interval":5,
     "IoTHubs":["iot_hub_name.azure-devices.net”],
     "SasTokens":["SharedAccessSignature sr=..."],
     "DevGroups":[
        {"Prefix":"ts","DeviceNum":20,"Firmware":"1.0","IoTHub": 0}
     ]
}

In the SasTokens array, replace SharedAccessSignature sr=… with a Sas token that has the necessary rights to submit events to the IoT Hub. One way of doing so, is with Device Explorer. Once installed, copy the connection string from step 1 in the connection string box and click Generate SAS. Copy the Sas token in the config.json file.

image

With the config.json correctly configured, from a command prompt, start iot-simulator.exe. It will connect to the IoT Hub, create the devices and start sending data every 5 seconds from every device. In the sample config file, you can set the interval in seconds (Interval) and the amount of devices (DeviceNum). To clean up the devices, run iot-simulator.exe –r.

Step 6: Visualize the data

Now go to https://insights.timeseries.azure.com and login with the credentials you used in step 3. You will get a screen to select data. I selected Last 60 Mins from the quick times dropdown and then clicked the search icon:

image

In the following screen, click Heatmap and then configure the box at the left with a descriptive title. Also select a split by deviceid to have an idea about the number of events per time window per device and to spot devices that stopped sending data.

image

Now, at the right top corner, click the circle with the four squares. You end up with:

image

Now click the + in the top, right section. Select a time range again and then, at the left, change the measure from Events to Temperature. Automatically, the temperature will be averaged over the interval size. Change the term (Term 1) to Temperature and click the circle with the four squares again.

The temperature line graph has been added and you can now click the copy icon and create the same visualization for humidity.

image

Now it’s easy to create the other panel with both temperature and humidity. Give it a go and try out other visualizations. When you are finished, you can click the Save icon and save this perspective. Yep, these visualizations are called perspectives!

It’s still early days for the service and many features will be added in the near future. If you are already working with event data coming into an Event Hub and IoT Hub, it should be easy to add a new consumer group and start analyzing the data with this service.

Getting started with Kubernetes on Azure

As you may or may not know, at Xylos we have developed an IoT platform to support sensor networks of any kind. The back-end components are microservices running as containers on Rancher, a powerful and easy to use container orchestration tool. In the meantime, we are constantly evaluating other ways of orchestrating containers and naturally, Azure Container Services is one of the options. Recently, Microsoft added support for Kubernetes so we decided to check that out.

Instead of the default “look, here’s how you deploy an nginx container”, we will walk through an example of an extremely simple microservices application written in Go with the help of go-micro, a microservices toolkit. Now, I have to warn you that I am quite the newbie when it comes to Go and go-micro. If you have remarks about the code, just let me know. This post will not explain the Go services however, so let’s focus on deploying a Kubernetes cluster first and deploying the finished containers. Subsequent posts will talk about the services in more detail.

With the help of Azure CLI 2.0, deploying Kubernetes could not be simpler. You will find full details about installation on https://docs.microsoft.com/en-us/cli/azure/install-azure-cli. The CLI runs on Windows, Linux and macOS. For this post, I used macOS. If you are a bit unsure about how the Azure CLI works, check out this post: https://docs.microsoft.com/en-us/cli/azure/get-started-with-azure-cli.

After installation, use az login to authenticate and az account to set the default subscription. After that you are all set to deploy Kubernetes. First, create a resource group for the cluster:

az group create --name=rgname --location=westeurope

After the above command (use any name as resource group), use the following command to create a Kubernetes cluster with only one master and two agents and use a small virtual machine size. We do this to keep costs down while testing.

az acs create --orchestrator-type=kubernetes --resource-group=rgname --name=clustername --generate-ssh-keys --agent-count=2 --master-count=1 --agent-vm-size=Standard_A1_v2

Tip: to know the other virtual machine sizes in a region (like westeurope) use az vm list-sizes --location=westeurope

Note that in the az acs command, we auto-generate SSH keys. These are used to interact with the cluster and you can of course create your own. When you use generate-ssh-keys, you will find them in your home folder in the .ssh folder (id_rsa and id_rsa.pub files).

Now you need a way to administer the Kubernetes cluster. You do that with the kubectl command-line tool. Get kubectl with the following command:

az acs kubernetes install-cli

The kubectl tool needs a configuration file that instructs the tool where to connect and the credentials to use. Just use the following command to get this configured:

az acs kubernetes get-credentials --resource-group=rgname --name=clustername

Running the above command creates a config file in the .kube folder of your home folder. In the config file, you will see a https location that kubectl connects to, in addition to user information such as a user name and certificates.

Now, as a test, lets deploy a part of the microservices application that exposes a REST API endpoint to the outside world (I call it the data API). To do so, do the following:

kubectl create -f https://raw.githubusercontent.com/gbaeke/go-data/master/go-data-dep.yaml

The above command creates a deployment from a configuration file that makes sure that there are two containers running that use the image gbaeke/go-data. Each container runs in its own pod. You can check this like so:

kubectl get pods

You will see something like:

image-2

Run kubectl get deployment to see the deployment. Use kubectl describe deployment dataapito obtain more details about the deployment.

You will not be able to access this API from the outside world. To do this, let’s create a service of type LoadBalancer which will also configure an Azure load balancer automatically (could have been done from the YAML file as well):

kubectl expose deployments dataapi --port=8080 --type=LoadBalancer

You can check the service with kubectl get service. After a while and by running the last command again, the external IP will appear. You should now be able to hit the service with curl like so:

curl http://IP_of_service:8080/data/device1

No matter what device id you type at the end, you will always get Device active: false because the device API has not been deployed yet. How the data API talks to the device API and how they use service registration in Kubernetes will be discussed in another post.

Tip: for those that cannot wait, just run kubectl create -f https://raw.githubusercontent.com/gbaeke/go-device/master/go-device-dep.yaml and then use curl again with device1 at the end (should return true). The above command deploys the device API so that the data API can find and use it to check if a device exists.

Controlling Sonos from a Particle Photon using a Sonos API on a Pi 3

In the previous article, Control Sonos with a easy to use API, we configured a Docker container on a Raspberry Pi 3 to run an easy to use Sonos API. I prefer this solution over writing code on the Photon to control Sonos. Now it is time to let the Photon talk to the API on the PI 3 to load a playlist and start playing or to stop playing at the press of a button.

Just create a new app with the Particle Build IDE and call the app SonosCtrl. Then add the following library: HttpClient. After adding the library, make sure you have the following includes:

image

To actually use HttpClient to make requests to the Sonos API, you will need some variables of specific types:

image

You will use the request variable to configure the request. When you configure request, you will need to specify a hostname or an IP address. I used the IP address of my RPi 3 (SonosController above).

To configure request:

image

The above just sets the port and IP address for the request. We do this in the setup() function. When we press a button, we toggle between playing from a playlist or pausing the Sonos:

image

By setting the request path appropriately, we can easily load a Sonos playlist or pause. See the GitHub page at https://github.com/jishi/node-sonos-http-api for more paths to use. There is much more you can do! Above, we target a specific Sonos Player (Living Room). As you can see, this is very simple to do and keeps the Particle Photon code cleaner. The code is kept pretty simple so no error handling, logging etc… You can find the full code in the following Gist: https://gist.github.com/gbaeke/9c185e82e7f23c0c4c9d803990d3660f. Have fun!!!

Control Sonos with an easy to use API

In an earlier post, Controlling Sonos from a Particle Photon, we created a small app to do just that. The app itself contained some C++ code to interact with a Sonos player on your network. Although the code works, it does not provide you with full control over your Sonos player and it’s tedious to work with.

Wouldn’t it be great if you had an API at your disposal that is both easy to use and powerful? And even better, has Sonos discovery built-in so that there is no need to target Sonos players by their IP? Well, look no further as something like that exists: https://github.com/jishi/node-sonos-http-api. The Sonos HTTP API is written in Node.js which makes it easy to run anywhere!

And I do mean ANYWHERE!!! I wanted to run the API as a Docker container on my Raspberry Pi 3, which is very easy to do. Here are the basic steps I took to configure the Raspberry Pi:

With Docker up and running, I created a Dockerfile and built the image. Here is the Dockerfile:

FROM hypriot/rpi-node
RUN git clone -q https://github.com/jishi/node-sonos-http-api.git
WORKDIR node-sonos-http-api
RUN npm install > /dev/null
EXPOSE 5005
CMD [“npm”,”start”]

Note: a Raspberry Pi uses an ARM architecture which means you need to use ARM compatible images; above I used hypriot/rpi-node (see https://hub.docker.com/r/hypriot/rpi-node/)

Note 2: I’m sure there already is a Docker image for this Sonos API; I just decided to build it myself

After building the image, I tagged it sonosctrl (using docker tag). You will see the tag of this image coming back later when we run the container.

Because the API server needs to discover the Sonos devices on the network, you should not use the Docker bridge network. The command to run the container from the sonosctrl image:

docker run –net=host –restart=always -d –name SonosController sonosctrl

Now you should have a container called SonosController up and running that accepts API requests to control your Sonos:

image

Note: you also see Portainer running above; I use that to get an easy GUI for Docker on this Pi

To actually test the API, use Postman or cURL. From Postman:

image

Above, you see a request to load the Sonos playlist called “car” on players in “Living Room”. The request was successful as can be seen in the response. This command will also start playing songs from the playlist right away. If you want to pause playing:

image

Great! We have a Sonos API running on a Raspberry Pi as a Docker container with a few simple steps. We can now more easily send commands to Sonos from devices like the Particle Photon or an Arduino. I will show you how to do that from a Particle Photon using the HttpClient library in a later article.

Temboo, Twilio and Nexmo: SMS and voice messages from your IoT device

In this post, I will provide an overview of how to use Twilio and Nexmo to send SMSs and voice messages directly from your device. I will use a Particle Photon but this also works from an Arduino, or a Raspberry Pi or basically any other system. The reason for this is that I will also use Temboo, an easy to use service that basically provides a uniform way to call a wide variety of APIs and even helps you with a code builder.

I will use the same basic sketch form earlier examples. This means there is a photoresistor which measures the amount of light but also a button that will trigger the calls to Temboo to send an SMS and a voice message with the current sensor value from the photoresistor.

Let’s get started shall we? You will first need accounts for all three services so go ahead and sign up. They all have free accounts to get started but remember they are all paying services. It’s up to you to decide how useful you find these services.

For Temboo, you will need to provide the account name, app key name and app key. Sadly, in the free Temboo tier, this key is only valid for a month and you will need to manually change it. I added these values as #defines in a header file called TembooAccount.h. Be sure to use #include “TembooAccount.h” in you .ino file. The contents of the TembooAccount.h:

image

In your .ino file, we’ll create two functions:

  • void runSendSMS(String body)
  • void runSendVoice(String body)

When you want to send an SMS or send a voice message, you call the appropriate function with the message you want to send or the text you want translated to speech.

The contents of the function is easy to write because you don’t have to. Temboo provides a code generator for you. When you are logged in, just go to https://temboo.com/library/ and select the Choreo you want to use. For the SMS, you select Twilio / SMSMessages / SendSMS. You will now be asked for parameters for the Choreo:

image

After providing all the inputs, you will find the code below and then you will pick and choose what you need. You can find an example for SMS and Voice in the following gist: https://gist.github.com/gbaeke/15596e3e2d185eb11720c965ab33e179. The voice Choreo uses Nexmo / Voice / TextToSpeech. Tip: Nexmo can also take input from your phone (like press ‘1’ to turn on sprinklers) and send them back to your device!

To actually fire off the SMS and voice message, we’ll do that when the button is pressed:

image

As you can see, Temboo and the APIs it exposes as Choreos makes it really easy to work with all sorts of APIs. I have only used Twilio and Nexmo here but there are many others. Again, these are all paying services and the lowest Temboo tier is quite pricey for home users. If you find it a bit too pricey, you can always use the Particle IFTTT integration to achieve similar results.

Controlling Sonos from a Particle Photon

Now for something fun! Let’s control a Sonos from a Particle Photon and a connected button. I connected a Grove Button to the Particle with simple male-to-female wires. The SIG line on the button should go to a digital port (D0 in my case). When the button is pressed, the port will read HIGH and otherwise LOW.

Controlling Sonos is another matter though. Sonos should really make simple APIs available and/or provide access through IFTTT and similar services. Until they do that, you will need to control Sonos the hard way, by connecting directly to it from the Particle and sending commands over their HTTP interface. Luckily, the people from Hover Labs, have some code on GitHub that you can build upon. I simply copied their code in my Particle app and removed references to the Hover device. By the way, the Hover is a cool device in its own right that you should definitely check out as well!

image

In the above snippet, you see part to the loop() code that checks for a button press. Since we want to toggle between Sonos PLAY and PAUSE, there’s some code for that. The hard work is done by the sonos() function which takes commands like PLAY, PAUSE, NEXT, PREVIOUS. You can check out the full code in the following gist: https://gist.github.com/gbaeke/240fb221204ff828dec06150014ec5fd. Note that the code also contains the LED and photoresitor code from earlier examples. The Sonos control is also very basic as it only implements PLAY and PAUSE so you need something in the queue. But at least you have a start to create more complex interactions.

You could also create a Particle Function that executes the Sonos code which would enable you to control your Sonos from the cloud and even connect this with other services via IFTTT. For instance, you could start playing your Sonos when you are arriving home.

Have fun controlling Sonos from your Particle!!!

Particle and Azure IoT Hub: forward events for storage and analysis

In a previous post about Partice published events, you have seen how to publish custom events to the Particle Cloud. Other devices or applications can subscribe to these events and act upon them. What if you want to do more and connect these events to custom applications? In that case, Particle has a couple of integrations that might help:

image

In this post, I will take a look at Azure IoT Hub integration which, at the moment of writing, is still in beta. Note that this integration works with events you publish from your device with Particle.publish and not with Particle Variables or Functions. Remember that in the post about events, we published a lights on and lights out event. For simplicity, we will build upon those events here.

To configure the IoT Hub integration, you will need a few things:

  • An Azure Subscription so you can logon to the portal at https://portal.azure.com (see https://azure.microsoft.com/en-us/free/ to get started)
  • An IoT Hub that you create from the portal; to get started, use the free tier which allows you to publish 8000 events per day (give or take; depends on message size as well); in the portal, use the + button

An IoT Hub has a name and works with shared access policies and access keys to be able to control the IoT Hub and send messages. To get to the policies, just click Shared Access Policies.

image

Although considered bad practice, I will use the iothubowner policy which has all required rights. Click iothubowner to view the access keys and note the primary access key. You will need that key in a moment.

In Particle Console, click the Integrations icon and click new integration. In the configuration screen, you will see:

image

It’s pretty self explanatory once you have your IoT Hub created in Azure. Just fill in the required information and note that the event name is the name of the event you have given in the call to Particle.publish. My events are called lights on and lights out and I will use lights as Event Name. This will catch both events!

To test this, the photoresistor was given enough light to fire the events. This is the result when you click on the integration after it was created:

image

When you click on one of the log entries, you will see more details:

image

You see the event payload that was sent to IoT Hub plus details about the call to IoT Hub using HTTP POST.

In IoT Hub, you will see a couple of things as well. First of all, the events:

image

In the list of devices, you will find a device with the id of the Particle Photon:

image

Note: Azure IoT Hub requires devices to authenticate but this is taken care of automatically by Particle Cloud

What you do now with these messages is up to you. You can use the new endpoints and routes feature of IoT Hub to forward events to Event Hubs or Service Bus. Or you could connect Stream Analytics to IoT Hub and save your events to Azure Storage, Data Lake, SQL, Document DB or stream the data to a real-time Power BI dashboard.

Note that although an Azure Subscription is free, not all services have free tiers. For instance, IoT Hub has a free tier but Stream Analytics does not. And although IoT Hub’s free tier is great to get started, it can only process a limited amount of events. It’s up to you to control the rate of events sent from your devices. For home use or small PoCs you should not run into issues though!