Visualizing Degu Sensor Data with Elasticsearch

In the previous tutorial, Getting Started with IoT Using Degu, we set up a Degu Sensor Unit and Degu Gateway to send temperature and barometric pressure data every minute to an email account via AWS SNS.

Sending data to an email is a good start, however we can take a step towards a more realistic IoT implementation by storing and visualising the captured data instead of emailing it to ourselves. In this tutorial, we’ll show how to use Elasticsearch and Kibana to do that. We'll cover:

  • Creating an AWS Elasticsearch Service Domain

  • Creating an AWS IoT rule for forwarding Degu sensor data to Elasticsearch

  • Configuring Kibana to accept Degu sensor data

  • Setting up a simple Kibana dashboard to visualise temperature and pressure sensor data captured by your Degu Sensor Unit

By the end of this tutorial, you will have built a simple end-to-end IoT system covering hardware configuration, cloud setup, data management and data visualisation.

System Overview

Building on the existing system from the previous tutorial, your IoT system will now have the following components:

  • A Degu Sensor Unit as an edge device

  • A Degu Gateway (A6 LAN in this case) to upload sensor data to the cloud

  • The cloud component, utilizing these three AWS services:

    • AWS IAM

    • AWS IoT Core

    • AWS Elasticsearch & Kibana

With the new components to your system being Elasticsearch and Kibana, let’s have a quick look at each.

Elasticsearch

Elasticsearch is a distributed, open source, JSON document search and analytics engine. It is designed to work with many different kinds of data, including textual, numerical, geospatial, structured as well as unstructured data types.

For more information, see the Elasticsearch product info.

Kibana

Kibana integrates closely with Elasticsearch to make visualisation of stored data easy and straightforward. It comes with a range of graphing tools, including geospatial as well as network graphing capabilities out of the box. It’s capable of both building dashboards to visualise captured data and also allow data exploration - a powerful combination well suited to IoT.

For more information, see the Kibana product info.

Step 1 - Create Elasticsearch Service Domain

To be able to use Elasticsearch and Kibana in your IoT system, we first need to create an AWS Elasticsearch Service Domain.

To create a new Elasticsearch Service Domain:

  1. Log into your AWS account

  2. Open your AWS console

  3. Go to Elasticsearch Service

  4. Click on ‘Create a new domain’

Follow through the domain creation process, filling in the relevant information detailed below at each step.

  1. Choose deployment type

    1. For ‘Deployment type’, select ‘Development and testing’, then click next

  2. Configure cluster

    1. Set the ‘Elasticsearch domain name’ field to ‘degu-iot-example’

    2. For the ‘Instance type’, choose ‘t2.small.elasticsearch’

    3. Leave all other options as their default values, and click next.

  3. Set up access to allow access only from specific IP(s)

    1. Under the ‘Network configuration’ heading, choose ‘Public access’

    2. Under the ‘Access policy’ heading, select ‘Allow access to the demain from specific IP(s)’, from the ‘Set the domain access policy’ to drop down menu and enter your IP address. AWS will automatically generate an access policy for you.

    3. Review and click next.

  4. Review the Elasticsearch Domain details and click confirm.

After successfully creating the domain, wait for the domain status change from ‘Loading’ to ‘Active’. This can take up to 10 minutes. The page will automatically update when it’s ready.

 
Elasticsearch Domain status.png
 

We can quickly confirm that everything has worked by opening the provided Kibana URL (in the format of search-degu-iot-example.xxxx.es.amazonaws.com/_plugin/kibana) and selecting ‘Explore on my own’, which will take you to the home page of Kibana. We will come back to configure Kibana after we set up the AWS IoT Rule.

Step 2 - Create AWS IoT Rule

AWS IoT rules have some built-in features that can define what AWS IoT should do if a message is received. In this example case we’d like to define a rule that passes the sensor data to AWS Elasticsearch Service we just created in the previous step.

To create the AWS IoT Rule:

  1. Open your AWS IoT console
  2. Go to Act, click on 'Create a rule'
  3. Set the rule 'Name' to 'BMP280_To_Kibana'
  4. Under the ‘Rule query statement’ field, replace the default contents with the following:

SELECT topic(3) as thing, timestamp, parse_time("yyyy-MM-dd'T'HH:mm:ssZZ", timestamp(), "Pacific/Auckland" ) as ts, abs(state.reported.temp) as temperature, abs(state.reported.pres) as pressure FROM '$aws/things/+/shadow/update/accepted'

    1. pres and temp are the keys reported in the shadow state document in Step 6 of the first tutorial.
    2. If you’d like to set a different time zone, you can find a list of accepted timezone IDs here under the Canonical ID column.

  1. Under the ‘Set one or more actions’ heading, click the ‘Add action’ button.
  2. From the list, choose ‘Send a message to the Amazon Elasticsearch Service’, then click ‘Configure action’
  3. From the ‘Domain name’ dropdown list, select ‘degu-iot-example’
  4. Set 'ID' to ${newuuid()}
  5. Set 'Index' to temperature_and_pressure
  6. Set 'Type' to bmp280.
  7. Click 'Create Role' and put AccessKibana as the role name, then click ‘Create role’.
  8. Click 'Add action'.
  9. Click 'Create rule'.
If everything has gone well, all the sensor data will now be sent to the AWS Elasticsearch Service domain we’ve just set up.

Step 3 - Configure Kibana

Create Index Patterns

In order to let Kibana recognise the sensor data, we will need to first create an Kibana index pattern. Index patterns help Kibana determine what information to retrieve for you from Elasticsearch.

In the Kibana home page:

  1. Under 'Manage and Administer the Elastic Stack', click on 'Index Patterns'.

  2. Click the 'Create index pattern' button.

  3. Under 'Index pattern', enter temperature_and_pressure.

    1. You should receive a notification “Success! Your index pattern matches 1 index.”

  4. Click 'Next step'.

  5. Under the ‘Time Filter field name’ dropdown choose 'ts'.

  6. Click 'Create index pattern'.

You should now have a table showing all of the fields that the index pattern has matched and that you can now search or aggregate data for.

 
Kibana index pattern.png
 

Create Visualizations

Now we are ready to create a simple dashboard containing the temperature and atmospheric pressure sensor data your Degu Sensor unit is collecting. We’ll first create a visualisation for the collected temperature data, then repeat the process for the barometric pressure data.

From the Kibana home page:

  1. Under 'Visualize and Explore Data', click on 'Visualize'.

  2. Click 'Create a visualization'.

  3. For the visualization type, choose 'Line'.

  4. For the index pattern, choose 'temperature_and_pressure'.

  5. In the 'Metrics' block, expand 'Y-Axis Count’,

    1. From the aggregation dropdown, select 'Average'.

    2. In the Field dropdown, select 'temperature'.

  6. In the ‘Buckets’ block, click 'X-Axis'.

    1. In the aggregation dropdown, select 'Date Histogram' as the aggregation type.

    2. Choose 'ts' as Field.

    3. Choose 'Minute' as Interval.

  7. Press the ‘play’ button (blue with white triangle) above the ‘Metrics’ block, and Kibana will show a preview of the collected data.

  8. Click 'Save' at the top left menu. Kibana will prompt you to set a title, which you can name appropriately.

 
Kibana visualisation temperature.png
 

Next, following the same series of steps as above, create a visualization for the barometric pressure data. However, this time use 'pressure' for ‘Field’ (step 7) and save it as 'Average Pressure' (step 13). Be careful to not override your existing visualisation.

Create Dashboard

Now, it’s time for us to create a dashboard which will display the two visualizations we’ve just created together.

Go back to the Kibana home page.

  1. Click on ‘Dashboard’ under 'Visualize and Explore Data'.

  2. Click 'Create new dashboard'.

  3. Click 'Add' at the top left menu.

  4. A pane will appear. Click on both of the names of the visualisations you saved earlier, in this case we used ‘Office Temperature Data’ and ‘Office Barometric Pressure Data’

  5. Close the pane listing the visualisations

  6. Click on ‘Save’ in the top left, and give your fantabulous new dashboard a name, then click ‘Confirm Save’.

 
Kibana finished dashboard temp _ pressure.png
 

Congratulations, you now have an Elasticsearch connected Kibana dashboard displaying aggregated live temperature and barometric pressure data from your Degu Sensor Unit.

This should be a good starting point to further explore AWS IoT Rules, Elasticsearch and Kibana, as well as try out some of the other sensor types available for your Degu Sensor Unit.

Guest User