IOT Analytics using Elastic, Logstash and Kibana (ELK Stack) — Part 1

Hey guys,

Today we’ll be talking about how to send data collected from a sensor (DHT11 temperature and humidity here) from a raspberry pi, and ship these readings off to ELK stack.

This setup will be done considering you do not have a machine that can constantly have the whole stack running at all times but still want to factor in all the data that gets recorded even when our stack isn’t running.

This tutorial will be split into two parts as we will be setting up an intermediate “Broker” to store data when we can’t have ELK running, but pick up these data points as soon as it is up and running.

The first part deals with the following:

1. Setting up Raspberry Pi to capture sensor data
2. Setting up a free Redis service on heroku
3. Shipping sensor data to Redis

The following part(Part 2) will deal with the actual ELK setup which will pull data from Redis.

Setting up the Raspberry Pi

Photo by Louis Reed on Unsplash

Installing the OS

Head over to https://www.raspberrypi.org/downloads/raspbian/

1. Download the image with desktop, thats the one I chose to use. You could get away with the lite version as well. As long as you have a shell with python3 installed, you should be good to go.
2. Unzip the downloaded image.
3. Use Etcher and burn the image to the SD card.
4. THERE. Your os has been installed and you’re good to go.

But wait, before you unplug in that SD card and fire the PI up, read through the Optional Steps mentioned below.

Optional Steps

If you have no monitor with an HDMI port to connect the Raspberrypi to; here are a few handy reference links.

Setting up WIFI without a monitor

The link mentions how to setup Wifi automatically on first boot. Read More

Even if your wifi network is WPA2-PSK, leave it as WPA-PSK itself on the wpa_supplicant.conf file, otherwise your wifi will not connect. (I learnt this the hard way.)

Enable SSH on Raspberry Pi without a monitor

To setup SSH without a monitor; Read More

I’m not writing in detail for how to do the steps in detail as the mentioned links do a pretty good job at it!

Sensor Connections

I’m using a DHT11 sensor. The reason why I picked this is because it was pretty cheap and for the price I was getting two readings, two useful ones for me at the time of writing this tutorial.

Setup

For the actual setup, connect the GPIO pins on the Raspberry Pi as follows:

• Connect the VCC to a 5 volt GPIO pin (PIN 2)
• Connect Signal to Raspberry Pi GPIO 4 (PIN 7)
• Connect Ground to any Raspberry Pi GND pin. (PIN 6)

You could use any of the other Ground and GPIO pins as well. For the GPIO pins, just make sure you change the code accordingly!

Once setup, you should end up with something that looks like this.

Now before we start getting the data, lets talk about the actual “Cache” or “Broker” we will be using.

Redis

Okay, if you’re someone who has never heard about Redis and has no clue what on earth is this?

Redis is a datastore that can stores data in memory. The data structures and are not like the conventional databases so it could take a while wrap your head around how this works.

If you’re like me, you might be wondering;

What on earth are we using Redis for?

Let’s just call Redis a temporary buffer that stores data. This temporary buffer helps us run ELK in a container that can be taken down at any time. When it goes up again, it would pull data stored on Redis during the time it our stack was down. Considering ELK is a huge memory hogger, this seemed like a not-so-bad pipeline.

Not bad for an entirely free service.

Figuring out how to store a “KEY VALUE” pair on Redis was a definitely a pain.

Elasticsearch supports lists, channels and pattern_channel from Redis.

What is a list?

What is a channel?

What on earth is a pattern_channel!!!!

Photo by Tim Gouw on Unsplash

That was my exact condition; until I pieced together information from multiple sources. So after multiple failures, here’s a summary about Redis of what I understand.

• Redis stores data in memory
• A list is basically an array of values (we’re only using Lists, the rest of out of syllabus)
• When we insert or append a value to a “KEY”, it means appending a value to an array which has the name “KEY”. In this case I was sending room sensor data, so I decided to call it “room_data”.
• Data inserted is appended, i.e. into a list that follows the rules of a queue.
• When Elasticsearch pulls data from Redis, it empties the list there-by clearing the queue.
• We store Json or key value pairs in Redis by actually storing a string version of the Dict/JSON! You can’t store a dictionary or a JSON as-is in Redis!

That should be good enough to get you going!

Now setup a free app on Heroku and add Redis from under the “Resources” tab. This tab will be visible to you only after you create an app on Heroku and open the app.

When you’re done, it should looks something like this

Heroku App with Redis Add-on

Once you’ve got redis added as a resource, head over to credentials of the DataStore which we will need for the next section.

View Credentials — Redis

Code:

Photo by Chris Ried on Unsplash

Follow the link to access the code for sending data from sensors to redis.

https://github.com/arjun921/raspia

I’d recommend cloning the repo on your Pi which I presume is powered on, has internet and you have shell access via SSH by now.

Set the following environment variables(with credentials you got from Redis):

• REDIS_HOST
• REDIS_PWD
• REDIS_PORT

Then, run the following command

python3 send_data_to_redis.py

On running this, if all goes well, you should be able to see a Json stdout on the screen with sensor values. This data is sent to Redis as a string that gets added to a list. This list is then emptied by logstash when it comes online.

Hurray! We’re done with shipping of our data!

We’ll talk about the actual ELK setup for pulling data from Redis in the next part.

Link-to-Part-2