Intro to MQTT

In the previous post, I talked about how MQTT came about. In this post I will talk a little about what it is but mostly how to use it.

As a publish/subscribe (pub/sub) architecture, the nodes collect data or do actions, using a network connection. In the middle is a server or agent that will allow the nodes to only have a single connection. All the nodes connect to the server, and send or receive data with that one server. The server contains the smarts about who will receive any messages sent to it. Nodes are clients in the TCP/IP nomenclature. The MQTT agent is the server.

There is a normal TCP/IP connection protocol, where a server is listening for clients to connect. The Server will accept connections, and handle multiple clients communication. The details I won't go into, but you can read about it.

As a node connects, it may register certain topics that it will listen to. When the node connects, it must inform the server what message topics it will subscribe to. The server will make a list of these devices that are subscribed to the various topics.

Nodes may send messages on any topic, but only to the server. The server will repeat the message to each node subscribed to the topic. An example would be if a node was to send a message on the topic "outside/temperature" with a payload of 52 to the server, and there were 3 nodes subscribed to this topic, then the server would send a unique message with the topic "outside/temperature" and a payload of 52 to each of the three nodes subscribed.

A single node may subscribe to any number of topics, and send to multiple other topics. The payload of the messages sent is 65535 bytes. MQTT payloads are mostly free format, and can be single values or json formatted. Messages can be commands to set, or status of a node.

 The free format nature of the MQTT Nodes, message and topics can be a challenge for interoperability. All my nodes work the way they do for me, but your nodes may work differently. If you have a node that says it is a temperature sensor, how do I know what it is measuring the temperature of? There is a proposed standard called "Homie" to address most of the confusion.

Typically an MQTT setup will have a machine dedicated to be the MQTT agent. This machine must be running all the time, and ready to act on MQTT messages. There is no connection necessary to the cloud or any outside systems. Connecting to the cloud may be something you want to do for a particular application, or to relay some information out to a remote monitoring and control system.

Dedicating a whole Windows system is probably overkill, unless there is a great deal of processing needed on the received messages. Normally the MQTT server will just pass the messages along to the required recipients. For my home control system I am using a RaspberryPi 3B running Raspian, and I am using the mosquito MQTT server. Using apt-get, from the normal Raspian repository, MQTT can be installed.

For most of the nodes, I am using ESP8266 or ESP32 boards. These boards can be purchased locally for about $15 or mail order for $3-10. I typically need to add sensors and output to these boards so I do a little customization before deploying them. The MQTT libraries are in the normal Arduino library UI. I do all the programming in the Arduino IDE, I am not pushing the limits of anything, and just trying to make a simple reliable system.

The ESP32 and ESP8266 come in various formats. The various formats make adding peripherals convenient and mounting the devices as easy or complex as the box I put it in. I have a 3D printer, so I am able to package up the nodes in pleasant looking boxes.

In future posts, I start building nodes.

Starting Home Automation

What does this have to do with an engine monitor?

Sensors!

Somewhere along about Nick O’Leary and Dave Conway-Jones of IBM’s Emerging Technology Services group had a side project where they came up with what we now call Node-Red. Before that (1999) there were two other engineers Andy Stanford-Clark (IBM) and Arlen Nipper (Eurotech) who wanted a way to monitor oil pipelines and came up with MQTT.

MQTT is a lightweight protocol for delivering messages from sensors to a server. The protocol is direction agnostic, allowing the server to send messages to the sensors as well. The messages provide multiple levels of Quality of Service (QoS) such that the messages can be reliable or unreliable depending on the application.

You've heard of IOT, and that is pretty boring, or not, and seems that people are excited about it or not. I was for a long time too, but then I started playing with things. I've been using ESP8266 systems for a couple years now. They are all that an Arduino can be, plus they support WiFi and other protocols. The ESP8266's can be programmed using the Arduino IDE, so that makes the very friendly to hobbiests and others. It is easy to come up with a quick project, and knock something out very quickly.

I'd been working on a bunch of sensors and control panels around the house just standalone, when I had a job where I was introduced to Node-Red, and saw the potential. I moved one of my projects from standalone to Node-Red and demonstrated that standalone was not the best way. Using Node-Red I can integrate sensors and control panels in varying ways, allowing easier updates, and more capabilities.

MQTT basics

The heart of the system is MQTT. Generally MQTT has "clients" around the edges, and a central "agent" or server. For open source projects, the easiest agent to work with is Mosquito since it runs on Windows or Linux systems equally easy. There are utilities included with Mosquito allowing monitoring and control without any other GUIs or anything.

MQTT is a publish/subscribe (pub/sub) protocol. Anyone can publish a message on any topic, but only the subscribers to that topic will receive the message. Mosquito is the arbiter of the messages. Mosquito will receive all the messages, but only send them on the the appropriate receivers.

A simple sensor can send a message, and the MQTT agent receive the message and make the message available to anyone registered on that topic. If a temperature sensor broadcasts a new reading, the door monitoring devices don't need to know about it. Likewise if the door open message is sent, the water sensing system doesn't need to know about it. Or maybe they do need some additional information, it all can be programmed later.

Node-Red teaser

Node-Red allows programming flows. A flow connects several inputs on various topics to any outputs based on a need to know.

The green "connected" boxes on the left are input MQTT nodes connected to an MQTT agent. The output of those boxes go into various other boxes. The green "msg.payload" boxes are there for debugging. The blue boxes are outputs to a User Interface (UI) and the The green "connected" boxes below are MQTT outputs nodes connected to an MQTT agent. 

This flow is looking at the sensors on the garage doors to know the current state (open/closed). They both feed the message into a door open alarm node. This node has some programming associated with it (something for a future post). If the doors are open during the night, an MQTT output node that has an ESP8266 and a sounder is in my bedroom that should wake me up. It isn't physically connected, it only is a WiFi device. 

I did something, else on the input ESP8266, not just the magnetic reed sensors. I also added a DS18b20 sensor that was to let me keep track of the outside temperature. The 18B20 works in Celsius, but I only think in Fahrenheit so I have a node here that converts the temperature from C to F. 

This code is fairly straight forward, and as you see, uses JavaScript as the language. The message came with a payload that contained a number. That number was multiplied by 9 divided by 5 and added to 32. The results was made to be the payload of the resultant message. The result message goes both to the debug output (msg.payload) and a new MQTT message on a new topic. 

I'll detail more of this in future blog posts. Come back soon. 

Floor Temperature

The building that I work in is extremely cold in the winter time. Many Mondays, especially, the area where I work is below 60 degrees. I bought a simple indoor outdoor thermometer, and put the outdoor sensor on the floor, and left the main unit on top of my desk. There is often a 15 degree difference from the floor to the top of the desk.

The building has a unique architectural feature. The first two floors are indented slightly over the third floor, where I sit. I actually have a window seat which, in this case, is a distinct disadvantage. The floor under my desk actually is a slab of concrete exposed to the outside air with little to no insulation. There is a layer of carpet, that is glued to the to the concrete that may be providing some insulation.

I decided to further instrument the situation. I build a matrix of temperature sensors connected to an Arduino. I decided to use the DS18B20 sensors since there are inexpensive, and provide a direct temperature readout with out using the A/D converters. There is a library for using these sensors, and several options for connecting them.

I modified the example to only read the sensors I was interested, and output CSV format to the USB interface. I collected this file, and was able to graph the output. I didn't use a RTC module, so the time output is relative to the start time.

I set up a grid of 6 sensors using yardsticks both for support and to give consistent distance:

The sensors closest to the wall of course gave the lowest readings, but even 3 feet in from the wall showed temperatures about 5 degrees warmer. The ones that were 5 feet from the wall were another 5 degrees warmer. The second set verified the results. 

// Include the libraries we need
#include <TimeLib.h>


#include <OneWire.h>
#include <DallasTemperature.h>


// Data wire is plugged into port 2 on the Arduino
#define ONE_WIRE_BUS 2
#define TEMPERATURE_PRECISION 9

// Setup a oneWire instance to communicate with any OneWire devices (not just Maxim/Dallas temperature ICs)
OneWire oneWire(ONE_WIRE_BUS);

// Pass our oneWire reference to Dallas Temperature. 
DallasTemperature sensors(&oneWire);

// arrays to hold device addresses
DeviceAddress  device[8];
int            numSensors=0;

void setup(void)
{
  // start serial port
  Serial.begin(115200);
  Serial.println("Dallas Temperature IC Control Library Demo");

  // Start up the library
  sensors.begin();

  // locate devices on the bus
  Serial.print("Locating devices...");
  Serial.print("Found ");
  numSensors = sensors.getDeviceCount();
  Serial.print(numSensors, DEC);
  Serial.println(" devices.");

  // report parasite power requirements
  Serial.print("Parasite power is: "); 
  if (sensors.isParasitePowerMode()) Serial.println("ON");
  else Serial.println("OFF");


  // Search for devices on the bus and assign based on an index. Ideally,
  // you would do this to initially discover addresses on the bus and then 
  // use those addresses and manually assign them (see above) once you know 
  // the devices on your bus (and assuming they don't change).
  // 
  // method 1: by index
  for(int i=0; i<numSensors; i++)
  {
      if (!sensors.getAddress(device[i], i)) Serial.println("Unable to find address for Device 0"); 
  }

  // show the addresses we found on the bus
  for(int i=0; i < numSensors; i++)
  {
      Serial.print("Device ");
      Serial.print(i);
      Serial.print(" Address: ");
      printAddress(device[i]);
      Serial.println();
  }

  // set the resolution to 9 bit per device
  for(int i=0; i< numSensors; i++)
      sensors.setResolution(device[i], TEMPERATURE_PRECISION);

}

// function to print a device address
void printAddress(DeviceAddress deviceAddress)
{
  for (uint8_t i = 0; i < numSensors; i++)
  {
    // zero pad the address if necessary
    if (deviceAddress[i] < 16) Serial.print("0");
    Serial.print(deviceAddress[i], HEX);
  }
}

// function to print the temperature for a device
void printTemperature(DeviceAddress deviceAddress)
{
  float tempC = sensors.getTempC(deviceAddress);
  //Serial.print("Temp C: ");
  //Serial.print(tempC);
  Serial.print((int)DallasTemperature::toFahrenheit(tempC));
}

// function to print a device's resolution
void printResolution(DeviceAddress deviceAddress)
{
  Serial.print("Resolution: ");
  Serial.print(sensors.getResolution(deviceAddress));
  Serial.println();    
}

// main function to print information about a device
void printData(DeviceAddress deviceAddress)
{
  printAddress(deviceAddress);
  Serial.print(", ");
  printTemperature(deviceAddress);
  //Serial.println();
}

/*
 * Main function, calls the temperatures in a loop.
 */
void loop(void)
{ 
  // call sensors.requestTemperatures() to issue a global temperature 
  // request to all devices on the bus
  //Serial.print("Requesting temperatures...");
  sensors.requestTemperatures();
  //Serial.println("DONE");

  Serial.print(hour());
  printDigits(minute());
  printDigits(second());
  Serial.print(", ");
  // print the device information
  for(int i=0; i<numSensors; i++)
  {
      printData(device[i]);
      if (i<numSensors-1)
         Serial.print(", ");
  }
  Serial.println();
  delay(10000);
}

void printDigits(byte digits){
  // utility function for digital clock display: prints preceding colon and leading 0
  Serial.print(":");
  if(digits < 10)
    Serial.print('0');
  Serial.print(digits,DEC);
}

So the output looks similar to:

0:11:12, 28FFD2890117, 70, 28FFF2F10117, 70, 28FF1A040217, 71, 28FFC1F70117, 72, 28FF59070217, 71, 28FFED8D0117, 70
0:11:23, 28FFD2890117, 70, 28FFF2F10117, 70, 28FF1A040217, 71, 28FFC1F70117, 72, 28FF59070217, 71, 28FFED8D0117, 70
0:11:33, 28FFD2890117, 70, 28FFF2F10117, 70, 28FF1A040217, 71, 28FFC1F70117, 72, 28FF59070217, 71, 28FFED8D0117, 70
0:11:43, 28FFD2890117, 70, 28FFF2F10117, 70, 28FF1A040217, 71, 28FFC1F70117, 72, 28FF59070217, 71, 28FFED8D0117, 70
0:11:53, 28FFD2890117, 70, 28FFF2F10117, 70, 28FF1A040217, 71, 28FFC1F70117, 72, 28FF59070217, 71, 28FFED8D0117, 70
0:12:03, 28FFD2890117, 70, 28FFF2F10117, 70, 28FF1A040217, 71, 28FFC1F70117, 72, 28FF59070217, 71, 28FFED8D0117, 70
0:12:13, 28FFD2890117, 70, 28FFF2F10117, 70, 28FF1A040217, 71, 28FFC1F70117, 72, 28FF59070217, 71, 28FFED8D0117, 70
 

CSV files are easier it ingest, and graph using whatever spreadsheet program one might like.

The night I took a sample, the temperature was down around 10-15 degrees below zero. The graph of the output looks like:

At the lowest, the temperature hit 49 degrees against the wall.