Intro to MQTT

In the previous post, I talked about how MQTT came about. In this post I will talk a little about what it is but mostly how to use it.

As a publish/subscribe (pub/sub) architecture, the nodes collect data or do actions, using a network connection. In the middle is a server or agent that will allow the nodes to only have a single connection. All the nodes connect to the server, and send or receive data with that one server. The server contains the smarts about who will receive any messages sent to it. Nodes are clients in the TCP/IP nomenclature. The MQTT agent is the server.

There is a normal TCP/IP connection protocol, where a server is listening for clients to connect. The Server will accept connections, and handle multiple clients communication. The details I won't go into, but you can read about it.

As a node connects, it may register certain topics that it will listen to. When the node connects, it must inform the server what message topics it will subscribe to. The server will make a list of these devices that are subscribed to the various topics.

Nodes may send messages on any topic, but only to the server. The server will repeat the message to each node subscribed to the topic. An example would be if a node was to send a message on the topic "outside/temperature" with a payload of 52 to the server, and there were 3 nodes subscribed to this topic, then the server would send a unique message with the topic "outside/temperature" and a payload of 52 to each of the three nodes subscribed.

A single node may subscribe to any number of topics, and send to multiple other topics. The payload of the messages sent is 65535 bytes. MQTT payloads are mostly free format, and can be single values or json formatted. Messages can be commands to set, or status of a node.

 The free format nature of the MQTT Nodes, message and topics can be a challenge for interoperability. All my nodes work the way they do for me, but your nodes may work differently. If you have a node that says it is a temperature sensor, how do I know what it is measuring the temperature of? There is a proposed standard called "Homie" to address most of the confusion.

Typically an MQTT setup will have a machine dedicated to be the MQTT agent. This machine must be running all the time, and ready to act on MQTT messages. There is no connection necessary to the cloud or any outside systems. Connecting to the cloud may be something you want to do for a particular application, or to relay some information out to a remote monitoring and control system.

Dedicating a whole Windows system is probably overkill, unless there is a great deal of processing needed on the received messages. Normally the MQTT server will just pass the messages along to the required recipients. For my home control system I am using a RaspberryPi 3B running Raspian, and I am using the mosquito MQTT server. Using apt-get, from the normal Raspian repository, MQTT can be installed.

For most of the nodes, I am using ESP8266 or ESP32 boards. These boards can be purchased locally for about $15 or mail order for $3-10. I typically need to add sensors and output to these boards so I do a little customization before deploying them. The MQTT libraries are in the normal Arduino library UI. I do all the programming in the Arduino IDE, I am not pushing the limits of anything, and just trying to make a simple reliable system.

The ESP32 and ESP8266 come in various formats. The various formats make adding peripherals convenient and mounting the devices as easy or complex as the box I put it in. I have a 3D printer, so I am able to package up the nodes in pleasant looking boxes.

In future posts, I start building nodes.

Starting Home Automation

What does this have to do with an engine monitor?

Sensors!

Somewhere along about Nick O’Leary and Dave Conway-Jones of IBM’s Emerging Technology Services group had a side project where they came up with what we now call Node-Red. Before that (1999) there were two other engineers Andy Stanford-Clark (IBM) and Arlen Nipper (Eurotech) who wanted a way to monitor oil pipelines and came up with MQTT.

MQTT is a lightweight protocol for delivering messages from sensors to a server. The protocol is direction agnostic, allowing the server to send messages to the sensors as well. The messages provide multiple levels of Quality of Service (QoS) such that the messages can be reliable or unreliable depending on the application.

You've heard of IOT, and that is pretty boring, or not, and seems that people are excited about it or not. I was for a long time too, but then I started playing with things. I've been using ESP8266 systems for a couple years now. They are all that an Arduino can be, plus they support WiFi and other protocols. The ESP8266's can be programmed using the Arduino IDE, so that makes the very friendly to hobbiests and others. It is easy to come up with a quick project, and knock something out very quickly.

I'd been working on a bunch of sensors and control panels around the house just standalone, when I had a job where I was introduced to Node-Red, and saw the potential. I moved one of my projects from standalone to Node-Red and demonstrated that standalone was not the best way. Using Node-Red I can integrate sensors and control panels in varying ways, allowing easier updates, and more capabilities.

MQTT basics

The heart of the system is MQTT. Generally MQTT has "clients" around the edges, and a central "agent" or server. For open source projects, the easiest agent to work with is Mosquito since it runs on Windows or Linux systems equally easy. There are utilities included with Mosquito allowing monitoring and control without any other GUIs or anything.

MQTT is a publish/subscribe (pub/sub) protocol. Anyone can publish a message on any topic, but only the subscribers to that topic will receive the message. Mosquito is the arbiter of the messages. Mosquito will receive all the messages, but only send them on the the appropriate receivers.

A simple sensor can send a message, and the MQTT agent receive the message and make the message available to anyone registered on that topic. If a temperature sensor broadcasts a new reading, the door monitoring devices don't need to know about it. Likewise if the door open message is sent, the water sensing system doesn't need to know about it. Or maybe they do need some additional information, it all can be programmed later.

Node-Red teaser

Node-Red allows programming flows. A flow connects several inputs on various topics to any outputs based on a need to know.

The green "connected" boxes on the left are input MQTT nodes connected to an MQTT agent. The output of those boxes go into various other boxes. The green "msg.payload" boxes are there for debugging. The blue boxes are outputs to a User Interface (UI) and the The green "connected" boxes below are MQTT outputs nodes connected to an MQTT agent. 

This flow is looking at the sensors on the garage doors to know the current state (open/closed). They both feed the message into a door open alarm node. This node has some programming associated with it (something for a future post). If the doors are open during the night, an MQTT output node that has an ESP8266 and a sounder is in my bedroom that should wake me up. It isn't physically connected, it only is a WiFi device. 

I did something, else on the input ESP8266, not just the magnetic reed sensors. I also added a DS18b20 sensor that was to let me keep track of the outside temperature. The 18B20 works in Celsius, but I only think in Fahrenheit so I have a node here that converts the temperature from C to F. 

This code is fairly straight forward, and as you see, uses JavaScript as the language. The message came with a payload that contained a number. That number was multiplied by 9 divided by 5 and added to 32. The results was made to be the payload of the resultant message. The result message goes both to the debug output (msg.payload) and a new MQTT message on a new topic. 

I'll detail more of this in future blog posts. Come back soon. 

Floor Temperature

The building that I work in is extremely cold in the winter time. Many Mondays, especially, the area where I work is below 60 degrees. I bought a simple indoor outdoor thermometer, and put the outdoor sensor on the floor, and left the main unit on top of my desk. There is often a 15 degree difference from the floor to the top of the desk.

The building has a unique architectural feature. The first two floors are indented slightly over the third floor, where I sit. I actually have a window seat which, in this case, is a distinct disadvantage. The floor under my desk actually is a slab of concrete exposed to the outside air with little to no insulation. There is a layer of carpet, that is glued to the to the concrete that may be providing some insulation.

I decided to further instrument the situation. I build a matrix of temperature sensors connected to an Arduino. I decided to use the DS18B20 sensors since there are inexpensive, and provide a direct temperature readout with out using the A/D converters. There is a library for using these sensors, and several options for connecting them.

I modified the example to only read the sensors I was interested, and output CSV format to the USB interface. I collected this file, and was able to graph the output. I didn't use a RTC module, so the time output is relative to the start time.

I set up a grid of 6 sensors using yardsticks both for support and to give consistent distance:

The sensors closest to the wall of course gave the lowest readings, but even 3 feet in from the wall showed temperatures about 5 degrees warmer. The ones that were 5 feet from the wall were another 5 degrees warmer. The second set verified the results. 

// Include the libraries we need
#include <TimeLib.h>


#include <OneWire.h>
#include <DallasTemperature.h>


// Data wire is plugged into port 2 on the Arduino
#define ONE_WIRE_BUS 2
#define TEMPERATURE_PRECISION 9

// Setup a oneWire instance to communicate with any OneWire devices (not just Maxim/Dallas temperature ICs)
OneWire oneWire(ONE_WIRE_BUS);

// Pass our oneWire reference to Dallas Temperature. 
DallasTemperature sensors(&oneWire);

// arrays to hold device addresses
DeviceAddress  device[8];
int            numSensors=0;

void setup(void)
{
  // start serial port
  Serial.begin(115200);
  Serial.println("Dallas Temperature IC Control Library Demo");

  // Start up the library
  sensors.begin();

  // locate devices on the bus
  Serial.print("Locating devices...");
  Serial.print("Found ");
  numSensors = sensors.getDeviceCount();
  Serial.print(numSensors, DEC);
  Serial.println(" devices.");

  // report parasite power requirements
  Serial.print("Parasite power is: "); 
  if (sensors.isParasitePowerMode()) Serial.println("ON");
  else Serial.println("OFF");


  // Search for devices on the bus and assign based on an index. Ideally,
  // you would do this to initially discover addresses on the bus and then 
  // use those addresses and manually assign them (see above) once you know 
  // the devices on your bus (and assuming they don't change).
  // 
  // method 1: by index
  for(int i=0; i<numSensors; i++)
  {
      if (!sensors.getAddress(device[i], i)) Serial.println("Unable to find address for Device 0"); 
  }

  // show the addresses we found on the bus
  for(int i=0; i < numSensors; i++)
  {
      Serial.print("Device ");
      Serial.print(i);
      Serial.print(" Address: ");
      printAddress(device[i]);
      Serial.println();
  }

  // set the resolution to 9 bit per device
  for(int i=0; i< numSensors; i++)
      sensors.setResolution(device[i], TEMPERATURE_PRECISION);

}

// function to print a device address
void printAddress(DeviceAddress deviceAddress)
{
  for (uint8_t i = 0; i < numSensors; i++)
  {
    // zero pad the address if necessary
    if (deviceAddress[i] < 16) Serial.print("0");
    Serial.print(deviceAddress[i], HEX);
  }
}

// function to print the temperature for a device
void printTemperature(DeviceAddress deviceAddress)
{
  float tempC = sensors.getTempC(deviceAddress);
  //Serial.print("Temp C: ");
  //Serial.print(tempC);
  Serial.print((int)DallasTemperature::toFahrenheit(tempC));
}

// function to print a device's resolution
void printResolution(DeviceAddress deviceAddress)
{
  Serial.print("Resolution: ");
  Serial.print(sensors.getResolution(deviceAddress));
  Serial.println();    
}

// main function to print information about a device
void printData(DeviceAddress deviceAddress)
{
  printAddress(deviceAddress);
  Serial.print(", ");
  printTemperature(deviceAddress);
  //Serial.println();
}

/*
 * Main function, calls the temperatures in a loop.
 */
void loop(void)
{ 
  // call sensors.requestTemperatures() to issue a global temperature 
  // request to all devices on the bus
  //Serial.print("Requesting temperatures...");
  sensors.requestTemperatures();
  //Serial.println("DONE");

  Serial.print(hour());
  printDigits(minute());
  printDigits(second());
  Serial.print(", ");
  // print the device information
  for(int i=0; i<numSensors; i++)
  {
      printData(device[i]);
      if (i<numSensors-1)
         Serial.print(", ");
  }
  Serial.println();
  delay(10000);
}

void printDigits(byte digits){
  // utility function for digital clock display: prints preceding colon and leading 0
  Serial.print(":");
  if(digits < 10)
    Serial.print('0');
  Serial.print(digits,DEC);
}

So the output looks similar to:

0:11:12, 28FFD2890117, 70, 28FFF2F10117, 70, 28FF1A040217, 71, 28FFC1F70117, 72, 28FF59070217, 71, 28FFED8D0117, 70
0:11:23, 28FFD2890117, 70, 28FFF2F10117, 70, 28FF1A040217, 71, 28FFC1F70117, 72, 28FF59070217, 71, 28FFED8D0117, 70
0:11:33, 28FFD2890117, 70, 28FFF2F10117, 70, 28FF1A040217, 71, 28FFC1F70117, 72, 28FF59070217, 71, 28FFED8D0117, 70
0:11:43, 28FFD2890117, 70, 28FFF2F10117, 70, 28FF1A040217, 71, 28FFC1F70117, 72, 28FF59070217, 71, 28FFED8D0117, 70
0:11:53, 28FFD2890117, 70, 28FFF2F10117, 70, 28FF1A040217, 71, 28FFC1F70117, 72, 28FF59070217, 71, 28FFED8D0117, 70
0:12:03, 28FFD2890117, 70, 28FFF2F10117, 70, 28FF1A040217, 71, 28FFC1F70117, 72, 28FF59070217, 71, 28FFED8D0117, 70
0:12:13, 28FFD2890117, 70, 28FFF2F10117, 70, 28FF1A040217, 71, 28FFC1F70117, 72, 28FF59070217, 71, 28FFED8D0117, 70
 

CSV files are easier it ingest, and graph using whatever spreadsheet program one might like.

The night I took a sample, the temperature was down around 10-15 degrees below zero. The graph of the output looks like:

At the lowest, the temperature hit 49 degrees against the wall.

Finally, ADS-B for under $1000.

I am sure everyone has heard of uavionix by now, but wow! This company has it together. They found a potential market for more than the GA market. They figured if they built it, they would come. The UAS market is starting to be a potential market for ADS-B out units. UAS may be a multi-million unit market for avionics, including ADS-B. (or there may be some dopey proprietary solution, that will be expensive, and less compatible).

Using the technology they have developed and mass manufacturing allowed the development costs to be amortized over many, many units. The basic unit, echoUAT is available today for $999 and can be installed on most aircraft. This will output WiFi to a tablet or phone, as well as integrate with existing in panel EFIS systems. It does require an external WAAS enabled GPS though.

I promised before that there were under $1000 solutions. NavWorx ran into a little trouble with their internal GPS, and it may have been outputting incorrect position accuracy information. This has been resolved, but now their cost is about $1500. The store seems to not have anything available right now, so I don't know what the costs are.

There is another version of the uAvionix echo, that looks like a wingtip light. The Skybeacon will replace a wingtip light, and be a full UAT ADS-B out system that can be added to any aircraft, even certified. It can be configured using WiFi, and listens to the transponder in the aircraft for squawk codes. There is an option to feed tablets in the cockpit with traffic information.

As you can see there are finally options for the rest of us. Items we can afford that deliver high performance.

Are you going to get one?

What Might Have Been part 2.

I was never a good reader. In high school and college I would go to the library to read Aviation Week, or look at books on flying (not many) and electronics. There were always little project books. About 1978, I went into the very back of a B. Dalton Book Store, and found a section on computers. I suddenly was someone who would buy books, and actually read them. B Dalton and Barnes and Noble merged in the early 80's so I would seek out book stores where ever I was.

Through the 90s I relied on Barnes and Noble (B&N) to have the books I needed for the various jobs I was pursuing. As technology changed, so did the book shelves. I think for a while my local B&N store had 2 or 3 rows with multiple shelves full of computer books. The variety was very good from end users, to developers.

In the late 90s Amazon was born, and my book buying shifted. I went from perusing the book shelves at B&N to shopping on line, and ordering whatever I needed. It didn't matter that it might take 3 days to get the book, it was probably available, and I was going to get it. Eventually Amazon gamed the system with their multiply rejected one-click patent, and I vowed never to buy anything there again.

It didn't matter, there were other book sellers on line. I mostly use Half.com, but there were others including A1 (now out of business). These book sellers would sell new and used books at a fraction of the price others were.

As technology started moving faster, some of the books were out of date by the time they hit the shelves. Online forums replaced the tech books. Open source software begat open source documentation, and eventually tech books seemed obsolete. I mostly stopped shopping in book stores, since everything I was looking for was on line.

A funny thing happened though. The magazine stores went out of business, so I would go to B&N about once a month to browse magazines for some of the fringe subjects I am interested in (Model Railroading, Wooden Boats). These are subjects I am interested in, not always for the actual heart of the subject, but they are craftsmen and hackers on a whole nuther level.

This month, I had an article published in Kitplanes Magazine and I wanted to get a couple copies. While wandering the aisles in B&N to see what else is going on, I noticed several kits that might be useful. The store carries Little Bits and Arduino kits, along with some other robots and RasberryPi kits.

The kits sold at B&N are not the cheapest, but are probably suitable to the retail space where clerks may not be able to provide much after the sales support. But maybe their staff knows a thing or two. On top of kit display, there was a sign:

Yes a mini maker faire at all the stores. I am hoping the staff is helping and participating. It sounds like they are moving in the right direction. Engaging the community. Go if you can, tell 'em it is a great idea.

If you go, please give me some feedback.

Becoming an Embedded Programmer

Some people want to be embedded software engineers. Not everyone is cut out for doing web pages, an making the page a little more blue (it happened to me when I was confused, early 2000's). No, some people like to get data in the raw, and make it something that people will use.

I've been hiring for the company I work for, doing interviews for the last year. It has been tough sledding trying to find young people out of school, and people with years of experience to understand what embedded systems can be. Some think, raw hardware, no operating system. Some others think once you have the OS, why not make it all web based. Others don't want to do Linux, and think the system ought to be Windows based.

The company I work for writes software for a Linux based embedded system. Most of the projects are reading data from some hardware device, quickly transforming it, and maybe jamming it back into another hardware device, or writing he values to a hard disk, or NVRAM. Data synchronization is paramount, it doesn't work to get something a few seconds from now mixed up with data that is happening now. Sometimes we need to keep two input sources in sync (IE audio and video), so two threads need to know when to start and finish.

Getting a Rasberry Pi or something similar (IE Orange Pi) is a great start. For the cost of a couple cups of coffee (or only one, depending on where you buy things) you can get an embedded system. The Pi systems are Linux based, and allow writing code right on the device. The Pi's also allow developing on desktop systems, cross compiling for the hardware.

One thing the Pi's all seem to have is rich AV capabilities. All seem to have HDMI out, along with audio in and out. Many folks use the Pi systems as desktop systems, all on their own. Most of a Linux system is available, including X windows, and desktop type programs (word processors, spreadsheets and browsers).

As a developer, play with one of these Pi systems, and use it as a data collector system. Go ahead, connect something to the GPIO pins. It could be an LM34 chip measure the temperature of the garage or something. Fire up a web server an built some Perl or Python to display he last 36 hours of temperature. Tada, you have become an embedded programmer.

Examples... Well give me a couple days. Do you want to collect the stuff you need? There is a service called Hackerboxes that will send you random projects every month to build. For September 2016, they have an OrangePi kit. I am downloading a desktop image while typing this in. Compressed the desktop is 540MB.

Keeping thing in sync, use mutexes and semaphores. If you don't know what they are, read on the web. They are a great tool to allow multiple threads to stay in sync, and keep the data correct and useful. If you code in Java, read about wait and notify Java.lang.object. If you use C or Perl look at posix threads.

It is cheap, and you can be an embedded programmer in no time.

What Could Have Been

Radio Shack is no more. I am both sad, and not surprised. I got my start with Radio Shack components, 80 in One kits, and Forest Mims books. I relied on Radio Shack for resistors, and my introduction to Integrated Circuits. They didn't have everything, but they had most of what I needed through college building computers and robots. If I needed some wire, or a 15k 1/4watt resistor at 8pm on a Thursday, Radio Shack was the answer.

Tandy wire and cable made some wonderful coax and other wire. The RG-58 was really nice, and available both with a braided center core and with a solid center core. Of course they sold connectors to go with it. Asking a clerk for help, they happily would cut custom lengths, or suggest pre-made cables.

Somewhere along they way, Radio Shack changed. After college many things were in flux. The computers that people built at home were no longer chips (yes, Radio Shack sold 8085, and 8080 CPU's plus others), but were boards. PC's that people built were now plug together boards. Radio Shack only sold some expansion boards, and pre-assembled computers.

Radios changed as well. Fewer and fewer people were buying component stereos, and self installing car stereos. I am sure the sales at Radio Shack declined during the late 80s and early 90s. To compensate, Radio Shack expanded into television sales, and other consumer electronics. Eventually Radio Shack was getting heavily into the cell phone sales for all the carriers.

Expansion of the consumer electronics area caused the reduction in the component floor space. The clerks didn't want to waste their time selling 10uf capacitors when a customer wanted a new cell phone. The knowledge of the staff was on the decline. I've heard stories about EE students that wanted to work at Radio Shack were actively discouraged because they might spend too much time helping with components.

The last couple of years, the president of Radio Shack was really making an effort to "fix" the decline of the brand. There were engagements with hackaday and other hobbiest web communities, The communities were slightly hostile, but most of the ideas were too little too late. The brand was bought by Sprint in 2015, and most of the stores have closed.

What Could Radio Shack Have Been?

I've been going to Micro Center for a bunch of years. They mostly were the place to get reasonably priced PC's. They had all the accessories, boards, power supplies cases, and cables. Their prices are good, for a retail operation. They had a closeout section that often had some amazing deals.

PC prices have plunged. When a PC or laptop used to cost $1000-1500 in the early 2000's, now a modern PC or laptop only cost $300-500. The gaming systems are not making as much money, it seems, and the book sales are almost non-existent. The shelves are still full of accessory boards, CPU's cases and power supplies.

Micro Center has expanded into the hobby space. They have a very large selection of Arduino boards and accessories. The Rasberry Pi selection is equally full. The prices are as good as mail order as well. At Christmas time, they had Rasberry Pi Zero boards for $5, but were sold out quickly. The aisles were full when I went there yesterday. I don't have any insight to the income this is generating, but people are buying the the hobby electronics, and I will continue to.

What if Radio Shack had followed the hobbyist trends? Well Radio Shack did sell Arduino boards toward the end. The prices were very high (I think I remember looking at an Uno for about $24). It was something I couldn't justify. What if they had kept the prices better? With the mall space rent as high as it is, they might have gone out of business sooner. Hard to say.

Do you think, if Radio Shack had done a better job with hobby electronics, they'd still be around?