ESP8266 Arduino IDE Web Server Using Callbacks

26th November 2015 by InternetOfHomeThings in All Posts, ESP8266 : An IoT System on Chip, Internet of Things | 21 Comments

While the Arduino IDE is a convenient and familiar platform to develop ESP8266 Web Server based projects, it does have a significant drawback. That is, once the Web Server is started, it is necessary to poll the server each iteration of the loop() function to check for new connections before processing the request.

Putting the check in a loop with other activities, such as reading sensors could and often does result in significant response delays. And possible loss of communication due to connection timeouts.

A better approach is to use a callback that fires whenever an external client connects to the ESP8266 server. This frees up the sketch loop() from the task of servicing http request using inefficient polling code.

And callbacks is how a Web Server is implemented using the ESP8266 EspressIf SDK. But why is it missing from the Arduino IDE? I suspect it is because that is the way it is done with Arduino-based systems—making it both familiar and perhaps suitable for Arduino code reuse with the ESP8266.

But it is not the best solution for new designs.

So that got me thinking…

Why not customize the typical web server sketch to use the SDK callback schema? After all, it is possible to call ESP8266 SDK functions from the Arduino IDE. While searching, I could not find any examples of this being done, so I’ve created one myself.

And here it is…

The SDK Server Architecture

The SDK API requires several layers of callbacks in a web server design. Since these callbacks are usable in a sketch without modification, the actual sketch functions included in this example are shown.

First, the server must be setup in the sketch setup() routine. This initialization code was encapsulated in the following function, “SdkWebServer_Init()”, called in setup():

void SdkWebServer_Init(int port) {
LOCAL struct espconn esp_conn;
LOCAL esp_tcp esptcp;
//Fill the connection structure, including "listen" port
esp_conn.type = ESPCONN_TCP;
esp_conn.state = ESPCONN_NONE;
esp_conn.proto.tcp = &esptcp;
esp_conn.proto.tcp->local_port = port;
esp_conn.recv_callback = NULL;
esp_conn.sent_callback = NULL;
esp_conn.reverse = NULL;
//Register the connection timeout(0=no timeout)
espconn_regist_time(&esp_conn,0,0);
//Register connection callback
espconn_regist_connectcb(&esp_conn, SdkWebServer_listen);
//Start Listening for connections
espconn_accept(&esp_conn);
}

Upon an external TCP connection, the callback “SdkWebServer_listen” is executed:

void SdkWebServer_listen(void *arg)
{
struct espconn *pesp_conn = ( espconn *)arg;
espconn_regist_recvcb(pesp_conn, SdkWebServer_recv);
espconn_regist_reconcb(pesp_conn, SdkWebServer_recon);
espconn_regist_disconcb(pesp_conn, SdkWebServer_discon);
}

This functions registers 3 connection-specific callbacks:

SdkWebServer_recv – Called when connection data is received. This callback replaces the polling loop when using the Arduino library.
SdkWebServer_recon – Called when TCP connection is broken.
SdkWebServer_discon-Called when a TCP connection is closed.

The SdkWebServer_recv() function processes http GET and POST requests in a similar manner as the processing in the loop() after an Arduino client connection is detected.

Putting It All Together

In order to contrast the commonly used Arduino Web Server library with the SDK API, I’ve put together an example sketch that supports both web server approaches. It’s on Github:

ESP8266-12 Arduino IDE Web Server Example

This example provide the same functionality with either server interface. The selection is made prior to building the sketch, using precompilation directives. When possible, the same code is used with both selections. The difference is that the SDK uses callbacks while the Arduino polls for connections.

While no additional hardware is needed to run and test the Web Server, this example assumes the ESP8266 circuit from this post. But the code will work with any ESP8266 circuit, even with no externally connected sensors or controls. But without the ADC multiplexer from the referenced post, the ADC will simply make the same measurement for each multiplexer setting. Which is just fine for the purposes of this example.

Testing the Server From A Web Browser

Since both versions of the Web Server provide the same functionality, the same test can be performed for verification.

You might need to changed the static IP set in this example to match your own LAN subnet. The defaults are currently set to:

IP: 192.168.0.132

Listen Port: 9701

The test url is thus:

192.168.0.132:9701/?request=GetSensors

And the JSON returned from the Web Server should be in the following format:

{
"Ain0":"316.00",
"Ain1":"326.00",
"Ain2":"325.00",
"Ain3":"314.00",
"Ain4":"316.00",
"Ain5":"163.00",
"Ain6":"208.00",
"Ain7":"333.00",
"SYS_Heap":"25408",
"SYS_Time":"26"
}

And if connected, the server will also support changing the LED on/off state:

To turn it on, use the URL:
192.168.0.132:9701/?request=LedOn

And the off URL i:
192.168.0.132:9701/?request=LedOff

Adding server functionality should be obvious after reviewing the sketch.

Conclusions

Using a callback to process Web Server connection requests is far superior to polling. CPU bandwidth is only used in response to client connection events. And with the example framework provided in this post, you can still utilize many of the Arduino sensor libraries without modification.

With a Web Server application, you get the efficiency of the callback with the familiarity of the Arduino IDE.

I hope you find this information useful…

ESP8266 MQTT Publication & External Subscription

11th November 2015 by InternetOfHomeThings in All Posts, ESP8266 : An IoT System on Chip, Internet of Things | Comments Off

Publishing data produced by ESP8266 sensors to an MQTT broker server is a great option to making your IoT data visible to outside consumers with minimal consumption of precious MCU bandwidth. Simple, easy to implement and very light-weight.

What a great distribution system!

Simply publish your ESP8266 sensor data once, and many subscribers can consume the information, virtually at the same time. Sweet!

MQTT Basics

In case you are new to MQTT, here are a few basics. MQTT is the acronym for message queuing telemetry transport. Now that’s a mouthful! It is essentially a protocol that follows the publish/subscribe model to distribute information from one source to many users.

Here’s a few links for more MQTT information:

I’ve put together an ESP8266 MQTT demo project using an Arduino IDE sketch to publish data for consumption by subscribers using the MQTT protocol. Several options for consuming the data as a subscriber are also presented…

MQTT Broker/Server

For demo purposes, I wanted to use a free broker. After testing a few of the available options, I settled on using the Mosquitto MQTT server. While unsecured, meaning anyone who knows your “topic” can subscribe to the data, it works great for a proving out your design.

server: test.mosquitto.org

port: 1883

Arduino IDE Sketch

There is one library that needs to be added to the Arduino IDE to access the MQTT broker. It’s called “PubSubClient”. Here is how to install it…

Step 1: Open the Arduino IDE

Step 2: From the menu, select Sketch>Include Library>Manage Libraries…

Step 3: Enter “PubSubClient” in the search box.

Step 4: Click on the “Install” button

Now that was easy.

Great!

But there is one more thing to do after installing PubSubClient. I have found that the default maximum packet size (128 bytes) is insufficient. You should at least double it to 256 bytes.

How?

Just open the file “…/Arduino/libraries/pubSubClient/src/PubSubClient.h and change the following line:

From:

#define MQTT_MAX_PACKET_SIZE 128

To:

#define MQTT_MAX_PACKET_SIZE 256

Now lets talk about the sketch to test MQTT with the ESP8266…

The setup() function performs 3 basic tasks:

Setup()

setup serial port
connect to wifi
set MQTT broker/server

And loop() maintains the Wifi and MQTT connections, reads the sensors, and publishes the values to the MQTT broker.

Loop()

Reconnect to WiFi if disconnected
Reconnect to MQTT broker/server if disconnected
Read Sensor(s)
Publish Sensor data

Here is a break-down of the sketch. First, the objects must be identified

#include <ESP8266WiFi.h>
#include <PubSubClient.h>
#define wifi_ssid "Your SSID"
#define wifi_password Your Wifi password"
#define mqtt_server "test.mosquitto.org"
#define mqtt_user "not used in this example"
#define mqtt_password "not used in this example"
#define topic1 "t1"
#define topic2 "t2"
WiFiClient espClient;
PubSubClient client(espClient);

And then the sketch setup…

void setup() {
Serial.begin(115200);
setup_wifi();
client.setServer(mqtt_server, 1883);
}

A typical Wifi connect function:

void setup_wifi() {
delay(10);
// We start by connecting to a WiFi network
Serial.println();
Serial.print("Connecting to ");
Serial.println(wifi_ssid);
WiFi.begin(wifi_ssid, wifi_password);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}
Serial.println("");
Serial.println("WiFi connected");
Serial.println("IP address: ");
Serial.println(WiFi.localIP());
}

The loop() simply makes sure a connection to the MQTT is made, reads the sensors, and publishes the results. The publish rate is set at 2 seconds minimum.

void loop() {
if (!client.connected()) {
reconnect();
}
client.loop();
//2 seconds minimum between Read Sensors and Publish
long now = millis();
if (now - lastMsg > 2000) {
lastMsg = now;
//Read Sensors (simulated by increasing the values, range:0-90)
t1 = t1>90 ? 0 : ++t1;
t2 = t2>90 ? 0 : ++t2;
//Publish Values to MQTT broker
pubMQTT(topic1,t1);
pubMQTT(topic2,t2);
}
}

Just copy and paste this complete sketch as a starting template to test your own MQTT publishing with the ESP8266. Remember, you will need to set the WIFI ssid & password in the sketch to your access point values before running.

#include <ESP8266WiFi.h>
#include <PubSubClient.h>
#define wifi_ssid "YOURSSID"
#define wifi_password "YOURPASSWORD"
#define mqtt_server "test.mosquitto.org"
#define mqtt_user "your_username"
#define mqtt_password "your_password"
#define topic1 "t1"
#define topic2 "t2"
WiFiClient espClient;
PubSubClient client(espClient);
void setup() {
Serial.begin(115200);
setup_wifi();
client.setServer(mqtt_server, 1883);
}
void setup_wifi() {
delay(10);
// We start by connecting to a WiFi network
Serial.println();
Serial.print("Connecting to ");
Serial.println(wifi_ssid);
WiFi.begin(wifi_ssid, wifi_password);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}
Serial.println("");
Serial.println("WiFi connected");
Serial.println("IP address: ");
Serial.println(WiFi.localIP());
}
void reconnect() {
// Loop until we're reconnected
while (!client.connected()) {
Serial.print("Attempting MQTT connection...");
// Attempt to connect
if (client.connect("TestMQTT")) { //* See //NOTE below
Serial.println("connected");
} else {
Serial.print("failed, rc=");
Serial.print(client.state());
Serial.println(" try again in 5 seconds");
// Wait 5 seconds before retrying
delay(5000);
}
}
}
//NOTE: if a user/password is used for MQTT connection use:
//if(client.connect("TestMQTT", mqtt_user, mqtt_password)) {
void pubMQTT(String topic,float topic_val){
Serial.print("Newest topic " + topic + " value:");
Serial.println(String(topic_val).c_str());
client.publish(topic.c_str(), String(topic_val).c_str(), true);
}
//Variables used in loop()
long lastMsg = 0;
float t1 = 75.5;
float t2 = 50.5;
void loop() {
if (!client.connected()) {
reconnect();
}
client.loop();
//2 seconds minimum between Read Sensors and Publish
long now = millis();
if (now - lastMsg > 2000) {
lastMsg = now;
//Read Sensors (simulate by increasing the values, range:0-90)
t1 = t1>90 ? 0 : ++t1;
t2 = t2>90 ? 0 : ++t2;
//Publish Values to MQTT broker
pubMQTT(topic1,t1);
pubMQTT(topic2,t2);
}
}

Testing Tools

While there are many tools available, here are a few I have found that you can use to verify the ESP8266 is actually publishing the data.

How?

Simply subscribe to the topic and watch the information updates in real-time.

Google Chrome Application

First try using MQTTLens, an off-the-shelf tool to subscribe to the published data. You can get MQTTLens here. Although this app is installed via Google Chrome, it runs as standalone application. At this time, Chrome apps are supported using Windows, Mac and Linux operating systems.

This app provides basic functionality for testing MQTT publish/subscribe messages.

Once the app has been installed and started, a few configuration steps are needed to connect to our MQTT broker and subscribe to the ESP8266 published topics.

Just click on the configuration icon and fill in the information as shown below:

Once you are connected to the broker, the only thing left to do is subscribe to the data feed. For this example, two topics are published by the ESP8266; t1 and t2. Enter one of these and click on the subscribe button to monitor the data as it is published.

Notice you can change the number of messages displayed using the “-+” icons.

Web Page using JavaScript

MQTTLens is a great tool to get a quick look at your MQTT published data. But when you move from merely testing the data feed to developing a custom application, full control to manipulate and display the information is essential.
Fortunately, this is a simple task using only JavaScript and html.
I’ve developed the following html code to subscribe to the ESP8266 published data. This may be useful as a template on your own website. Simply open a file with this code in a web browser to test. Name this file as you please, using “.html” as the file name extension. In this example, I have named the file “mqtt_subscribe.html”.

<!DOCTYPE html>
<html>
<head>
<title>MQTT JavaScript Client Example</title>
</head>
<body>
<div><strong>Published ESP8266 topics via MQTT:</strong></div><br>
<div id="t1"></div>
<div id="t2"></div>
<script type="text/javascript" src="http://test.mosquitto.org/mosquitto.js"></script>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.2/jquery.min.js"></script>
<script>
// Create a client instance
client = new Mosquitto();
// Connect to MQTT broker
var url = "ws://test.mosquitto.org:8080/mqtt";
client.connect(url);
// Callback executed upon connection
client.onconnect = function(rc){
var topic = 't1';
client.subscribe('t1', 0);
client.subscribe('t2', 0);
};
//Callback executed upon disconnection
client.ondisconnect = function(rc){
client.connect(url);
};
//Callback executed upon receipt of MQTT message
client.onmessage = function(topic, payload, qos){
if(topic=="t1") {
$("#t1").html("Topic: t1 Value: " + payload);
}
if(topic=="t2") {
$("#t2").html("Topic: t2 Value: " + payload);
}
};
</script>
</body>
</html>

Just to keep things clean and simple, no css styling is included in this example. Here is a snapshot of the browser windows when this file is viewed:

Android

While I have not yet explored the development of a custom Android App to interface with an MQTT broker, it is certainly possible. What immediately comes to mind, in the spirit of App portable, would be to develop an Android Cordova App. This would allow you to reuse the above html code, without modification.
Here is my step-by-step guide to get started.
But if you are looking for an off-the-shelf MQTT test application, I recommend using MyMQTT . This is a free App available in Google Play Store. Once installed, there are only a few steps needed to subscribe to the ESP8266 example MQTT feed.
The App will startup with the following screen:

First set the app to connect with the broker we are using for this example. Click on “Settings” and enter the broker information as shown:

Then subscribe to the ESP8266 published topics (t1 and t2):

The dashboard will then display the topic data as it is published:

Iphone/iPad

MQTTInspector is the most popular App to test MQTT feeds from an iOS device. At $1.99, is not free, yet certainly low-cost. As a Windows, Android and Linux user, I cannot provide any additional information regarding this tool. It appears to be similar to other MQTT client test tools, and can be found on iTunes here.

In Conclusion

Here you have it. A simple guide to publishing topics from an ESP8266 device to an MQTT broker. All you need is a simple Arduino IDE sketch.

Once published, this data feed can be consumed cross-platform, on any device that supports MQTT. From this article, this includes but is certainly not limited to Windows, Mac and iOS. There are also Linux packages available to support MQTT clients.

As always, I hope you find this information useful…

Plugin Free WordPress Bootstrap

4th November 2015 by InternetOfHomeThings in All Posts, Web Coding Tips | 1 Comment

Bootstrap Plugins for WordPress are great for extending functionality and style without much coding. That is, until something breaks. There is nothing more frustrating than dealing with a website that is broken when you have not changed anything.

How could it happen?

I really don’t know. And while there are many great Bootstrap plugins for WordPress, I was depending on one of the most popular plugins (not mentioning names here) to support my responsive navigation menu. Worked great for almost a year, collapsing to the well-known 3-bar button for smartphones and small user browser windows-and expanding for larger screens. But then recently, it just stopped working.

It did not make sense. All I had done over the past few weeks was add a few posts to the site. That should have no effect on the navigation menu. Deactivating and reinstalling the plugin did not help. Yet the site needed to be fixed.

And fast!

After taking the “insanity” route for hours…trying the same things over and over with no improvement, I decided to delete the plugin and add Twitter Bootstrap manually to my WordPress site. While everything did not go smoothly at first, I did manage to restore my site’s bootstrap features. This time, without the non-transparent layer of a plugin.

Perhaps this guide may be of use to someone who encounters a similar situation…

Installing Bootstrap into a WordPress Site

First thing to do is to get the bootstrap files. I was unsuccessful in pointing to the CDN files so the actual files were downloaded and installed on the site. Go to getbootstrap.com and click on the “Download Bootstrap” button to get the needed files. The second download button looks like this:

You will also need the Jquery file, even if it is already installed as a WordPress plugin. As of this writing, Jquery is at version 2.1.4 and can be downloaded from this site.

It is important to copy the files to your WordPress theme folder as shown in the following table. It will not work if copied to a different directory.

	File Name	WordPress Folder
	bootstrap.min.css	/css
	bootstrap-theme.min.css	/css
	bootstrap.min.js	/js
	jquery-2.1.4.min.js	/js

The next thing to do is to modify your themes functions.php file. This will make bootstrap functional within your WordPress theme. unfortunately, since functions.php is unique for each theme, the modifications must be made separately for each theme used.

Let’s first include the bootstrap style sheets when WordPress starts up. That is simply a matter of adding the following two “wp_register_style” and “wp”enqueue_style” lines to the function <your theme name>_of_styles():

function <your theme name>_of_styles() {
wp_register_style( 'bootstrap', get_template_directory_uri() . '/css/bootstrap.min.css' );
wp_register_style( 'bootstraptheme', get_template_directory_uri() . '/css/bootstrap-theme.min.css' );
.
.
.
wp_enqueue_style( 'bootstrap' );
wp_enqueue_style( 'bootstraptheme' );
.
.
.
}

Similarly, the Jquery files need to be included when WordPress starts up. This, however needs to be handled differently. That is because these files need to be introduced just before the bootstrap navigation bar html code is added, not when WordPress starts up.

This requires the addition of a new function to the functions.php file. Since the code is php, it simply echos the html script tags used to include the files. It is important for the jquery file to be listed before bootstrap:

function ld_bootstrap_js() {
echo '<script src="/js/jquery-2.1.4.min.js"></script>';
echo '<script src="/js/bootstrap.min.js"></script>';
}

As noted, this function needs to be called just before the navigation bar menu is introduced. This requires a change to one more theme file, “header.php”. Add the call just after the stylesheet in this file:

.
.
<link rel="stylesheet" type="text/css" media="all" href="<?php echo get_stylesheet_uri(); ?>" />
<?php ld_bootstrap_js(); ?>
.
.
</head>

That’s it. Not really that difficult once you know how. With these changes, your WordPress website will support a responsive bootstrap navigation bar.

In Closing

While you may never have any problems using a bootstrap plugin, you might want to keep this information handy, just in case it fails you. This will bring your site back on line quickly. Independent of a third-party plug-in.

Archives for November 2015