All Posts

Triple Server Update – Part 2: Web Configuration

11th January 2016 by InternetOfHomeThings in All Posts, Arduino, ESP8266 : An IoT System on Chip, Internet of Things | 8 Comments

Have you ever moved your ESP8266 system from one Wifi location to another? Or perhaps made an extra one for your friend? In many cases, the initial setup requires a change to the sketch, to adjust the Wifi parameters to match the new access point—that is—unless the local settings are stored in the ESP8266 EEPROM.

What’s great about the EEPROM is that once updated, the configuration is non-volatile, That is, the values remain the same after the system power is removed and re-applied.

This post provides a method of setting and saving parameters likely to change with WiFi location, without changing the installed firmware sketch.

Once you’ve got you IoT device finished, you really should not have to modify the software again. That is, unless enhancements are made or bugs corrected.

Configurable Parameters

I started out with intentions on building a reusable configuration template. This would include the parameters most likely to change with system location and cloud-based servers used.

Using the Triple Server sketch I had developed recently as a test case, the configurable network and MQTT parameters  were first identified. Then, these parameters were mapped to the ESP8266 EEPROM.

For the project presented in this post, the following parameters were singled out to be configurable:

  • WIFI SSID
  • WIFI Password
  • IP – Static IP assigned to ESP8266 when used in Station mode
  • NM – Network Mask
  • GW – Gateway (Wifi Router IP)
  • AP – Static IP assigned to ESP8266 when used in Access Point mode
  • Web Server network port
  • MQTT Broker
  • MQTT Username
  • MQTT Password
  • MQTT Request Topic
  • MQTT Reply Topic
  • Serial Port baud
  • Arduino Server Enable

EEPROM Memory Map

The ESP8266 EEPROM configuration space is typically the first 512 bytes of memory. And here is how I allocated the configurable parameters to this EEPROM.

Parameter EEPROM Default Size(bytes)
WIFI SSID 0x10 YOURWIFISSID 32
WIFI Password 0x30 YOURWIFIPASSWORD 16
IP0 - IP Byte 1 0x40 192 1
IP1 - IP Byte 2 0x41 168 1
IP2 - IP Byte 3 0x42 0 1
IP3 - IP Byte 4 0x43 132 1
NM0 - Network Mask 0x44 255 1
NM1 - Network Mask 0x45 255 1
NM2 - Network Mask 0x46 255 1
NM3 - Network Mask 0x47 0 1
GW0 - Gateway 0x48 192 1
GW1 - Gateway 0x49 168 1
GW2 - Gateway 0x4A 0 1
GW3 - Gateway 0x4B 1 1
AP0 = Access point 0x4C 192 1
AP1 = Access point ox4D 168 1
AP2 = Access point ox4E 4 1
AP3 = Access point ox4F 1 1
Server Port 0x50 9701 2
MQTT Broker 0x60 test.mosquitto.org 32
MQTT User Name 0x80 mqtt_username 32
MQTT Password 0xA0 mqtt_password 32
MQTT Request Topic 0xC0 MyMqttSvrRqst 32
MQTT Reply Topic 0xE0 MyMqttSvrRply 32
Serial Baud 0x100 1200 3
Arduino Server Enable 0x108 1

Note that while 32 bytes are allocated for the character array parameters, this size is the maximum; it is not required to use this entire space. But each character array must be terminated in EEPROM with a NULL (0x00) character.

Webpage Structure

The next thing needed was a way to construct a webpage for viewing, altering, and saving these configuration parameters. All of the page rendering is done on the client side browser. With this in mind, the web page was developed using only html, css for styling, and Javascript.

In order to conserve program memory, the webpage source is saved into flash memory. Two character arrays were used, one for the html and JavaScript and a separate one for the css styling.

Next time, I am planning to separate out the JavaScript as well for cleaner partitioning. This can easily be done from the example code provided here and is highly encouraged as an exercise.

A token was placed at the top of the html character array for insertion of the css styling. Likewise, tokens have been placed in the Javascript jquery section for each configurable parameter. These tokens are replaced with the values stored in EEPROM prior to sending the completed webpage string to the client browser for rendering.

In this example:

 
 
  1.     $("#ssid").val("set_ssid");
  2.      $("#password").val("set_pass");
  3.      $("#ip_0").val("set_ip0");
  4.      $("#ip_1").val("set_ip1");
  5.      $("#ip_2").val("set_ip2");
  6.      $("#ip_3").val("set_ip3");
  7. .
  8. .

set_ssid is the first parameter token, which is replaced with the value read from EEPROM.

Here is what the rendered configuration page looks like:

config_page

The parameters are structured in html as a form with the two buttons shown. The save button is of type “submit” which sends all of the parameters back to the ESP8266 as a GET request for saving to the EEPROM.

But this merely saves the values back to EEPROM. Since the startup setup() function has already been run, the saved parameters do not become effect until an ESP8266 reset is performed. That is accomplished either with a power cycle or clicking on the “RESET ESP8266” button on this configuration page.

ESP8266 Webpage Request/Reply Code

Code was added to the ESP8266 URL request processing algorithm to handle web page requests. This is performed in the sketch’s Server_ProcessRequest function. For webpage requests, a slightly different URL format is used than requests to read a sensor.

The familiar example is the URL to get the sensor values:

192.168.0.132:9701/?request=GetSensors

When we want the ESP8266 to display the configuration page, enter:

192.168.0.132:9701/config

Note that there is no “?” for webpage requests. In order to parse the URL correctly for this case, the SdkWebServer_parse_url_params() function required modification.

Note that while the sketch provided in this example only supports the configuration page, a review of the code should make it obvious how to expand this to support your own custom webpages.

 
 
  1. // --------------------------------------------------------------
  2. // Serving Web Pages
  3. // --------------------------------------------------------------
  4. //
  5. // ------------- Load Config Page -------------------------------
  6. if(os_strcmp(pURL_Param->pParam[0], "config")==0) {
  7. GetEepromCfg(&WebPage);
  8.     SdkWebServer_senddata_html(ptrespconn, true, 
  9. (char *) WebPage.c_str());
  10. }
  11. // -------------- Save config or Reset ESP8266 -------------------
  12. if(os_strcmp(pURL_Param->pParam[0], "ssid")==0) {
  13. if(os_strcmp(pURL_Param->pParVal[0], "reset")==0){
  14. // ------------- Reset ESP8266 -------------------------------
  15.      WebPage = reinterpret_cast<const __FlashStringHelper *>
  16. (PAGE_WaitAndReset);
  17.         SdkWebServer_senddata_html(ptrespconn, true, 
  18. (char *) WebPage.c_str());
  19.         os_timer_arm(&ResetEspTimer, 3000, false); 
  20. }
  21.     else {
  22. // --- Save config to EEPROM and reload config page ------
  23. SetEepromCfg((void *)pURL_Param);        
  24.         WebPage = reinterpret_cast<const __FlashStringHelper *>
  25. (PAGE_WaitAndReload); //reload config page
  26.         SdkWebServer_senddata_html(ptrespconn, true, (char *) 
  27. WebPage.c_str());
  28. }
  29. }

The first parameter parsed is used to determine the webpage request type. This example only processes 2 possibilities. If the first parameter is “config”, the configuration page is rendered.

That is accomplished by building the webpage string. First using the template, and then substituting values for the css and parameter tokens. This is all done in the GetEepromCfg() function:

 
 
  1. void GetEepromCfg(String *page)
  2. {
  3.     *page = reinterpret_cast<const __FlashStringHelper *>
  4. (PAGE_NetCfg);
  5.     String css = reinterpret_cast<const __FlashStringHelper *>
  6. (PAGE_Style_css);
  7.     page->replace("ADDSTYLE",css); 
  8.     css = "";
  9.     
  10.     SetCfgPageWithEepromVal(page, "set_ssid", EEPROM_WIFISSID, 
  11. EEPROM_CHR);
  12.     SetCfgPageWithEepromVal(page, "set_pass", EEPROM_WIFIPASS, 
  13. EEPROM_CHR);
  14.     SetCfgPageWithEepromVal(page, "set_ip0", EEPROM_WIFI_IP0, 
  15. EEPROM_INT);
  16.     SetCfgPageWithEepromVal(page, "set_ip1", EEPROM_WIFI_IP1, 
  17. EEPROM_INT);
  18. .
  19. .

But if the first parameter is ssid, the value of the parameter must first be read in order to determine what action is requested. That is because the method of returning values to the ESP8266 code for both of the WebPage button is by submitting the form.

When the reset button is clicked, the ssid value is changed to “reset” before submitting. In that case (above code), as message is displayed indicating the ESP8266 is resetting. Note that a 3-second delay is implemented using a timer before the ESP8266 resets itself. That is necessary to allow time for the reloading of the configuration webpage to complete prior to the reset.

And in the case that the Save button is clicked, the ssid value from the webpage form input is not modified (set to “reset”) and thus, the values are simply saved to EEPROM.

Webpage Expansion

Again, all you need to do is add a suffix to the ESP8266 access url and add your processing code to expand this webpage server:

192.168.0.132:9701/YOURCUSTOMPAGE

 
 
  1. if(os_strcmp(pURL_Param->pParam[0], "YOURCUSTOMPAGE")==0) {
  2.     //Do whatever you want here to handle this request
  3. }

In case you missed part 1, the same code is used for this part 2. It is available on GitHub here.

In Closing

That’s it. A working framework for saving configuration parameters to ESP8266 EEPROM. And as an added bonus, you have a structure to expand this to render webpages of your own choosing. To control or monitor your own IoT things visually.

Hope you find this information as useful to you as it is for me…

Credits

Developing this example required research and reuse of material found in public domain sources. Special credit must be given to the following source which inspired my effort presented here. It also may be of use to others…

MQTT and Easy Web Config – by “The Godfather” – Posted October 22, 2015

Loading

Share This:
FacebooktwitterredditpinterestlinkedintumblrFacebooktwitterredditpinterestlinkedintumblr

Triple Server Update – Part 1: Serving Arduino

9th January 2016 by InternetOfHomeThings in All Posts, Arduino, ESP8266 : An IoT System on Chip, Internet of Things | 2 Comments

httpmqtt2arduino

Happy New Year! I’m picking things up right where they ended last year…

There were some basic enhancements needed to my Triple Protocol Server. But in the process of implementing the updates, the limitations of the ESP8266 resources were reached. To make room for the new features, something had to go.

In this case, the CoAP server was eliminated. The choice was obvious. This protocol consumed a large chunk of the available heap, leaving little room for program execution. As result, the system became unstable. CoAP also does not lend itself easily to web browser based clients.

So with that gone, a valuable new feature was added. A light-weight modification that did not significantly burden the system. Something that Arduino users should find extremely useful.

What is it?

An Arduino Serial Port Web Server

What? We already have access from the Arduino using the AT command set. “So what’s so special about this?”, you say.

Well…this implementation removes most of the web server tasks from the Arduino and puts it in the ESP8266. The only thing the Arduino needs to do is read small strings (requests) from the ESP8266 and return a small string back. This frees the Arduino to concentrate on sensor readings and control functions.

This balances the load on the two micro-controllers, reducing the load on the Arduino while fully leveraging the available power of the ESP8266 as a web server.

So how does this work?

It’s really quite simple. The ESP8266 web server is already in place. All that is needed is a few new URL-based (or MQTT topic payload) commands to provide web-based access to the Arduino resources.

The New Arduino Commands

The commands are parsed and decoded by the ESP8266. The command is reduced to a small string that is passed to the Arduino for action with it’s hardware resources.

URL Suffix ESP8266 to Arduino Arduino Reply
Set Digital Channel 04 HI /?arduino=SetDigital&chan=04&state=1 Arduino_SD041 Digital Channel 04 is HI
Get Digital Channel 04 /?arduino=GetDigital&chan=04 Arduino_GD04 Digital Channel 04 is HI
Get Analog Channel 04 /?arduino=GetAnalog&chan=04 Arduino_GA04 Analog Channel 04 is 2.234

The Arduino returns a reply to the ESP8266, which, in turn, is returned to the http or mqtt client.

This structure is similar to the slash (/) delineated bridge interface used with the Arduino Yun to communicate between it’s Arduino and Linux processors. Instead of using a pricey Arduino Yun, you can use any model Arduino along with the very inexpensive ESP8266. In this test case, I used a $7 Arduino nano clone.

The example presented here only includes the basic digital and analog access to the Arduino. The digital interface is obvious, setting or reading the state of each pin. The analog channel reads returns a voltage between 0 and 5 volts, based on the 10-bit Arduino ADC channel reading.

From this command set, it should be obvious how to expand it to include bus oriented sensors such as 1-wire temperature and i2c connected devices.

ESP8266 Sketch

The ESP8266 Arduino IDE sketch is based upon my triple server sketch presented in a prior post. The default parameters are defined at the top of the sketch in the section titled “Initial EEPROM Values”. These can be revised to match your network setting as described in Part 2 of this post, and in the sketch’s github repository.

Additional code was needed to support the Arduino requests. The commands received are repacked into a small string to be sent to the Arduino via the ESP8266 serial port. The server code to process the three recognized requests (Set Digital, GetDigital and Get Analog) shown here can easily be expanded for addition requests:

 
 
  1. // -----------------------------------------------------------------------
  2. // Serving Arduino via serial port
  3. // -----------------------------------------------------------------------
  4. if(os_strcmp(pURL_Param->pParam[0], "arduino")==0) {
  5. //-------------- Request = SetDigital ---------------------
  6. if(os_strcmp(pURL_Param->pParVal[0], "SetDigital")==0){
  7. if(os_strcmp(pURL_Param->pParam[1], "chan")==0) {
  8.           ArduinoRequest = "Arduino_SD";
  9.            ArduinoRequest += pURL_Param->pParVal[1];
  10.         }
  11.        else {
  12.             ArduinoRequest = "Invalid Request";  
  13.         }
  14.         if((os_strcmp(pURL_Param->pParam[2], "state")==0)&&(os_strcmp(
  15. ArduinoRequest.c_str(), "Invalid Request")!=0)) {
  16.             ArduinoRequest += pURL_Param->pParVal[2];
  17.         }
  18.         else {
  19.             ArduinoRequest = "Invalid Request";  
  20.         }
  21. }
  22.     //-------------- Request = GetDigital ----------------------
  23.     else if(os_strcmp(pURL_Param->pParVal[0], "GetDigital")==0){
  24. if(os_strcmp(pURL_Param->pParam[1], "chan")==0) {
  25.          ArduinoRequest = "Arduino_GD";
  26.             ArduinoRequest += pURL_Param->pParVal[1];
  27.      }
  28.       else {
  29.             ArduinoRequest = "Invalid Request";  
  30.      }
  31.    }
  32.     //-------------- Request = GetAnalog ----------------------
  33.     else if(os_strcmp(pURL_Param->pParVal[0], "GetAnalog")==0){
  34.      if(os_strcmp(pURL_Param->pParam[1], "chan")==0) {
  35.          ArduinoRequest = "Arduino_GA";
  36.             ArduinoRequest += pURL_Param->pParVal[1];
  37.         }
  38.         else {
  39.             ArduinoRequest = "Invalid Request";  
  40.         }
  41.    }
  42.    //-------------- Request is not recognized -----------------
  43.    else {
  44.     ArduinoRequest = "Invalid Request";  
  45.    }
  46.    //-------- Execute valid request & get reply string --------
  47.    if(os_strcmp(ArduinoRequest.c_str(), "Invalid Request")==0) {
  48.      payld = ArduinoRequest;
  49.    }
  50.    else {
  51.     rparam.request = ARDUINO_REQUEST;
  52.        rparam.requestval = 0;
  53.        Server_ExecuteRequest(servertype, rparam, &payld, ArduinoRequest);
  54.   }
  55.   Server_SendReply(servertype, REPLY_TEXT, payld);
  56. }

Arduino Sketch

A simple Arduino sketch is provided here  as an example of how to receive and process requests from the ESP8266. Using the low-end Arduino nano, this example uses a software serial port (Arduino digital pins 10 and 11) to communicate with the ESP8266.

Due to the limited bandwidth for reliable operation with this software port, the baud was reduced to 1200 baud. If a dedicated hardware serial port is used, such as found with the Arduino mega, the baud can be increased considerably.

The example provided is fully functional, yet simple enough to understand and build upon with even modest programming skills.

Part 2: Web-based ESP8266 Configuration

So there you have it, an ESP8266 web server sketch that supports Arduino communication using either MQTT or url-based http protocols. But there is more to this update. I’ve also added web-browser configuration of the Network and MQTT settings.

Read on here for the details…

Loading

Share This:
FacebooktwitterredditpinterestlinkedintumblrFacebooktwitterredditpinterestlinkedintumblr

ESP8266 Triple Protocol Server

19th December 2015 by InternetOfHomeThings in All Posts, ESP8266 : An IoT System on Chip, Internet of Things | 12 Comments

http-mqtt-coap

In my previous post, a comparison was made between ESP8266 Http and MQTT hosted servers. But the evaluation was not complete…it did not include CoAP. So I decided to add CoAP to the ESP8266 server sketch and revisit the performance.

Like MQTT and Http, CoAP is deployed in IoT things for M2M communication. But what is it?

Simply stated, CoAP, or Constrained Application Protocol is a web communication structure designed for use with limited resource Internet Things. For some light reading (zzzz) with all the details, refer to RFC 7252 upon which it is based. This protocol is designed to be light-weight, using connection-less UDP over IP.

But this is not a CoAP tutorial.

The purpose of this article is to compare the performance of CoAP with my previous comparison of MQTT to Http. To keep things on a level playing field, CoAP was evaluated using the same server get command that we used to test MQTT and Http servers.

Building on the initial design, the dual server sketch was simply updated to include CoAP. This example now supports three IoT communication protocols: CoAP, Mqtt and http. And just as before, there are two build options for the http server; one using the Arduino Wifi library for ESP8266 and the other using the EspressIf SDK API.

And as is the case with many of my first-time endeavors, an unexpected finding was uncovered…

Triple Protocol Server Sketch

Since this was my first attempt with a CoAP implementation, I started out searching for a suitable example as a basis for the sketch. Fortunately, there was one available, microcoap. That library is used in my sketch. It turned out to be a bit more than a trivial library addition to the sketch.

In addition to the sketch changes, the CoAP protocol required 3 new files;

 
 
  1. coap.h       - Header file for coap library
  2. coap.c       - Coap library code
  3. endpoints.c  - CoAP interface for this example sketch

The coap.h,c files were added to the Arduino library while the endpoints.c is added to the sketch folder.

All of the typedefs and #defines were moved from the sketch to a separate header file; sketch.h.

The CoAP communication relies on UDP over IP. While a callback is preferred, this CoAP implementation requires polling in the loop() function to check for incoming CoAP packets. This added step to the 3-protocol server loop() function is implemented in the function “ProcessCOAPMsg()”.

 
 
  1. void loop() {
  2.     util_startWIFI();                    // Connect wifi if connection dropped
  3.     #if MQTT_SVR_ENABLE==1
  4.     MqttServer_Processor();              // Service MQTT
  5.     #endif
  6.     #if SVR_TYPE==SVR_HTTP_LIB
  7.     ArduinoWebServer_Processor();        // Service Web Server
  8.     #endif
  9.     
  10.     ProcessCOAPMsg();                    // Service CoAP Messages
  11.     ReadSensors(2500);                   // Read 1 sensor every 2.5 seconds or longer
  12. }

But there is one way to reduce the loop() tasks. Fortunately, we have the option of  using the SDK for the web server, which uses callbacks instead of polling in loop(). As you can see, the pre-compiler directive only includes the web server in loop() when the LIB option is selected. The callback is executed anytime an external http connection is made, eliminating the need to poll in loop().

In order to add some functionality to the original server, a fourth command was added to the 3 included with the dual mqtt and http server:

  1. Turn LED On
  2. Turn LED Off
  3. Get Sensor Values
  4. Blink LED n times

The added command in this example serves two new purposes:

First, this was an opportunity to have a command example that also required a parameter, In this case, the “blink” command needs a parameter to identify the number of blinks to perform.

Secondly, this command needs to be run asynchronous with the server reply. So that the reply is complete while the LED blinks continued. This would also be the case for any command that may require significant time to complete. Requiring a server reply before the command execution finishes.

In this case, a non-blocking function is launched using a timer. The timer receives a parameter identifying how many times remain in the blink sequence, as well as the state (on or off) to set the LED. Each time the timer is called (with a 500 ms delay to provide time for the observer to see the LED state), the LED changes state and the timer is called again. This continues until the commanded number of blinks finishes.

 
 
  1. ********************************************************
  2.  * Callback for Blink timer
  3.  * Function: BlinkCallback(int *pArg)
  4.  * 
  5.  * Parameter    Description
  6.  * ---------    -----------------------------------------
  7.  * *pArg        int - number of times to blink LED
  8.  * *pArg+1      int - set LED state (0=off,1=on) 
  9.  * *pArg+2      int - set to 1 first time to print header 
  10.  * return       no return value
  11.  ********************************************************/
  12. void BlinkCallback(int *pArg) {
  13.     int i;
  14.     int nblinks,ledstate,start;
  15.     
  16.     nblinks = *(int *)pArg;                           // Number of LED blinks
  17.     ledstate = *(int *)(pArg+1);                      // LED state to program
  18.     start = *(int *)(pArg+2);                         // Set to 1 first time this is 
  19. called
  20.     if(start == 1)
  21.     {
  22.         Serial.print("Blink countdown:");             // LED countdown header to serial port
  23.         blinkarg[2] = 0;                              // Do not print header next time
  24.     }
  25.     if(ledstate==1)
  26.     {
  27.          Serial.print( nblinks);                      // Current Blink number
  28.          if(nblinks>1) 
  29.          {
  30.              Serial.print(".");                       // Blink countdown separator
  31.          }
  32.          else
  33.          {
  34.              Serial.println("...");                   // Separator after last blink
  35.          }
  36.          digitalWrite(LED_IND, HIGH);                 // Set LED On
  37.          blinkarg[1] = 0;                             // Set LED off next time
  38.          os_timer_arm(&BlinkLedTimer, 500, false);    // Execute this function 1 time 
  39. after 500 ms
  40.     }
  41.     else
  42.     {
  43.          digitalWrite(LED_IND, LOW);
  44.          blinkarg[1] = 1;                             // start with led on cmd
  45.          if(--nblinks!=0) {                           // Execute until blinks remaining  is 0
  46.             --blinkarg[0];                            // Decrement remaining blinks 
  47.             os_timer_arm(&BlinkLedTimer, 500, false); // Execute this function 1 time 
  48. after 500 ms
  49.          }     
  50.     }
  51. }

Ready to download this Arduino IDE server code?

You can get the triple CoAP, MQTT & HTTP Web Server sketch and associated files here.

CoAP Server

The commands the CoAP server responds to is defined in the file “endpoints.c” All the methods for the CoAP server are defined in the “endpoints array:

 
 
  1. //////////////////////////////////////////////////////////////////////////
  2. // Define all CoAP Methods for this Server
  3. //////////////////////////////////////////////////////////////////////////
  4. const coap_endpoint_t endpoints[] =
  5. {
  6.     {COAP_METHOD_GET, handle_get_well_known_core, &path_well_known_core, "ct=40"},
  7.     {COAP_METHOD_GET, handle_get_light, &path_light, "ct=0"},
  8.     {COAP_METHOD_GET, handle_get_light_blink, &path_light_blink, "ct=0"},
  9.     {COAP_METHOD_GET, handle_get_request, &path_request, "ct=0"},
  10.     {COAP_METHOD_PUT, handle_put_request, &path_request, NULL},
  11.     {COAP_METHOD_PUT, handle_put_light, &path_light, NULL},
  12.     {COAP_METHOD_PUT, handle_put_light_blink, &path_light_blink, NULL},
  13.     {(coap_method_t)0, NULL, NULL, NULL}
  14. };
  15. The function names, starting with "handle_", identify the callbacks executed when a request is received corresponding

The function names, beginning with “handle_”, identify the callbacks executed when a request is received corresponding the service path. And the service paths which are returned from CoAP “Discovery” requests are defined in this file for each method:

 
 
  1. //////////////////////////////////////////////////////////////////////////
  2. // Define URI path for all CoAP Methods for this Server
  3. //////////////////////////////////////////////////////////////////////////
  4. static const coap_endpoint_path_t path_well_known_core = {2, {".well-known", "core"}};
  5. static const coap_endpoint_path_t path_light_blink = {1, {"light_blink"}};
  6. static const coap_endpoint_path_t path_request = {1, {"request"}};
  7. static const coap_endpoint_path_t path_light = {1, {"light"}};

The list can be expanded to meet your own custom server requirements. In order to test the CoAP server in operation, a CoAP client is needed.

CoAP Client

My initial plan was to expand the html/Javascript test page used to compare Mqtt with http servers to include CoAP.  But every implementation found required an http to CoAP proxy. This proved to be more challenging to implement than I had planned. At this time, I was unsuccessful. This may come later. But for now, for this evaluation, I settled for a browser plugin. The choices were limited. That is to say, only one was available.

It is called Copper (Cu) CoAP user-agent. At this time, it is only supported with the Mozilla Firefox browser.

Installation is simple:

First, if not already on your computer,  you need to install the Mozilla Firefox browser.

Then, using this browser open the URL “https://addons.mozilla.org/en-US/firefox/addon/copper-270430/” and click on the “+ Add to Firefox” button.

So how do we use this client?

Fortunately, the interface to the ESP8266 CoAP server is easy to access with this plugin. Once added to Mozilla Firefox, start the ESP8266 with the triple server sketch running and enter the following into the browser URL field:

coap://192.168.0.132:5683/

Of course, if you have your own domain pointing to your broadband connection with your router configured to forward port 5683 to your ESP8266, you would enter:

coap://<your domain>:5683/

This will open your interactive CoAP interface tool. It should appear initially similar to this:

firefox1

A nice feature of the CoAP protocol is discovery. Click on the Discover button for the CoAP server (our ESP8266 device) to report back the services it will support:

firefox2

The command/response mechanism is straightforward. Click on one of the 3 services shown, enter the “Outgoing” payload, and click on the PUT button. The ESP8266 CoAP server reply will appear at the “Incoming” payload tab. Here are all the services & payloads supported.

Service Payload Server Reply
Turn LED on light 1 1
Turn LED off light 0 0
Blink LED 3 times light_blink 3 3
Turn LED on request /?request=LedOn LED is now ON
Turn LED off request /?request=LedOff LED is now OFF
Blink LED 6 times request /?request=BlinkLed&nblink=6 LED is now blinking 6 times.
Get Sensor values request /?request=GetSensors JSON Sensor values string

The “request” service supports identical commands for all three server types. Again, to keep things consistent, the “GetSensors” request will be used for this evaluation, the same command that was used for the dual server test.

In that case, select the “request” service, enter /?request=GetSensors in the Outgoing payload window, and click on the “PUT” button.

firefox3

The response (Incoming payload) will be the same as the reply received using the MQTT or HTTP servers:

firefox4

Response Delay Evaluation

For the MQTT and HTTP servers, a simple web page, using only html and JavaScript was used to measure the delay from the time a request was sent to the time the reply was received. The web page file is called “mqtt_server.html”; it is on GitHub here.

And the CoAP server evaluation used the Firefox Modzilla browser plug-in.

MQTT and HTPP Response Time

With mqtt_server.html open, the time starts when the button is clicked to send a request to the server. When the reply is received, the end time-stamp was captured. But what I found was that this end stamp was not saved immediately, resulting in inaccurate readings. This is likely due to the asynchronous nature of JavaScript. To correct this, a 1 second callback was added from the time the reply time-stamp was captured to the time the lapse time was calculated. This resulted in repeatable results which appear to accurately report the response delay.

CoAP Response Time

The response time is reported directly from the Firefox Mozilla tool when a request was sent by clicking on the “Put button. The time is referenced as RTT in the Mozilla windows in the format:

<IP or domain name>:5683 (RTT: 26 ms)

In this example, the response time was 26 ms.

Results

Ten requests were made for each of the three server setups. The average of the ten readings are shown in this table:

CoAP MQTT HTTP: WiFi Library HTTP: SDK API
Sample 1 23 361 273 54
Sample 2 23 322 488 96
Sample 3 125 315 505 39
Sample 4 271 360 537 58
Sample 5 27 409 482 63
Sample 6 99 423 559 94
Sample 7 12 345 488 59
Sample 8 11 333 455 102
Sample 9 10 325 434 36
Sample 10 103 325 329 21
Average 70 352 455 62

The time is measured in milliseconds. From this experiment, the HTTP SDK API and CoAP servers offer significantly less latency than either the HTTP Wifi Library or MQTT. From this, we can conclude that callbacks that do not use the sketch loop() function are more efficient. And the UDP over IP result in faster replies than TCP connections when polling is used, but reply at comparable speeds when HTTP callbacks are used.

The Unexpected Discovery

As I mentioned in the beginning, an unexpected discovery was made in the course of this evaluation. Unfortunately, it is not a good thing, but it must be told. Hopefully, this may allow someone to avoid the pitfall. This finding was made using Arduino IDE version 1.6.5. Later versions might have corrected the issue.

This has to do with the HTTP Wifi Library based server. What I have found is that with every server request made, the system heap shrinks. This is returned as the “SYS_Heap” value (bytes) from a “GetSensors” request. The bytes do not return to the heap for a while, sometimes minutes later, sometimes, it appears never to be relinquished.

What does this mean?

Well, if requests are made often enough, the heap will shrink to the point that the ESP8266 resets, sometimes without recovery (crashes). None of the other server types presented here exhibit any heap shrinkage from receiving and replying to request.

I checked this with some of my earlier Arduino IDE sketches. Regretfully, they all exhibit this same heap shrinkage when server requests are made.

Recommendation: Avoid using the HTTP Wifi Library based server. Instead, use the EspressIf SDK API, similar to what has been presented in this sketch.

Especially if you have experienced unexplained ESP8266 crashes or resets when your sketch runs for an extended period of time. The depletion of the heap could definitely be the root cause.

In Closing

As you can see from the results, CoAP performs well as compared to polling MQTT and HTTP servers.

CoAP is indeed a viable option as an IoT M2M communication protocol, This method provides a solution that is lightweight, fast and there are many implementation options available. Unfortunate, these options do not directly include client side JavaScript webpages. A proxy is needed to translate the UDP CoAP to http. This proxy would undoubtedly slow down the message transfers, but this alone is unlikely to be significant enough to discount CoAP.

And as “lightweight” as CoAP is intrinsically, the implementation code is more complex and lengthy than any other server.

I hope this example provides anyone interested with a framework to pursue ESP8266 CoAP communications further. My next step will to create custom CoAP clients. For my needs, two platforms will be useful. An Android based smartphone App and a browser solution accessible from any browser.

How would you use best use CoAP with your ESP8266 IoT thing?

Loading

Share This:
FacebooktwitterredditpinterestlinkedintumblrFacebooktwitterredditpinterestlinkedintumblr

ESP8266 HTTP vs MQTT Servers

2nd December 2015 by InternetOfHomeThings in All Posts, ESP8266 : An IoT System on Chip, Internet of Things | 10 Comments

http-mqtt

The ability to communicate with your IoT things — to send commands and receive responses is an essential pillar of your “connected” system. You need a network link to control things and to read sensors. Two of the most common Internet to machine communication protocols used with the ESP8266 and other MCUs are http and MQTT.

There are advantages and drawbacks using either one of these methods.

So which one is better?

For an assessment, I created a simple Arduino IDE sketch with just two metrics in mind for the evaluation. First was the simplicity of the application code. How clean was the framework needed to implement the server?

The other concern was latency. That is, how long  is required from the time of a client request to server reply? I was curious to measure this time. Is the delay magnitude in the order of milliseconds, or seconds?

The Sketch

I started out with the intent on creating two sketches; one of an http server and one for MQTT. But what became apparent was that other than the code needed to connect, read and write to the protocol, the software was almost identical. So for simplicity and efficiency, the http and MQTT servers were combined into a single sketch.

This also made the evaluation easier-there was no need to reload the firmware to switch protocols.

The typical Arduino IDE based http server requires the use of two class objects. These objects are defined in WiFi.h for Arduino based systems and the ESP8266WiFi.h library for the ESP8266.

  • WiFiServer
  • WiFiClient

The major shortcoming with the Arduino IDE library is the need in the user sketch to poll for new connections in the loop(). It would be far more efficient to register a callback function that is executed upon an external client connection. Like the EspressIf SDK Iot_demo example web server.

This constraint alone would be sufficient justification to use the Espressif SDK over the Arduino IDE for all but the simplest Web Server application.

Fortunately, however, there is a way to use the SDK callbacks with the Arduino IDE. Here is how it is done. That is just what we will use for comparing the http and MQTT servers. The sketch for this test is just a reuse of that sketch with the added code needed to support MQTT. The Dual MQTT & HTTP Web Server code is here.

Code Assessment

When looking at the code structure for the MQTT broker vs the Arduino Wifi library vs the SDK API, the most efficient design was the SDK API. While all 3 methods worked well, the SDK design uses a callback to service client connections without any code needed in the loop() function.

The Arduino Wifi library requires polling for connections each iteration of loop().

And sure, the MQTT broker server, like the SDK API, also uses callbacks. But it still requires the execution of “client.loop()” in the sketch’s loop() function. This consumes bandwidth which undoubtedly will add latency to other tasks added to loop().

If loop() latency becomes an issue in your sketch, use of the SDK API web server callbacks will minimize the delays.

Response Delay Evaluation

A simple web page, using only html and JavaScript was used to measure the delay from the time a request was sent to the time the reply was received. The web page file is called “mqtt_server.html”; it is on GitHub here.

The time starts when the button is clicked to send a request to the server. When the reply is received, the end time-stamp was captured. But what I found was that this end stamp was not saved immediately, resulting in inaccurate readings. This is likely due to the asynchronous nature of JavaScript. To correct this, a 1 second callback was added from the time the reply time-stamp was captured to the time the lapse time was calculated. This resulted in repeatable results which appear to accurately report the response delay.

Twelve requests were made for each of the three server setups. The average of the twelve, and the average of ten with the high and low readings removed are shown in this table:

HTTP: WiFi Library HTTP: SDK API MQTT
Sample 1 429 89 341
Sample 2 457 32 362
Sample 3 367 30 338
Sample 4 441 85 383
Sample 5 365 42 375
Sample 6 400 28 417
Sample 7 465 26 343
Sample 8 489 445 333
Sample 9 620 42 566
Sample 10 358 32 339
Sample 11 731 212 325
Sample 12 432 29 432
Average of 12 463 91 380
Average of 10(w/o Hi & Lo) 447 62 366

The time is measured in milliseconds. From this experiment, it is clear that the SDK API offers significantly less latency than either the Wifi Library or MQTT. From this, it should be clear that callbacks that do not use the sketch loop() function are more efficient.

In Closing

There are advantages and shortcomings when using either MQTT or http for web server applications. MQTT offers a clean and simple code interface, but it does require the use of a broker. That adds an additional layer of complexity and potential failure to your system.

In my opinion, MQTT is best suited to be used for data distribution, not as a web server. The MQTT publish/subscribe model works great to simultaneously send information from one source to many consumers. If speed is critical, the EspressIf SDK API is the best option to use the ESP8266 as a web server.

 

Loading

Share This:
FacebooktwitterredditpinterestlinkedintumblrFacebooktwitterredditpinterestlinkedintumblr

ESP8266 Arduino IDE Web Server Using Callbacks

26th November 2015 by InternetOfHomeThings in All Posts, ESP8266 : An IoT System on Chip, Internet of Things | 21 Comments

sdkcall

While the Arduino IDE is a convenient and familiar platform to develop ESP8266 Web Server based projects, it does have a significant drawback. That is, once the Web Server is started, it is necessary to poll the server each iteration of the loop() function to check for new connections before processing the request.

Putting the check in a loop with other activities, such as reading sensors could and often does result in significant response delays. And possible loss of communication due to connection timeouts.

A better approach is to use a callback that fires whenever an external client connects to the ESP8266 server. This frees up the sketch loop() from the task of servicing http request using inefficient polling code.

And callbacks is how a Web Server is implemented using the ESP8266 EspressIf SDK. But why is it missing from the Arduino IDE? I suspect it is because that is the way it is done with Arduino-based systems—making it both familiar and perhaps suitable for Arduino code reuse with the ESP8266.

But it is not the best solution for new designs.

So that got me thinking…

Why not customize the typical web server sketch to use the SDK callback schema? After all, it is possible to  call ESP8266 SDK functions from the Arduino IDE. While searching, I could not find any examples of this being done, so I’ve created one myself.

And here it is…

The SDK Server Architecture

The SDK API requires several layers of callbacks in a web server design. Since these callbacks are usable in a sketch without modification, the actual sketch functions included in this example are shown.

First,  the server must be setup in the sketch setup() routine. This initialization code was encapsulated in the following function, “SdkWebServer_Init()”, called in setup():

 
 
  1. void SdkWebServer_Init(int port) {
  2.     LOCAL struct espconn esp_conn;
  3.     LOCAL esp_tcp esptcp;
  4.     //Fill the connection structure, including "listen" port
  5.     esp_conn.type = ESPCONN_TCP;
  6.     esp_conn.state = ESPCONN_NONE;
  7.     esp_conn.proto.tcp = &amp;esptcp;
  8.     esp_conn.proto.tcp-&gt;local_port = port;
  9.     esp_conn.recv_callback = NULL;
  10.     esp_conn.sent_callback = NULL;
  11.     esp_conn.reverse = NULL;
  12.     //Register the connection timeout(0=no timeout)
  13.     espconn_regist_time(&amp;esp_conn,0,0);
  14.     //Register connection callback
  15.     espconn_regist_connectcb(&amp;esp_conn, SdkWebServer_listen);
  16.     //Start Listening for connections
  17.     espconn_accept(&amp;esp_conn); 
  18. }

Upon an external TCP connection, the callback “SdkWebServer_listen” is executed:

 
 
  1. void SdkWebServer_listen(void *arg)
  2. {
  3.     struct espconn *pesp_conn = ( espconn *)arg;
  4.     espconn_regist_recvcb(pesp_conn, SdkWebServer_recv);
  5.     espconn_regist_reconcb(pesp_conn, SdkWebServer_recon);
  6.     espconn_regist_disconcb(pesp_conn, SdkWebServer_discon);
  7. }

This functions registers 3 connection-specific callbacks:

  1. SdkWebServer_recv – Called when connection data is received. This callback replaces the polling loop when using the Arduino library.
  2. SdkWebServer_recon – Called when TCP connection is broken.
  3. SdkWebServer_discon-Called when a TCP connection is closed.

The SdkWebServer_recv() function processes http GET and POST requests in a similar manner as the processing in the loop() after an Arduino client connection is detected.

Putting It All Together

In order to contrast the commonly used Arduino Web Server library with the SDK API, I’ve put together an example sketch that supports both web server approaches. It’s on Github:

ESP8266-12 Arduino IDE Web Server Example 

This example provide the same functionality with either server interface. The selection is made prior to building the sketch, using precompilation directives. When possible, the same code is used with both selections. The difference is that the SDK uses callbacks while the Arduino polls for connections.

While no additional hardware is needed to run and test the Web Server, this example assumes the ESP8266 circuit from this post. But the code will work with any ESP8266 circuit, even with no externally connected sensors or controls. But without the ADC multiplexer from the referenced post, the ADC will simply make the same measurement for each multiplexer setting. Which is just fine for the purposes of this example.

Testing the Server From A Web Browser

Since both versions of the Web Server provide the same functionality, the same test can be performed for verification.

You might need to changed the static IP set in this example to match your own LAN subnet. The defaults are currently set to:

IP: 192.168.0.132

Listen Port: 9701

The test url is thus:

192.168.0.132:9701/?request=GetSensors

And the JSON returned from the Web Server should be in the following format:

{
"Ain0":"316.00",
"Ain1":"326.00",
"Ain2":"325.00",
"Ain3":"314.00",
"Ain4":"316.00",
"Ain5":"163.00",
"Ain6":"208.00",
"Ain7":"333.00",
"SYS_Heap":"25408",
"SYS_Time":"26"
}

And if connected, the server will also support changing the LED on/off state:

To turn it on, use the URL:
192.168.0.132:9701/?request=LedOn

And the off URL i:
192.168.0.132:9701/?request=LedOff

Adding server functionality should be obvious after reviewing the sketch.

Conclusions

Using a callback to process Web Server connection requests is far superior to polling. CPU bandwidth is only used in response to client connection events. And with the example framework provided in this post, you can still utilize many of the Arduino sensor libraries without modification.

With a Web Server application, you get the efficiency of the callback with the familiarity of the Arduino IDE.

I hope you find this information useful…

Loading

Share This:
FacebooktwitterredditpinterestlinkedintumblrFacebooktwitterredditpinterestlinkedintumblr

ESP8266 MQTT Publication & External Subscription

11th November 2015 by InternetOfHomeThings in All Posts, ESP8266 : An IoT System on Chip, Internet of Things | Comments Off on ESP8266 MQTT Publication & External Subscription

mqtt

Publishing data produced by ESP8266 sensors to an MQTT broker server is a great option to making your IoT data visible to outside consumers with minimal consumption of precious MCU bandwidth. Simple, easy to implement and very light-weight.

What a great distribution system!

Simply publish your ESP8266 sensor data once, and many subscribers can consume the information, virtually at the same time. Sweet!

MQTT Basics

In case you are new to MQTT, here are a few basics. MQTT is the acronym for message queuing telemetry transport. Now that’s a mouthful! It is essentially a protocol that follows the publish/subscribe model to distribute information from one source to many users.

Here’s a few links for more MQTT information:

I’ve put together an ESP8266 MQTT demo project using an Arduino IDE sketch to publish data for consumption by subscribers using the MQTT protocol. Several options for consuming the data as a subscriber are also presented…

MQTT Broker/Server

For demo purposes, I wanted to use a free broker. After testing a few of the available options, I settled on using the Mosquitto MQTT server. While unsecured, meaning anyone who knows your “topic” can subscribe to the data, it works great for a proving out your design.

server: test.mosquitto.org

port: 1883

Arduino IDE Sketch

There is one library that needs to be added to the Arduino IDE to access the MQTT broker. It’s called “PubSubClient”. Here is how to install it…

Step 1: Open the Arduino IDE

Step 2: From the menu, select Sketch>Include Library>Manage Libraries…

Step 3: Enter “PubSubClient” in the search box.

PubSubClient

Step 4: Click on the “Install” button

Now that was easy.

Great!

But there is one more thing to do  after installing PubSubClient. I have found that the default maximum packet size (128 bytes) is insufficient. You should at least double it to 256 bytes.

How?

Just open the file “…/Arduino/libraries/pubSubClient/src/PubSubClient.h and change the following line:

From:

#define MQTT_MAX_PACKET_SIZE 128

To:

#define MQTT_MAX_PACKET_SIZE 256

Now lets talk about the sketch to test MQTT with the ESP8266…

The setup() function performs 3 basic tasks:

Setup()

  • setup serial port
  • connect to wifi
  • set MQTT broker/server

And loop() maintains the Wifi and MQTT connections, reads the sensors, and publishes the values to the MQTT broker.

Loop()

  • Reconnect to WiFi if disconnected
  • Reconnect to MQTT broker/server if disconnected
  • Read Sensor(s)
  • Publish Sensor data

Here is a break-down of the sketch. First, the objects must be identified

 
 
  1. #include &lt;ESP8266WiFi.h&gt;
  2. #include &lt;PubSubClient.h&gt;
  3. #define wifi_ssid "Your SSID"
  4. #define wifi_password Your Wifi password"
  5. #define mqtt_server "test.mosquitto.org"
  6. #define mqtt_user "not used in this example"
  7. #define mqtt_password "not used in this example"
  8. #define topic1 "t1"
  9. #define topic2 "t2"
  10. WiFiClient espClient;
  11. PubSubClient client(espClient);

And then the sketch setup…

 
 
  1. void setup() {
  2.    Serial.begin(115200);
  3.    setup_wifi();
  4.    client.setServer(mqtt_server, 1883);
  5. }

A typical Wifi connect function:

 
 
  1. void setup_wifi() {
  2.    delay(10);
  3.    // We start by connecting to a WiFi network
  4.    Serial.println();
  5.    Serial.print("Connecting to ");
  6.    Serial.println(wifi_ssid);
  7.    WiFi.begin(wifi_ssid, wifi_password);
  8.    while (WiFi.status() != WL_CONNECTED) {
  9.      delay(500);
  10.      Serial.print(".");
  11.    }
  12.    Serial.println("");
  13.    Serial.println("WiFi connected");
  14.    Serial.println("IP address: ");
  15.    Serial.println(WiFi.localIP());
  16. }

The loop() simply makes sure a connection to the MQTT is made, reads the sensors, and publishes the results. The publish rate is set at 2 seconds minimum.

 
 
  1. void loop() {
  2.   if (!client.connected()) {
  3.     reconnect();
  4.   }
  5.   client.loop();
  6.   //2 seconds minimum between Read Sensors and Publish
  7.   long now = millis();
  8.   if (now - lastMsg &gt; 2000) {
  9.       lastMsg = now;
  10.       //Read Sensors (simulated by increasing the values, range:0-90)
  11.       t1 = t1&gt;90 ? 0 : ++t1;
  12.       t2 = t2&gt;90 ? 0 : ++t2;
  13.       //Publish Values to MQTT broker
  14.       pubMQTT(topic1,t1);
  15.       pubMQTT(topic2,t2);
  16.   }
  17. }

Just copy and paste this complete sketch as a starting template to test your own MQTT publishing with the ESP8266. Remember, you will need to set the WIFI ssid & password in the sketch to your access point values before running.

 
 
  1. #include &lt;ESP8266WiFi.h&gt;
  2. #include &lt;PubSubClient.h&gt;
  3. #define wifi_ssid "YOURSSID"
  4. #define wifi_password "YOURPASSWORD"
  5. #define mqtt_server "test.mosquitto.org"
  6. #define mqtt_user "your_username"
  7. #define mqtt_password "your_password"
  8. #define topic1 "t1"
  9. #define topic2 "t2"
  10. WiFiClient espClient;
  11. PubSubClient client(espClient);
  12. void setup() {
  13.    Serial.begin(115200);
  14.    setup_wifi();
  15.    client.setServer(mqtt_server, 1883);
  16. }
  17. void setup_wifi() {
  18.    delay(10);
  19.    // We start by connecting to a WiFi network
  20.    Serial.println();
  21.    Serial.print("Connecting to ");
  22.    Serial.println(wifi_ssid);
  23.    WiFi.begin(wifi_ssid, wifi_password);
  24.    while (WiFi.status() != WL_CONNECTED) {
  25.      delay(500);
  26.      Serial.print(".");
  27.    }
  28.    Serial.println("");
  29.    Serial.println("WiFi connected");
  30.    Serial.println("IP address: ");
  31.    Serial.println(WiFi.localIP());
  32. }
  33. void reconnect() {
  34.    // Loop until we're reconnected
  35.    while (!client.connected()) {
  36.      Serial.print("Attempting MQTT connection...");
  37.      // Attempt to connect
  38.      if (client.connect("TestMQTT")) { //* See //NOTE below
  39.        Serial.println("connected");
  40.      } else {
  41.        Serial.print("failed, rc=");
  42.        Serial.print(client.state());
  43.        Serial.println(" try again in 5 seconds");
  44.        // Wait 5 seconds before retrying
  45.        delay(5000);
  46.      }
  47.    }
  48. }
  49. //NOTE: if a user/password is used for MQTT connection use:
  50. //if(client.connect("TestMQTT", mqtt_user, mqtt_password)) {
  51. void pubMQTT(String topic,float topic_val){
  52.     Serial.print("Newest topic " + topic + " value:");
  53.     Serial.println(String(topic_val).c_str());
  54.     client.publish(topic.c_str(), String(topic_val).c_str(), true);
  55. }
  56. //Variables used in loop()
  57. long lastMsg = 0;
  58. float t1 = 75.5;
  59. float t2 = 50.5;
  60. void loop() {
  61.   if (!client.connected()) {
  62.     reconnect();
  63.   }
  64.   client.loop();
  65.   //2 seconds minimum between Read Sensors and Publish
  66.   long now = millis();
  67.   if (now - lastMsg &gt; 2000) {
  68.       lastMsg = now;
  69.       //Read Sensors (simulate by increasing the values, range:0-90)
  70.       t1 = t1&gt;90 ? 0 : ++t1;
  71.       t2 = t2&gt;90 ? 0 : ++t2;
  72.       //Publish Values to MQTT broker
  73.       pubMQTT(topic1,t1);
  74.       pubMQTT(topic2,t2);
  75.   }
  76. }

Testing Tools

While there are many tools available, here are a few I have found that you can use to verify the ESP8266 is actually publishing the data.

How?

Simply subscribe to the topic and watch the information updates in real-time.

Google Chrome Application

First try using MQTTLens, an off-the-shelf tool to subscribe to the published data. You can get MQTTLens here. Although this app is installed via Google Chrome, it runs as standalone application. At this time, Chrome apps are supported using Windows, Mac and Linux operating systems.

This app provides basic functionality for testing MQTT publish/subscribe messages.

Once the app has been installed and started, a few configuration steps are needed to connect to our MQTT broker and subscribe to the ESP8266 published topics.

Just click on the configuration icon and fill in the information as shown below:
mqttlenssetup
Once you are connected to the broker, the only thing left to do is subscribe to the data feed. For this example, two topics are published by the ESP8266; t1 and t2. Enter one of these and click on the subscribe button to monitor the data as it is published.
mqqtlens_subscribe
Notice you can change the number of messages displayed using the “-+” icons.

Web Page using JavaScript

MQTTLens is a great tool to get a quick look at your MQTT published data. But when you move from merely testing the data feed to developing a custom application, full control to manipulate and display the information is essential.
Fortunately, this is a simple task using only JavaScript and html.
I’ve developed the following html code to subscribe to the ESP8266 published data. This may be useful as a template on your own website. Simply open a file with this code in a web browser to test. Name this file as you please, using “.html” as the file name extension. In this example, I have named the file “mqtt_subscribe.html”.

 
 
  1. &lt;!DOCTYPE html&gt;         
  2. &lt;html&gt;         
  3. &lt;head&gt;         
  4. &lt;title&gt;MQTT JavaScript Client Example&lt;/title&gt;         
  5. &lt;!-- Latest compiled and minified CSS - add here as needed --&gt;         
  6. &lt;/head&gt;         
  7. &lt;body&gt;         
  8. &lt;!-- HTML to display MQTT topic values --&gt;         
  9. &lt;div&gt;&lt;strong&gt;Published ESP8266 topics via MQTT:&lt;/strong&gt;&lt;/div&gt;&lt;br&gt;         
  10. &lt;div id="t1"&gt;&lt;/div&gt;         
  11. &lt;div id="t2"&gt;&lt;/div&gt;         
  12. &lt;!-- mosquitto MQTT --&gt;         
  13. &lt;script type="text/javascript" src="http://test.mosquitto.org/mosquitto.js"&gt;&lt;/script&gt;         
  14. &lt;!-- jQuery --&gt;         
  15. &lt;script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.2/jquery.min.js"&gt;&lt;/script&gt;         
  16. &lt;!-- Custom MQTT client for this example --&gt;         
  17. &lt;script&gt;         
  18. // Create a client instance         
  19. client = new Mosquitto();         
  20. // Connect to MQTT broker         
  21. var url = "ws://test.mosquitto.org:8080/mqtt";         
  22. client.connect(url);         
  23. // Callback executed upon connection         
  24. client.onconnect = function(rc){         
  25. var topic = 't1';         
  26. client.subscribe('t1', 0);         
  27. client.subscribe('t2', 0);         
  28. };         
  29. //Callback executed upon disconnection         
  30. client.ondisconnect = function(rc){         
  31. client.connect(url);         
  32. };         
  33. //Callback executed upon receipt of MQTT message         
  34. client.onmessage = function(topic, payload, qos){         
  35. if(topic=="t1") {         
  36. $("#t1").html("Topic: t1 Value: " + payload);         
  37. }         
  38. if(topic=="t2") {         
  39. $("#t2").html("Topic: t2 Value: " + payload);         
  40. }         
  41. };         
  42. &lt;/script&gt;         
  43. &lt;/body&gt;         
  44. &lt;/html&gt;

Just to keep things clean and simple, no css styling is included in this example. Here is a snapshot of the browser windows when this file is viewed:

mqqt_javascript_

Android

While I have not yet explored the development of a custom Android App to interface with an MQTT broker, it is certainly possible. What immediately comes to mind, in the spirit of App portable, would be to develop an Android Cordova App. This would allow you to reuse the above html code, without modification.
Here is my step-by-step guide to get started.
But if you are looking for an off-the-shelf MQTT test application, I recommend using MyMQTT . This is a free App available in Google Play Store. Once installed, there are only a few steps needed to subscribe to the ESP8266 example MQTT feed.
The App will startup with the following screen:
Mymqtt_main
First set the app to connect with the broker we are using for this example. Click on “Settings” and enter the broker information as shown:
Mymqtt_setup
Then subscribe to the ESP8266 published topics (t1 and t2):
Mymqtt_subscribe
The dashboard will then display the topic data as it is published:
Mymqtt_messages

Iphone/iPad

MQTTInspector is the most popular App to test MQTT feeds from an iOS device. At $1.99, is not free, yet certainly low-cost. As a Windows, Android and Linux user, I cannot provide any additional information regarding this tool. It appears to be similar to other MQTT client test tools, and can be found on iTunes here.

In Conclusion

Here you have it. A simple guide to publishing topics from an ESP8266 device to an MQTT broker. All you need is a simple Arduino IDE sketch.

Once published, this data feed can be consumed cross-platform, on any device that supports MQTT. From this article, this includes but is certainly not limited to Windows, Mac and iOS. There are also Linux packages available to support MQTT clients.

As always, I hope you find this information useful…

Loading

Share This:
FacebooktwitterredditpinterestlinkedintumblrFacebooktwitterredditpinterestlinkedintumblr

Plugin Free WordPress Bootstrap

4th November 2015 by InternetOfHomeThings in All Posts, Web Coding Tips | 1 Comment

glycon

Bootstrap Plugins for WordPress are great for extending functionality and style without much coding. That is, until something breaks. There is nothing more frustrating than dealing with a website that is broken when you have not changed anything.

How could it happen?

I really don’t know. And while there are many great Bootstrap plugins for WordPress, I was depending on one of the most popular plugins (not mentioning names here) to support my responsive navigation menu. Worked great for almost a year, collapsing to the well-known 3-bar button for smartphones and small user browser windows-and expanding for larger screens. But then recently, it just stopped working.

It did not make sense. All I had done over the past few weeks was add a few posts to the site. That should have no effect on the navigation menu. Deactivating and reinstalling the plugin did not help. Yet the site needed to be fixed.

And fast!

After taking the “insanity” route for hours…trying the same things over and over with no improvement, I decided to delete the plugin and add Twitter Bootstrap manually to my WordPress site. While everything did not go smoothly at first, I did manage to restore my site’s bootstrap features. This time, without the non-transparent layer of a plugin.

Perhaps this guide may be of use to someone who encounters a similar situation…

Installing Bootstrap into a WordPress Site

First thing to do is to get the bootstrap files. I was unsuccessful in pointing to the CDN files so the actual files were downloaded and installed on the site. Go to getbootstrap.com and click on the “Download Bootstrap” button to get the needed files. The second download button looks like this:

download

You will also need the Jquery file, even if it is already installed as a WordPress plugin. As of this writing, Jquery is at version 2.1.4 and can be downloaded from this site.

It is important to copy the files to your WordPress theme folder as shown in the following table. It will not work if copied to a different directory.

File Name WordPress Folder
bootstrap.min.css /css
bootstrap-theme.min.css /css
bootstrap.min.js /js
jquery-2.1.4.min.js /js

The next thing to do is to modify  your themes functions.php file. This will make bootstrap functional within your WordPress theme. unfortunately, since functions.php is unique for each theme, the modifications must be made separately for each theme used.

Let’s first include the bootstrap style sheets when WordPress starts up. That is simply a matter of adding the following two “wp_register_style” and “wp”enqueue_style” lines to the function <your theme name>_of_styles():

 
 
  1. function &lt;your theme name&gt;_of_styles() {
  2.        wp_register_style( 'bootstrap', get_template_directory_uri() . '/css/bootstrap.min.css' );            
  3.        wp_register_style( 'bootstraptheme', get_template_directory_uri() . '/css/bootstrap-theme.min.css' );            
  4.        .
  5. .
  6. .            
  7.        wp_enqueue_style( 'bootstrap' );            
  8.        wp_enqueue_style( 'bootstraptheme' );            
  9.        .
  10. .
  11. .            
  12. }

Similarly, the Jquery files need to be included when WordPress starts up. This, however needs to be handled differently. That is because these files need to be introduced just before the bootstrap navigation bar html code is added, not when WordPress starts up.

This requires the addition of a new function to the functions.php file. Since the code is php, it simply echos the html script tags used to include the files. It is important for the jquery file to be listed before bootstrap:

 
 
  1. function ld_bootstrap_js() {
  2.    echo '&lt;script src="/js/jquery-2.1.4.min.js"&gt;&lt;/script&gt;';
  3.    echo '&lt;script src="/js/bootstrap.min.js"&gt;&lt;/script&gt;';
  4. }

As noted, this function needs to be called just before the navigation bar menu is introduced. This requires a change to one more theme file, “header.php”. Add the call just after the stylesheet in this file:

 
 
  1. .
  2. .
  3. &lt;link rel="stylesheet" type="text/css" media="all" href="&lt;?php echo get_stylesheet_uri(); ?&gt;" /&gt;
  4. &lt;?php ld_bootstrap_js(); ?&gt;
  5.    .   
  6. .
  7. &lt;/head&gt;

That’s it. Not really that difficult once you know how. With these changes, your WordPress website will support a responsive bootstrap navigation bar.

In Closing

While you may never have any problems using a bootstrap plugin, you might want to keep this information handy, just in case it fails you. This will bring your site back on line quickly. Independent of a third-party plug-in.

Loading

Share This:
FacebooktwitterredditpinterestlinkedintumblrFacebooktwitterredditpinterestlinkedintumblr

ESP8266 Dual AT & Web Server Mode

30th October 2015 by InternetOfHomeThings in All Posts, Arduino, ESP8266 : An IoT System on Chip, Internet of Things | 13 Comments

Here is an exciting framework for fully utilizing the huge capabilities of the ESP8266.

This is an original.

Something that I have not seen anywhere before. You see, using this structure opens up many more possibilities and applications for this amazing device…

Most folks in the ESP8266 world seem to fall into one of two camps. The first group, typically diehard Arduino developers, want to use the ESP8266 as a serial to WIFI shield. A couple bucks to add WIFI to the very popular Arduino platform is most cost effective method of bringing it on-line. Much cheaper than alternative WiFi shields.

Others see the ESP8266 as a complete solution. It is often referred to as a System On a Chip (SoC). They not only use the ESP8266 for WIFI internet access, but also use the platform to read sensors, control “things”, process data and provide web server functionality.

Among The Examples

The majority of those that have delved into the ESP8266 world started by loading the serial to WIFI firmware often referred to as the AT command set. This is available as a binary file downloaded to the ESP8266 via it’s serial interface. That’s where I began my ESP8266 journey.

It was not long after my start that I searched for the AT firmware source code. Fortunately, it is available within earlier versions of the EspressIf SDK. You see, prior to version 1.0, many example applications were included in the development package. And yes, the AT command set was among these examples.

The other most useful example application is the IoT_Demo. This software has survived every release of the SDK, including the current version. This example provides a framework for a web server to make your “things” Internet accessible.

After studying the software structure of these examples I got to thinking…

Wouldn’t it be great to merge these two applications? So you benefit from both features?

It indeed did look feasible.

The combined application would not only service AT commands through the serial port, but also service http GET or POST commands via the Internet.

And that’s not all…

This single ESP8266 firmware application could also be used to read sensors and control devices.

I am stoked to say I have successfully merged these examples. This broadens the capabilities and applications of a single ESP8266 based system.

It works!

Here’s how…

ESP8266 Dual Server Architecture

The ESP8266 is used to perform 3 primary functions. First, it provides basic serial to Wifi capability using the standard AT command set that many first-time users tinker with. Note that since the design presented here includes the AT firmware source code, the command set can be expanded to also provide access to the Esp8266 Daq and Control features. This is depicted in the following diagram as two-way arrows from the AT Server.

dualserver

The ESP8266 also provides Web Server functionality. This feature operates completely independently from the serial AT Server. The web server responds to http GET commands received form the Internet via the built-in Wifi capability. Just like the AT Server, the Web Server has access to the Daq and Control functions.

And finally, the data acquisition (DAQ) and control function, also running independently from the other two features, controls all the Esp8266 outputs and receives all the sensor inputs. Because of the many possible sensors connected to the system, the DAQ function must be managed so that it does not monopolize the Esp8266 processor. This is accomplished by limiting it’s execution to the reading of one sensor each time the function is called.

Task Distribution

Separate callbacks are registered for each of the three ESP8266 primary functions. Both the AT and Web Server callbacks are event driven. They are only executed upon user request. The AT callback is executed anytime data is received on the serial port, while the Web Server callback is run upon receipt of an http GET request.

The callback for the Daq/Control features are different. This feature is set up to execute periodically (once every second) from a Timer callback. This keeps the sensor data fresh, available for consumption by the AT or Web Server, upon request.

AT Callback – The AT serial port server callback is installed in the uset_init() function when at_init() is executed. The AT server responds to any of the requests defined in at_cmd.h.

 
 
  1. at_funcationType at_fun[at_cmdNum]={
  2. {NULL, 0, NULL, NULL, NULL, at_exeCmdNull},
  3. .
  4. .
  5. {"+CIPSERVER", 10, NULL, NULL,at_setupCmdCipserver, NULL},
  6.   {"+CIPMODE", 8, NULL, at_queryCmdCipmode, at_setupCmdCipmode, NULL},
  7.   {"+CIPSTO", 7, NULL, at_queryCmdCipsto, at_setupCmdCipsto, NULL},
  8.   {"+CIUPDATE", 9, NULL, NULL, NULL, at_exeCmdCiupdate},
  9.   {"+CIPING", 7, NULL, NULL, NULL, at_exeCmdCiping},
  10.   {"+CIPAPPUP", 9, NULL, NULL, NULL, at_exeCmdCipappup},
  11.   {"+GETSENSOR", 10, NULL, NULL, at_exeGetSensorVal, NULL},

The new request, “+GETSENSOR”, has been added to provide a method for the serial port server to get the sensor readings. Upon receiving this request, the function at_exeGetSensorVal() is called. This function returns the current sensor reading over the ESP8266 serial port.

 
 
  1. void ICACHE_FLASH_ATTR
  2. at_exeGetSensorVal(uint8_t id, char *pPara)
  3. {
  4.       int isensno = atoi(++pPara);
  5.       switch(isensno) {
  6.             case 1:
  7.                   uart0_sendStr(tInside);
  8.                   break;
  9.             case 2:
  10.                   uart0_sendStr(tOutside);
  11.                   break;
  12.             case 3:
  13.                   uart0_sendStr(tAttic);
  14.                   break;
  15.             case 4:
  16.                   uart0_sendStr(tDht11);
  17.                   break;
  18.             case 5:
  19.                   uart0_sendStr(hDht11);
  20.                   break;
  21.             case 6:
  22.                   uart0_sendStr(pBmp085);
  23.                   break;
  24.             case 7:
  25.                   uart0_sendStr(tBmp085);
  26.                   break;
  27.             case 8:
  28.                   uart0_sendStr(aBmp085);
  29.                   break;
  30.             default:
  31.                   uart0_sendStr("out of range");
  32.                   break;
  33.       }
  34.       uart0_sendStr("\r\n");
  35. }

The requests are made by sending the following string over the serial port:

AT+GETSENSOR=n

“n” is the sensor number and corresponds to the case number in the code above.

As you can see, it is not difficult to add your own custom commands to the standard ESP8266 AT command set.

Web Server Callback – This application uses the Web Server code provided in the IoT_Demo example. The user_init() function sets up and connects to the local Wifi before launching the Web Server. The callback webserver_recv() is registered during the initialization sequence. That function is executed any time an http GET request is received.

For this demo application, the Web Server only responds to “request-GetSensors”. More requests can be added as noted in the code that follows. When received, the server replies by sending a JSON string containing all the sensor readings.

 
 
  1. LOCAL void ICACHE_FLASH_ATTR
  2. webserver_recv(void *arg, char *pusrdata, unsigned short length)
  3. {
  4.     URL_Param *pURL_Param = NULL;
  5.     char *pParseBuffer = NULL;
  6.     bool parse_flag = false;
  7.     struct espconn *ptrespconn = arg;
  8.     int i;
  9.     espconn_set_opt(ptrespconn, ESPCONN_REUSEADDR);
  10.     if(upgrade_lock == 0){
  11.           parse_flag = save_data(pusrdata, length);
  12.         if (parse_flag == false) {
  13.               response_send(ptrespconn, false);
  14.         }
  15.         pURL_Param = (URL_Param *)os_zalloc(sizeof(URL_Param));
  16.         parse_url_params(precvbuffer, pURL_Param);
  17.         switch (pURL_Param-&gt;Type) {
  18.             case GET:
  19.                 if(os_strcmp(pURL_Param-&gt;pParam[0], "request")==0) {
  20.                     <strong>// GetSensors is the only request the server currently supports</strong>
  21. <strong>                    if(os_strcmp(pURL_Param-&gt;pParVal[0], "GetSensors")==0) {</strong>
  22. <strong>                          json_send(ptrespconn, GET_SENSORS);</strong>
  23. <strong>                    }</strong>
  24.                     // Add additional requests here
  25.                 }
  26.                 json_send(ptrespconn, CONNECT_STATUS);
  27.                 break;
  28.             case POST:
  29.                   ets_uart_printf("We have a POST request.\n");
  30.                  break;
  31.         }
  32.         if (precvbuffer != NULL){
  33.               os_free(precvbuffer);
  34.               precvbuffer = NULL;
  35.         }
  36.         os_free(pURL_Param);
  37.         pURL_Param = NULL;
  38.     }
  39.     else if(upgrade_lock == 1){
  40.           local_upgrade_download(ptrespconn,pusrdata, length);
  41.             if (precvbuffer != NULL){
  42.                   os_free(precvbuffer);
  43.                   precvbuffer = NULL;
  44.             }
  45.             os_free(pURL_Param);
  46.             pURL_Param = NULL;
  47.     }
  48. }

 

Acquisition/Control Timer Callback – This callback serves as the application’s periodic loop() function. It is called once every second. The functions is mechanized as a state machine. In this example, 5 states are implemented, one for each sensor in the system. Exactly one state is executed each time the function is called. Upon completion of the sensor read, the code is set to execute the next state the next time the function is called. The code can be modified as needed to add or delete states and sensor hardware.

 
 
  1. LOCAL void ICACHE_FLASH_ATTR loop_cb(void *arg)
  2. {
  3.     char szT[32];
  4.     DHT_Sensor DHsensor;
  5.     DHT_Sensor_Data data;
  6.     DHsensor.pin = 5;  //GPIO14
  7.     DHsensor.type = DHT11;
  8.     int32_t temperature;
  9.     int32_t pressure;
  10.     //---------------------------------------------------
  11.     //This state machine reads 1 sensor each iteration
  12.     //---------------------------------------------------
  13.       switch(nTcnt%5) {
  14.             case 0: //Read first DS18B20 Temperature Sensor
  15.                 get_temp_ds18b20(1,1,tInside);
  16.                 break;
  17.             case 1: //Read second DS18B20 Temperature Sensor
  18.                 get_temp_ds18b20(2,1,tOutside);
  19.                 break;
  20.             case 2: //Read third DS18B20 Temperature Sensor
  21.                 get_temp_ds18b20(3,1,tAttic);
  22.                 break;
  23.             case 3: //Read DHT11 temperature and humidity Sensor
  24.                 DHTRead(&amp;DHsensor, &amp;data);
  25.                 DHTFloat2String(tDht11, ((9/5) * data.temperature)+32);
  26.                 DHTFloat2String(hDht11, data.humidity);
  27.                 break;
  28.             case 4: //Read BMP085 Temperature and pressure Sensor
  29.                 temperature = BMP180_GetTemperature();
  30.                 pressure = BMP180_GetPressure(OSS_0);
  31.                 os_sprintf(pBmp085,"%ld.%01d", pressure/3386,(pressure%3386)/1000);
  32.                 os_sprintf(tBmp085,"%ld.%01d", ((temperature*18)/100) + 32,(temperature*18)%100);
  33.                 os_sprintf(aBmp085,"%03d", 328 * (BMP180_CalcAltitude(pressure)/100000));
  34.                 break;
  35.             default:
  36.                 break;
  37.       }
  38. }

Want the code for this framework? Please feel free to use and modify it for your own custom requirements. It is available on Github here.

Testing the Code

Let’s test the dual server by sending a command over the serial port and via http GET. The command will retrieve sensors values read by the ESP8266. For this test case, the 1 second loop() function has been modified as follows to populate the sensor variables without actually reading any sensors.

 
 
  1. LOCAL void ICACHE_FLASH_ATTR loop_cb(void *arg)
  2. {
  3.     //-----------------------------------------------------
  4.     //This test code populates the variables associated
  5. //with the sensor values in lieu of actually reading
  6. //sensors. This code is used to test the functionality
  7. //of the dual server application. All variables are
  8. //set with fixed values except iTinside, which is
  9. // incremented by 1.5 degrees each time this function
  10. //is called.
  11.     //-----------------------------------------------------
  12. if(tI&lt;800) {
  13. tI += 15;
  14. }
  15. else {
  16. tI = 600;
  17. }
  18. os_sprintf(tInside,"%d.%d",tI/10,tI%10); //Sensor 1
  19. os_sprintf(tOutside,"%s","79.2"); //Sensor 2
  20. os_sprintf(tAttic,"%s","88.5"); //Sensor 3
  21. os_sprintf(tDht11,"%s","69.1"); //Sensor 4
  22. os_sprintf(hDht11,"%s","34.7"); //Sensor 5
  23. os_sprintf(pBmp085,"%s","29.7"); //Sensor 6
  24. os_sprintf(tBmp085,"%s","71.1"); //Sensor 7
  25. os_sprintf(aBmp085,"%s","555.0"); //Sensor 8
  26. }

AT Serial Server Test

The added function in this demo application to the AT command set is “AT+GETSENSOR=n”. “n” is filled in with an integer value representing the sensor number. Enter the following to test this new command:

Sensor Serial Port Command Expected Reply
Inside Temperature AT+GETSENSOR=1 increasing value in range 60-80 F
Outside Temperature AT+GETSENSOR=2 79.2
Attic Temperature AT+GETSENSOR=3 88.5
DHT11 Temperature AT+GETSENSOR=4 69.1
DHT11 Humidity AT+GETSENSOR=5 34.7
BMP085 Pressure AT+GETSENSOR=6 29.7
BMP085 Temperature AT+GETSENSOR=7 71.1
BMP085 Altitude AT+GETSENSOR=8 550.0

Web Server Test

The web server can easily be tested by entering a single URL in any web browser. In this demo application, the IP is hard-coded to “192.168.0.106” and the server responds to port 9703 requests. Thus, the test URL is:

http://192.168.0.106:9703/?request=GetSensors

And the expected reply will be the following JSON string:

 
 
  1. {
  2. "B_Pressure":"29.7",
  3. "B_Temperature":"71.1",
  4. "B_Altitude":"555.0",
  5. "DS_TempInside":"79.5", &lt;--NOTE: This value will range from 60-80 F in 1.5 degree increments
  6. "DS_TempOutside":"79.2",
  7. "DS_TempAttic":"88.5",
  8. "DH_Humidity":"34.7",
  9. "DH_Temperature":"69.1"
  10. }

Your demo application is working properly if the expected responses are observed.

In Closing

This opens up new possibilities. With this structure, you use the ESP8266 as an Arduino serial to WIFI shield. Yet at the same time use this same ESP8266 as a web server. And a sensor acquisition engine. And to control “things”. All you need to do is add meat to the bones provided here. What will you come up with?

I hope you find this information useful…

Loading

Share This:
FacebooktwitterredditpinterestlinkedintumblrFacebooktwitterredditpinterestlinkedintumblr

ESP8266 Free Space Assessment

21st October 2015 by InternetOfHomeThings in All Posts, ESP8266 : An IoT System on Chip, Internet of Things, Web Coding Tips | 1 Comment

percentage

Have you ever used the EspressIf SDK to compile code and wondered how much free space you have? How much more code you can add before running out of memory?

That information is very useful when developing code, and is readily available when using the Arduino IDE. So why is it missing from the SDK console output when you execute a build?

That used to bug me, until I figured out how to include it. Read on if you want a better view of the free space available after building your ESP8266 code.

Here is how I did it…

Evaluating Solution Options

First thing I did was review the Makefile that was provided with the SDK examples. I thought it might be a simple matter of adding a few lines to output the desired information. After all, the current Makefile outputs the number of byte used. And we know how much memory is available. A simple calculation and boom, you got the answer.

Wrong!

What I found was that the current memory statistics are generated by the utility “MemAnalyzer.exe” . This tool is included with the EspressIf SDK. So I figured all that it would take was a few changes to this utility to format the memory statistics in the format I wanted. Problem is, the source code for this utility is nowhere to be found. At least I had no luck finding it.

So what I decided to do was create another utility. One that would format the information provided by MemAnalyzer.exe and include it in the build Console output.

The Solution

Obviously, this requires some changes to the project’s “Make” file. The current information, while I find it inadequate, is produced by the following Makefile file line:

 
 
  1.  $(Q) $(SDK_TOOLS)/memanalyzer.exe $(OBJDUMP).exe $@

And the output from the memanalyzer tool is displayed like this example:

 
 
  1.    Section|              Description| Start (hex)| End (hex)|Used space
  2. ------------------------------------------------------------------------------
  3.       data|   Initialized Data (RAM)|    3FFE8000|  3FFE8D8C|    3468
  4.     rodata|      ReadOnly Data (RAM)|    3FFE8D90|  3FFEA3F4|    5732
  5.        bss| Uninitialized Data (RAM)|    3FFEA3F8|  3FFF4C08|   43024
  6.       text|       Cached Code (IRAM)|    40100000|  4010782C|   30764
  7. irom0_text|      Uncached Code (SPI)|    40240000|  4026F574|  193908
  8. Total Used RAM : 52224
  9. Free RAM : 29696
  10. Free IRam : 2022

In order to access this information, it was redirected to a text file (mem.txt) by adding a new line to the MakeFile:

 
 
  1. $(Q) $(SDK_TOOLS)/memanalyzer.exe $(OBJDUMP).exe $@
  2. <strong><span style="color: red;">$(Q) $(SDK_TOOLS)/memanalyzer.exe $(OBJDUMP).exe $@ &gt;mem.txt</span></strong>

My new utility reads the mem.txt file and outputs the re-formatted memory statistics to the build Console. The utility is named “EspMemUsage.exe” and is called in the Makefile with the addition of a second line:

 
 
  1. $(Q) $(SDK_TOOLS)/memanalyzer.exe $(OBJDUMP).exe $@
  2. $(Q) $(SDK_TOOLS)/memanalyzer.exe $(OBJDUMP).exe $@ &gt;mem.txt
  3. <strong><font style="color: red;">$(Q) $(SDK_TOOLS)/EspMemUsage.exe $(OBJDUMP).exe $@</font></strong>

And here is what the added Console output information looks like:

 
 
  1. ------------------------------------------------------------------------------
  2. Resource            |Size(bytes)|    Used|       %Used|        Free|   %Free
  3. --------------------|-----------|--------|------------|------------|----------
  4. IRAM - Cached Code  |      32768|   30746|          94|        2022|       6
  5. SPI - Uncached Code |     253952|  193908|          76|       60044|      24
  6. RAM - Data          |      81920|   52224|          63|       29696|      37
  7. ------------------------------------------------------------------------------

It help me a lot to see the memory usage in this format. I hope you find it useful too.

The utility, makefile, and source code are available on Github here. Installation instructions are provided in the readme file. It was developed using the free Microsoft code development  tool Visual Studio 2015 Community. Customize as you see fit to meet your specific needs and wants.

What It Does

The utility code is written using the c# programming language. The Program class is structured into 3 static functions.

  1. Main – This console application executes Main upon entry.  Main opens the file “mem.txt” and reads each line separately. If the line contains a memory value of interest, the information is extracted and formatted for output by calling the function “MemStat”. After completing the evaluation of the file “mem.txt”, the formatted results are output to the console.
  2. MemStat – This function extracts the value from the line read. Any spaces in the extracted value are removed by calling the 3rd and final function, “RemoveSpaces”. The memory used and free statistics are then calculated and formatted. The formatted string is returned to the function caller.
  3. RemoveSpaces – Removes spaces from a string.

In Closing

The utility presented here provides clarity to the ESP8266 code developer using the EspressIf SDK. So you know exactly how much memory is available for expansion, delineated by memory resource. Once installed, all you need to do is use the modified Makefile in your new projects to get the information displayed in the console output.

I hope you find this information useful.

 

Loading

Share This:
FacebooktwitterredditpinterestlinkedintumblrFacebooktwitterredditpinterestlinkedintumblr

Capturing ESP8266 Max Value on ThingSpeak

12th October 2015 by InternetOfHomeThings in All Posts, ESP8266 : An IoT System on Chip, Internet of Things | Comments Off on Capturing ESP8266 Max Value on ThingSpeak

This article provides a method to capture and store the maximum value reached in a ThingSpeak channel field.

So what’s the big deal? You’ve got all the data captured. Just retrieve the data points and search for the maximum among them. Well, read on…

You see, like many users, my ThingSpeak channel receives data captured by an ESP8266 micro-control unit (MCU). Both ESP8266 System Status and Sensor data are collected.

One of my key data points of interest is the lapse time between ESP8266 resets. The value is saved to the ThingSpeak channel in a field containing the ESP8266 ms running time. This, along with the sensor data is added to the channel data once each hour.

What I found was that reading all of the captured data and searching through it for a maximum value became increasingly slower as my database record count grew. It needed to run quicker. So I came up with a simple solution…

A method of comparing each new ESP8266 reading with the maximum value saved in a ThingSpeak channel field. This requires a CRON script running on my web server. The script is executed once every 20 minutes. What? It’s really not that complicated.

Here’s the details…

System Overview

The system is comprised of an ESP8266, a web host server capable of running CRON scripts, and a ThingSpeak channel. The CRON script is the key component used to move the information from the ESP8266 to the ThingSpeak channel.

The ESP8266 runs it’s own firmware serving two primary purposes. Periodically, it refreshes sensor readings and various system parameters, including the MCU run time since the last reset. This process runs on an internally registered timer callback.

The ESP8266 also acts as a web server. In this capacity, the server responds to http GET requests, returning sensor and system values in JSON format.

It should be noted here that this is not limited to an ESP8266. Any MCU capable of these two functions can be used.

overview

PHP CRON Script

So what exactly is a CRON script?

CRON originated in the Unix world. In this case, it is simply a service provide by the web host to automatically execute a script at a predetermined time. There are a couple of  options available to deploy automated scripts for data exchange to a ThingSpeak channel.

Organically, ThingSpeak provides the TimerControl App to perform an action at a specific time or on a regular schedule. This, along with the ThingSpeak ThingHTTP App is all that is needed. ThingHTTP interacts with the micro-controller using http GET or POST.

But there is a limitation with this approach…

You see, in my case, I also need to interface with a mySQL database. And ThingSpeak does not provide an API for that purpose. At least not yet. Sure, a proxy interface could be developed using a php script, that would unnecessarily  complicate the design. So I have used a different approach.

For this discussion, the CRON script uses php to interface both with ThingSpeak and a mySQL database. This eliminates the need for a proxy to overcome the ThingSpeak App’s lack of a mySQL interface. But even this approach comes with restrictions…

In order to prevent excessive resource consumption, my web host limits my account to three scheduled scripts.

The other stipulation is that the scripts cannot run continuously. The job should execute and complete within a “reasonable” amount of time. For this simple case, less than 10 seconds should be required.

The script tasks are simple and straight-forward…

  1. Read the current ESP8266 Sensor and status values (HTTP GET)
  2. Write the Sensor and status values to ThingSpeak
  3. Write all values to the mySQL database
  4. Read the max run-time value from ThingSpeak
  5. Evaluate whether current value for run-time is greater than the max value saved
  6. Write the max value back to ThingSpeak
  7. Write the other sensor and status values to ThingSpeak
  8. Write all values to the mySQL database

And here is the script.

PHP mySQL/ThingSpeak interface
 
  1. &lt;?php
  2.     include("sensors_weather_save_utils.php");
  3.     $diostatus = "";
  4.     define("MAXONTIME", 420); //7 minutes (60 * 7)
  5.     //----------------------------------------
  6.     //-----Get current Epoc time (to local)
  7.     //----------------------------------------
  8.     $now = time();
  9.     date_default_timezone_set('America/Los_Angeles');
  10.     $localtime_assoc = localtime(time(), true);
  11.     $current_hr = $localtime_assoc['tm_hour'];
  12.     $current_mn = $localtime_assoc['tm_min'];
  13.     //----------------------------------------
  14.     //specific time tasks
  15.     //----------------------------------------
  16.     if( $current_mn==0 ) {
  17. //Save Home Weather Sensors to database on-the-hour
  18.         require 'sensors_weather_save.php'; 
  19.     }
  20.     if( ($current_hr==7) &amp;&amp; ($current_mn==20) ) {
  21. //Now lets turn off the front porch light if it is 7:20 am
  22.         require 'iot_dtdoff.php'; 
  23.     }
  24.     //----------------------------------------
  25.     //Run every 20 minutes)
  26.     //----------------------------------------
  27.     //Check max time: Get ms since ESP8266 started
  28.     $temp_tm = getEsp8266Sensor($Esp8266SensorURL,"Y","SYS_Time");
  29.     //Get Thingspeak Channel 45834 field1:Max System Time)
  30.     $ThingsSpeakURL = "https://api.thingspeak.com/channels/"."45834".
  31. "/feed/last.json?api_key=".$thingspeak45834;
  32.     $max_up = getEsp8266Sensor($ThingsSpeakURL,"Y","field1");
  33.     if($temp_tm &gt; $max_up) {
  34.      //Update ThingSpeak field1
  35.         $ThingsSpeakURL = "https://api.thingspeak.com/update?key=".$thingspeak45834.
  36. "&amp;field1=".$temp_tm;
  37.         get_fcontent($ThingsSpeakURL);
  38.     }
  39.     $status = file_get_contents($modtronixURL);
  40.     $status_array = json_decode($status);
  41.     //----------------------------------------
  42.     //-----find the dio port f byte value-----
  43.     //----------------------------------------
  44.     foreach($status_array as $key =&gt; $value) {
  45.         if(strstr($key,"pf")) {
  46.          $diostatus .= $value;
  47.         }
  48.     }
  49.     $diostatus = "B".$diostatus; //Binary prefix
  50.     //----------------------------------------
  51.     //-----get the dio port f byte last value
  52.     //----------------------------------------
  53.     // Connect to database (host,username,password,databasename) 
  54. // "@" suppresses errors/warnings
  55.     $link = @mysqli_connect("localhost",$mysqlUser,$mysqlPass,$mysqlUser);
  56.     //check if connection errors
  57.     if(mysqli_connect_error() ) {
  58.      die("Error: Could not connect to database"); //stops php script
  59.     }
  60.     //Selects(takes something out of database) everything(*) from the users table
  61.     $query = "SELECT * FROM myhome";
  62. if($result = mysqli_query($link, $query)) { //returns "TRUE" if successful
  63. $row = mysqli_fetch_array($result);
  64.     }
  65.     //----------------------------------------
  66.     //-----Get database status
  67.     //----------------------------------------
  68.     $laststatus = $row[PortF]; //Get Last Port F Status
  69.     $lasttime = $row[timetag]; //Get Last Epoc time
  70.     //----------------------------------------
  71.     //-----clear dio if no change
  72.     //----------------------------------------
  73.     $lapsesec = $now - $lasttime;
  74.     echo ($lapsesec);
  75.     if($diostatus != "B11111111") {
  76.     if($lapsesec &gt; MAXONTIME) {
  77.             file_get_contents($resetportfURL); //Reset Port F (Sprinklers off)
  78.         }
  79.     }
  80. //----------------------------------------
  81.     //-----update database status
  82.     //----------------------------------------
  83.     $query = "UPDATE `myhome` SET `PortF` = '".$diostatus."' WHERE id=0 LIMIT 1";
  84.     mysqli_query($link, $query);
  85.     $query = "UPDATE `myhome` SET `timetag` = '".$now."' WHERE id=0 LIMIT 1";
  86.     mysqli_query($link, $query);
  87. ?&gt;

This script runs every 20 minutes. And while the topic of this post is saving a max value, the other tasks performed by my script are also shown. I figured this would arm you with some actually used ideas on what is possible using a scheduled script to interact with your “things”.

As you can see, for example, the values are read and saved every hour using the “sensors_weather_save.php” script, while at 7:20 am, my dusk to dawn porch light is turned off. I found this necessary, especially on darker or overcast days. Physically, a bright LED is flashed briefly in front of the light sensor to extinguish the light, if it is on-simply by controlling a digital output signal.

The script to save values to mySQL is not shown. In lieu of that, the mySQL interaction exposed in the script illustrates the php interface. Here, it is used to turn off my sprinklers, if they remain on for more than 7 minutes. The check is made every 20 minutes. This was necessary as a watchdog in the event the sprinklers, under IoT control, were inadvertently left or stuck on.

Conclusion

This information provides a method to automatically update the max value of a ThingSpeak channel field. It also illustrates how to interface with a mySQL database using php. This php bridge script can be used to analyse and manipulate ThingSpeak data in any way needed.

Taking this the next logical step, I am planning a follow-up post to use the ThingSpeak Timer and ThingHTTP to accomplish the same thing within the ThingSpeak framework. This will, however, require the added complexity of a php proxy in the absence of a ThingSpeak mySQL API.

Stay tuned for that…

Loading

Share This:
FacebooktwitterredditpinterestlinkedintumblrFacebooktwitterredditpinterestlinkedintumblr
← Older posts
Newer posts →
Press Ctrl+C to copy the following code.
"