Okay, so you got your ultra low-cost ESP8266 workhorse setup to read some sensors. And you have an awesome webpage built to visually display the readings. But how do you get the data from the ESP8266 to your website?

PHP and/or Javascript are the two most popular methods. I have used both.

This writing presents a simple Javascript solution. I have also posted an approach to this task using PHP here.

My need came about while developing a mobile app to monitor and control devices in my home. My personal IoTs. Using the Apache Cordova API with the Eclipse IDE. I was developing my App with just HTML, CSS, and JavaScript. Yet it will compile and install onto my Android device like any other application. It is my understanding that the same code can be built into an iOS App as well…but…being strictly a PC user, I have not attempted that.

And let’s get back to using Javascript to pull information from an ESP8266 based web server.

The thing was, I needed to pull my weather, sprinkler, garage and other sensor data from the ESP8266 circuit that monitors these devices to my App. I also wanted maintain control of the data. To be free from a cloud middleman such as ThingsSpeak.

With this in mind, we have two sides to this solution; The micro-controller unit (MCU), which is the ESP8266 in this case, and the client App. The ESP8266 must read all the devices connected to it and then send this data, on request, to the App client. The preferred format is either JSON or XML.

I use JSON strings.

Here is an example of a short function that can be used to create a JSON string in an Arduino IDE sketch. It is critical to include the http header information provided in this example. The header authorizes your domain, allowing the Javascript AJAX to retrieve the ESP8266 http response.

Here is the Javascript with AJAX code used to request a JSON string from the  ESP8266, decode it, and display the values on a webpage.

function updateWeatherSensorData() {
    requestURL = "http://myrouterdomain.com:9999/?request=GetSensors";
    if ( typeof updateWeatherSensorData.timeout == 'undefined' ) {
        // It has not... perform the initialization
        updateWeatherSensorData.timeout = 0;
    }
    //Get Weather Sensor Value
    $.ajax({
        url: requestURL,
        error: function(error){
            if(updateWeatherSensorData.timeout++ <10) {
                setTimeout(updateWeatherSensorData, 1000);
            }
            else {
                updateWeatherSensorData.timeout = 0;
            }
        },
        success: function(thedata){
            $("#baro").html(thedata.B_Pressure + " inHg");
            $("#tempIn").html(thedata.DS_TempInside + " F");
            $("#tempOut").html(thedata.DS_TempOutside + " F");
            $("#tempAttic").html(thedata.DS_TempAttic + " F");
            $("#humidity").html(thedata.DH_Humidity + " %");
            updateWeatherSensorData.timeout = 0;
        },
        timeout: 7000 // sets timeout to 7 seconds
    });
}

I am assuming you have a domain name pointed to your router and your router is configured to port forward internet requests. For reference, here is how to set this up.

How this Javascript code works:

First, the URL used to send the http GET request to the ESP8266 is defined. This URL has 3 parts

1. The domain name: http://myrouterdomain.com
2. The port your ESP8266 web server is listening on: 9999
3. The request string recognized by the web server: /?request=GetSensors

Then we need to define a timeout property for this function. The timeout is actually used to store the number of attempts to retrieve the JSON from the ESP8266 before successful.

And now, the AJAX call. As you can see, the code is structured like a JSON string, with 4 keys and 4 values:

1. url: This defines where to send the request.
2. error: In case the request fails, the call is repeated, up to 10 times
3. success: When successful, the webpage elements with the indicated ID are filled with the value retrieved from the ESP8266
4. timeout: Up to 7 seconds is allowed before a timeout error is declared.

That’s it! This is what I have been using to retrieve JSON strings from my ESP8266 using an AJAX call.

In Summary

Here are a few closing thoughts of my observations for your information…

• The error handling is essential. This call fails frequently, perhaps every 2 or 3 requests.
• Most of the time, the error is because a blank string is filled from this call. I do not know if that is an ESP8266 issue or a problem with AJAX.
• The timeout was set to 7 seconds because I have seen it take up to 4 seconds to get a response.

Hope you find this information useful for your ESP8266, Arduino and other MPU projects.

Share This:

While I have published another post, covering this topic using a Javascript AJAX call, using php has unique attributes, and must be used in certain cases. Please refer to that post is anything here is unclear.

In my case, I needed to periodically save my home weather sensor values into a mySQL database. While this can be accomplished with a CRON triggered php server side script, the client sided AJAX will not work. So here is my php script to pull data from my ESP8266 and save the values in a mySQL database.

<?php

include("access.php");
session_start(); //We need to have static variables
$Esp8266SensorURL = $esp8266_9702_GetSensors_URL;

//---------------------------------------------------------------------
//Functions
//---------------------------------------------------------------------
//
//---------------------------------------------------------------------
//Name: get_fcontent( $url, $javascript_loop = 0, $timeout = 5 )
//Function: CURL that gets URL contents
//Parameter 1: $url - URL to retrieve
//Parameter 2: $javascript_loop - IDK, this function pulled from internet
//Parameter 3: $timeout - IDK, this function pulled from internet
//---------------------------------------------------------------------
function get_fcontent( $url, $javascript_loop = 0, $timeout = 15 ) {
    $url = str_replace( "&", "&", urldecode(trim($url)) );
    $cookie = tempnam ("/tmp", "CURLCOOKIE");
    $ch = curl_init();
    curl_setopt( $ch, CURLOPT_URL, $url );
    curl_setopt( $ch, CURLOPT_COOKIEJAR, $cookie );
    curl_setopt( $ch, CURLOPT_FOLLOWLOCATION, true );
    curl_setopt( $ch, CURLOPT_ENCODING, "" );
    curl_setopt( $ch, CURLOPT_RETURNTRANSFER, true );
    curl_setopt( $ch, CURLOPT_AUTOREFERER, true );
    curl_setopt( $ch, CURLOPT_SSL_VERIFYPEER, false ); # required for https urls
    curl_setopt( $ch, CURLOPT_CONNECTTIMEOUT, $timeout );
    curl_setopt( $ch, CURLOPT_TIMEOUT, $timeout );
    curl_setopt( $ch, CURLOPT_MAXREDIRS, 10 );
    $content = curl_exec( $ch );
    $response = curl_getinfo( $ch );
    curl_close ( $ch );

    if ($response['http_code'] == 301 || $response['http_code'] == 302) {
        ini_set("user_agent", "Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.7.3) Gecko/20041001 Firefox/0.10.1");

        if ( $headers = get_headers($response['url']) ) {
            foreach( $headers as $value ) {
                if ( substr( strtolower($value), 0, 9 ) == "location:" )
                    return get_url( trim( substr( $value, 9, strlen($value) ) ) );
            }
        }
    }

    if ( ( preg_match("/>[[:space:]]+window\.location\.replace\('(.*)'\)/i", $content, $value) || preg_match("/>[[:space:]]    +window\.location\=\"(.*)\"/i", $content, $value) ) && $javascript_loop < 5) {
        return get_url( $value[1], $javascript_loop+1 );
    } 
    else {
        return array( $content, $response );
    }
}

//---------------------------------------------------------------------
//Name: getEsp8266Sensor($getUrl,$update,$k)
//Function: Gets value from ESP8266 (Return last value if 0)
//Parameter 1: $getUrl - ESP8266 URL that returned JSON with sensor values
//Parameter 2: $update - if "Y", get JSON from ESP8200, else use last JSON
//Parameter 3: $k - key to retrieve value of (key:value)
//---------------------------------------------------------------------
function getEsp8266Sensor($getUrl,$update,$k) {
    if($update=="Y") {
        $_SESSION['fetches']=0;
        $status[0]="";
        ob_start();
        while((substr($status[0],0,1)!='{')&&($_SESSION['fetches']<20)) {
            $status=get_fcontent($getUrl);
            $_SESSION['fetches']=$_SESSION['fetches']+1;
            ob_flush();
            sleep(5);
        }
        $_SESSION['jsonstr'] = $status[0];
    }
    if($_SESSION['fetches']<20) {
        $status_array = json_decode($_SESSION['jsonstr']); //Decode json
        foreach($status_array as $key => $value) {
            if(strstr($key,$k)) {
                $val = $value;
            }
        }
    }
    else {
        $val=0;
    }
    //return last value if current value is 0
    if($val!=0) {
        return $val;
    }
    else {
        if(strstr($k,"DS_TempInside")) return $last_in;
        if(strstr($k,"DS_TempOutside")) return $last_ou;
        if(strstr($k,"DS_TempAttic")) return $last_at;
        if(strstr($k,"DH_Humidity")) return $last_hu;
        return $val;
    }
}

//Insert "spaces" spaces
function insertSpace($spaces) {
    for ($cnt=0; $cnt<$spaces; $cnt++) {
        $sp .= "&nbsp";
    }
    return $sp;
}

//---------------------------------------------------------------------
//Connect to database
//---------------------------------------------------------------------

$link = mysqli_connect("localhost", $mysqlUser, $mysqlPass, $mysqlUser);
if (mysqli_connect_error()) {
    die("Could not connect to database");
}
//---------------------------------------------------------------------
//Get latest values from database
//---------------------------------------------------------------------
$query = "SELECT * FROM temperature ORDER BY id DESC LIMIT 1";

if($result=mysqli_query($link, $query)) {
    $row = mysqli_fetch_array($result);
    $last_id = $row['id'];
    $last_in = $row['inside'];
    $last_ou = $row['outside'];
    $last_at = $row['attic'];
    $last_hu = $row['humidity'];
}

//---------------------------------------------------------------------
//Get Sensor Data
//---------------------------------------------------------------------

//----------------------------------------
//-----Get current LOCAL time
//----------------------------------------
$now = time();
date_default_timezone_set('America/Los_Angeles');
$localtime_assoc = localtime($now, true);

//Time
$now_hr = sprintf('%02u',$localtime_assoc['tm_hour']);
$now_mn = sprintf('%02u',$localtime_assoc['tm_min']);
$now_sc = sprintf('%02u',$localtime_assoc['tm_sec']);
$now_tm = $now_hr.":".$now_mn.":".$now_sc;

//Date
$now_yr = $localtime_assoc['tm_year']+1900;
$now_mo = sprintf('%02u',$localtime_assoc['tm_mon']+1);
$now_dy = sprintf('%02u',$localtime_assoc['tm_mday']);
$now_dt = $now_yr."-".$now_mo."-".$now_dy;

//Get Temperatures
$temp_in = getEsp8266Sensor($Esp8266SensorURL,"Y","DS_TempInside");
$temp_ou = getEsp8266Sensor($Esp8266SensorURL,"N","DS_TempOutside");
$temp_at = getEsp8266Sensor($Esp8266SensorURL,"N","DS_TempAttic");

//Get Humidity
$temp_hu = getEsp8266Sensor($Esp8266SensorURL,"N","DH_Humidity");

//Get Free Heap and ms since ESP8266 started
$temp_hp = getEsp8266Sensor($Esp8266SensorURL,"N","SYS_Heap");
$temp_tm = getEsp8266Sensor($Esp8266SensorURL,"N","SYS_Time");

//Insert a record into mySql
$query = "INSERT INTO `temperature` (`inside`,`outside`,`attic`,`humidity`,`time`,`date`,`FreeHeap`,`SysTime`,`fetches`)
VALUES('".$temp_in."','".$temp_ou."','".$temp_at."','".$temp_hu."','".$now_tm."','".$now_dt."','".$temp_hp."','".$temp_tm."','".$_SESSION['fetches']."')";
//Report Results
if ($result=mysqli_query($link, $query)) {
    echo('<br>Success<br>');
}
else {
    echo('<br>Failed<br>');
}

?>

Now let’s break this down.

include("access.php");
$Esp8266SensorURL = $esp8266_9702_GetSensors_URL;

The included “access.php” file contains definitions for my URLs used to access my home devices. Maintaining these in a separate file keeps things tidy and hidden. The variable $esp8266_9702_GetSensors_URL is assigned in the access.h file to my ESP8266 that implements a web server listening on port 9702.

session_start(); //We need to have static variables $Esp8266SensorURL = $esp8266_9702_GetSensors_URL;

session_start() enables the use of static variables

function get_fcontent( $url, $javascript_loop = 0, $timeout = 15 )

This function retrieves the contents returned from an URL, using CURL.

function getEsp8266Sensor($getUrl,$update,$k) {
    if($update=="Y") {
        $_SESSION['fetches']=0;
        $status[0]="";
        ob_start();
        while((substr($status[0],0,1)!='{')&&($_SESSION['fetches']<20)) {
            $status=get_fcontent($getUrl);
            $_SESSION['fetches']=$_SESSION['fetches']+1;
            ob_flush();
            sleep(5);
        }
        $_SESSION['jsonstr'] = $status[0];
    }

Now this is the meat of this method. The function getEsp8266Sensor() mechanizes the process of retrieving values from a JSON string returned from an ESP8266. Since a single GET request returns a JSON string that contains all the sensor values, only one request is needed, the first one. The “$update” parameter is set to “Y” from the caller to perform the GET request. The GET is declared a success if the first character (after the header) returned is “{“, the start of the expected JSON string.

I have observed this request to fail sometimes, which is why it is put in a while loop, allowing up to 20 retries. Since one of the values stored in the mySQL is the number of GET requests required before a successful attempt, I have data to characterize the failure rate. After running this script 3 times per hour for a few days now, a maximum of six attempts before success have been recorded. Twenty retries were set to provide some margin. I’ll be reviewing the data as more run time is accumulated.

Notice the ob_start() and ob_flush() statements. This was necessary to clear out the buffers from the last GET request. Without these, the request often failed continuously, regardless of how many retries were attempted.

The session variable $_SESSION[‘jsonstr’] saves the returned JSON string for subsequent calls to this function, when a request is not needed.

The rest of the function simply decodes the JSON string and extracts the requested key value.

The code snippet below retrieves the latest values stored in the database.

//---------------------------------------------------------------------
//Get latest values from database
//---------------------------------------------------------------------
$query = "SELECT * FROM temperature ORDER BY id DESC LIMIT 1";

if($result=mysqli_query($link, $query)) {
    $row = mysqli_fetch_array($result);
    $last_id = $row['id'];
    $last_in = $row['inside'];
    $last_ou = $row['outside'];
    $last_at = $row['attic'];
    $last_hu = $row['humidity'];
}

These values are used if no valid values are received from the ESP8266 during the http GET transaction.

I leave the rest of the code up to you to understand. It is all very straightforward stuff.

Here is an extract from the data stored in mySQL. The data is recorded every hour. This slice is from 3 am to 2 pm April 30,2015. As you can see, the temperatures bottomed at 6 am and the worst case performance for the software was at 9 and 11 am when 6 GET requests were required before it was received successfully.

Wow, things really got toasty in my attic that day! 119.9 F at 1 pm.

Hope you find this information as useful to you as it has been to me. Until next time…

Share This:

Arduino IDE for ESP8266

The Arduino IDE version 1.6.1 with ESP8266 support worked well as a webserver using the WiFiWebServer example sketch as a starting point. This basic example supports on/off control of an LED from a web browser. The example demonstrates how to interpret and respond to requests. It can easily be expanded to support responses any request strings added to the domain:port URL used to access the device. The example, which simply controls an LED, uses domain:port/gpio/0 to turn the LED off and domain:port/gpio/1 to turn it on.

Building my sketch on the back of this example, I soon uncovered issues which had to be addressed in order for my IoT sensor reads to function as required. This included:

1. WiFi Connection

The connection can and sometimes does get dropped. It is important to have a check in the sketch loop() function to reconnect if it is disconnected.

if (WiFi.status() != WL_CONNECTED) {
delay(1);
WiFi.config(ipadd, ipgat, ipsub);
WiFi.begin(ssid, password);
//Wait here until connected, up to a timeout period
while (WiFi.status() != WL_CONNECTED) {
Serial.print(“.”);
delay(500);
}
return;
}

2. Ticker

Ticker is supposed to provide the capability to perform periodic tasks. This must be using the SDK timer/callback API. While it worked flawlessly for me with the SDK, it did not work well with the Arduino loop(). I was trying to separate the sensor reads from the server handler, using the timer for sensor reads. Erratic resets occurred. From my experience, Ticker is not usable at this time. With the current Arduino IDE, it is necessary to handle all code single threaded within the sequence of the loop() function.

3. Watchdog Timer Resets

Disable the watchdog timer and the MPU will halt on the first timeout. It must remain enabled during normal code execution. But I have observed many resets when trying to read attached sensors while also supporting the web server. These resets can be reduced or eliminated by structuring the loop() function with returns after every time consuming task, like reading a sensor. The resets are also reduced by using yield(), delay() and ESP.wdtFeed() statements after time consuming tasks.

4. Math Functions

Using the BMP085 to read Barometric Pressure require use of the pow function. This math function is not supported with the core library and attempts to link the math.h library blows the memory, resulting in a build error. Since the pow function is only used for the calculation of altitude values, the BMP085 will work with the Arduino IDE if you comment out the pow functions in the library and only read the pressure and temperature values from the device driver.

5. JSON Encoding

The JSON library included  with the SDK is not available with the Arduino IDE. And linking a JSON library will blow the memory. Since it is advantageous to return a JSON string with sensor values in response to “http GET” requests, this becomes a significant shortcoming. But a simple JSON encoding function used to build the return string is all that is needed to overcome this. This works with minimal code:

void jsonAdd(String *s, String key,String val) {
*s += ‘”‘ + key + ‘”‘ + “:” + ‘”‘ + val + ‘”‘;
}
void jsonEncode(int pos, String * s, String key, String val) {
switch (pos) {
case ONEJSON:
case FIRSTJSON:
*s += “{\r\n”;
jsonAdd(s,key,val);
*s+= (pos==ONEJSON) ? “\r\n}” : “,\r\n”;
break;
case NEXTJSON:
jsonAdd(s,key,val);
*s+= “,\r\n”;
break;
case LASTJSON:
jsonAdd(s,key,val);
*s+= “\r\n}”;
break;
}
}

6. Watchdog Timer Functionality

A richer API to the watchdog timer is needed. This may just be a wish list, but it would be very helpful if the timeout period could be set and a user defined callback function could be registered for timeouts. This would go a long way in resolving the device resets experienced by many ESP8266 users.

In summary 

The Arduino IDE is a very well-known and supported platform. These shortcomings will probably be overcome with updates in the future. Despite the limited flexibility and functionality of the Arduino IDE, it will likely be the platform of choice for many ESP8266 application developers.

Share This:

Like many, I became very excited when first introduced to the ESP8266. Imagine, a complete IoT system on a chip for a couple of USDs! It was very easy to bring a factory-new ESP8266 module up and running from the wealth of information available on-line. The “OK” response to the “AT” command was the “Hello World” for this device, and on you go from there.

With more education, it became obvious that a lot of folks were using NodeMCU to develop Lua scripts for the device. It looked simple enough. And the demo applications to control an LED from a web browser were easy to duplicate. Wow, this really works!

But then I started to develop a larger server, which responded to multiple requests.

And that’s when the problems started…

Initially, I blamed the problems I encountered on my coding. And a lack of experience working with the module. But it seemed that every time a problem was resolve, another few reared their ugly head. In the end, I have concluded that the NodeMCU/Lua development environment (at least at this time) is not suitable for ESP8266 application development.

Here is why…

#1. Insufficient memory provided for your application

This environment does not compile your application, but rather interprets it while running. And every byte counts, including code comments. They all reduce the available free heap. Anyone that has worked with the NodeMCU/Lua development has encountered the “not enough memory” wall. It does not take much code and you start scrounging for bytes just to get the script to run.

I tried to work around this by creating smaller scripts with limited functionality and switching between them with module resets to start a different lua script. Before resetting, the code would simply write the name of the next script to start in a file, which the lua start-up script would read and call upon reset. Each time the device reset, the free heap was freed to it’s maximum size. The code was restarted based on the request received from a web client. Besides being a clumsy way to code, it was also unreliable as the resets did not always start up correctly.

#2. Web Server stops replying to client

During repeat-ability testing, I noticed that at some point, usually within a few web browser requests, the ESP8266 simply stopped replying to “http GET” requests. I expected the application to respond to the same request in the same manner every time, but this proved not to be the case. Perhaps this might be related to the well known issue with the memory usage lingering, reducing the free heap until the system stops functioning.

#3. Crashes

Many times, I would start the application with a restart. The first time an “http GET” request was sent, the system would crash. Reset it again and it would work. Without changing anything! Ugh…not very repeatable. One can only endure this so long…

#4. Better options

After circling the wagons with NodeMCU/Lua for over a week, insanity set in. I really was expecting different results. But nooo – the problems persisted.

So I investigated other options.

The Espressif SDK offers native ESP8266 support. Your code is developed in C language and compiled.  There is plenty of space available for your application. It seemed to be a far more robust development option than using NodeMCU. With the SDK, you have direct control of the callback function registration and content. And the make file output provides a clear picture of the memory usage and partitioning. But, unfortunately, I discovered some issues with the ESP8266 SDK (Version 1.0) that forced me to abandon it as well, at least for now. I really did like this IDE and anxiously await the problem resolving updates.

I am currently using the Arduino IDE for ESP8266 application development. While not as robust as the SDK, this is the only development environment that I have found (at this time) to work. But here, too, this environment is still under development. The Arduino IDE for ESP8266 has unique issues of it’s own.

Share This:

As I have continued to port my home sensor measurement tasks over the low-cost ESP8266 units, the next driver needed was for the Barometric Pressure Sensor. I have been using the popular BMP085 module for this purpose. The library for this sensor is at:

https://github.com/adafruit/Adafruit-BMP085-Library

Problem: There is one incompatibility using this library with the ESP8266 Arduino IDE platform. When attempting to compile a sketch that uses this library without modification, an error is encountered:

Adafruit_BMP085.cpp:(.text+0x3b8): undefined reference to `pow’

This occurs because the library is not properly linked to the core math.h library.

And changing the platform.txt file linker options to include “-lm” only makes things worse. In that case, the compiler returns an error that there is insufficient space to load the ‘.text’ segment.

Any implementation in c that involves fractional power (such as a double value that is not an even integer) requires the log function, also a math function. And the math.h library does not link correctly when compiling a sketch. So until this is addressed in the Arduino IDE build, the pow function simply cannot be used. Fortunately, pow is only required for the calculated value of altitude. Therefore, the bmp085 altitude features value cannot be used until the Arduino IDE for the ESP8266 supports it.

Solution: For now, BMP085 library can be tweaked slightly to make it usable with this IDE.

What I have not found is an acceptable substitute for the function implemented in c code. I tried some of the recursive algorithms available on the net to replace the missing function. But none of them worked properly. They either hung in endless recursive calls to the function or returned incorrect results.

What does work is the addition of a dummy function to the BMP085 library to replace the math.h “pow” that cannot currently be linked to properly . The name of the function must also be revised to avoid a conflict with the math.h prototype.

Here is the code I added to the BMP085 library, it now compiles properly:

double powr(double a,double b){
return 1.0;
}

Note that the function name was changed from “pow” to “powr”.

And remember to change the function name in the library code. It occurs in two places.

Here:

int32_t Adafruit_BMP085::readSealevelPressure(float altitude_meters) {
float pressure = readPressure();
return (int32_t)(pressure / powpowr(1.0-altitude_meters/44330, 5.255));
}

And here:

float Adafruit_BMP085::readAltitude(float sealevelPressure) {
float altitude;
float pressure = readPressure();
altitude = 44330 * (1.0 – powpowr(pressure /sealevelPressure,0.1903));
return altitude;
}

Remember, this will result in bogus values for altitude. But it will allow you to use the BMP085 device with the ESP8266 Arduino IDE version 1.6.1 to retrieve the sensor’s pressure and temperature readings.

Hope someone finds this information useful…

Share This: