Okay, so you got your ultra low-cost ESP8266 workhorse setup to read some sensors. And you have an awesome webpage built to visually display the readings. But how do you get the data from the ESP8266 to your website?
PHP and/or Javascript are the two most popular methods. I have used both.
This writing presents a simple Javascript solution. I have also posted an approach to this task using PHP here.
My need came about while developing a mobile app to monitor and control devices in my home. My personal IoTs. Using the Apache Cordova API with the Eclipse IDE. I was developing my App with just HTML, CSS, and JavaScript. Yet it will compile and install onto my Android device like any other application. It is my understanding that the same code can be built into an iOS App as well…but…being strictly a PC user, I have not attempted that.
And let’s get back to using Javascript to pull information from an ESP8266 based web server.
The thing was, I needed to pull my weather, sprinkler, garage and other sensor data from the ESP8266 circuit that monitors these devices to my App. I also wanted maintain control of the data. To be free from a cloud middleman such as ThingsSpeak.
With this in mind, we have two sides to this solution; The micro-controller unit (MCU), which is the ESP8266 in this case, and the client App. The ESP8266 must read all the devices connected to it and then send this data, on request, to the App client. The preferred format is either JSON or XML.
I use JSON strings.
Here is an example of a short function that can be used to create a JSON string in an Arduino IDE sketch. It is critical to include the http header information provided in this example. The header authorizes your domain, allowing the Javascript AJAX to retrieve the ESP8266 http response.
Here is the Javascript with AJAX code used to request a JSON string from the ESP8266, decode it, and display the values on a webpage.
function updateWeatherSensorData() {
requestURL = "http://myrouterdomain.com:9999/?request=GetSensors";
if ( typeof updateWeatherSensorData.timeout == 'undefined' ) {
// It has not... perform the initialization
updateWeatherSensorData.timeout = 0;
}
//Get Weather Sensor Value
$.ajax({
url: requestURL,
error: function(error){
if(updateWeatherSensorData.timeout++ <10) {
setTimeout(updateWeatherSensorData, 1000);
}
else {
updateWeatherSensorData.timeout = 0;
}
},
success: function(thedata){
$("#baro").html(thedata.B_Pressure + " inHg");
$("#tempIn").html(thedata.DS_TempInside + " F");
$("#tempOut").html(thedata.DS_TempOutside + " F");
$("#tempAttic").html(thedata.DS_TempAttic + " F");
$("#humidity").html(thedata.DH_Humidity + " %");
updateWeatherSensorData.timeout = 0;
},
timeout: 7000 // sets timeout to 7 seconds
});
}
I am assuming you have a domain name pointed to your router and your router is configured to port forward internet requests. For reference, here is how to set this up.
How this Javascript code works:
First, the URL used to send the http GET request to the ESP8266 is defined. This URL has 3 parts
- The domain name: http://myrouterdomain.com
- The port your ESP8266 web server is listening on: 9999
- The request string recognized by the web server: /?request=GetSensors
Then we need to define a timeout property for this function. The timeout is actually used to store the number of attempts to retrieve the JSON from the ESP8266 before successful.
And now, the AJAX call. As you can see, the code is structured like a JSON string, with 4 keys and 4 values:
- url: This defines where to send the request.
- error: In case the request fails, the call is repeated, up to 10 times
- success: When successful, the webpage elements with the indicated ID are filled with the value retrieved from the ESP8266
- timeout: Up to 7 seconds is allowed before a timeout error is declared.
That’s it! This is what I have been using to retrieve JSON strings from my ESP8266 using an AJAX call.
In Summary
Here are a few closing thoughts of my observations for your information…
- The error handling is essential. This call fails frequently, perhaps every 2 or 3 requests.
- Most of the time, the error is because a blank string is filled from this call. I do not know if that is an ESP8266 issue or a problem with AJAX.
- The timeout was set to 7 seconds because I have seen it take up to 4 seconds to get a response.
Hope you find this information useful for your ESP8266, Arduino and other MPU projects.
Share This:
i love your instruction. is it possible to post the code in the html to call the updateWeatherSensorData() function every 10 seconds without refreshing the whole page. thanks!
Thanks for your comment.
Here is the html to initiate the sensor data refresh without reloading the entire page:
With this added to the html <body> tag, updateWeatherSensorData() is called immediately after the page is loaded and at a 7 second interval after that. This interval can be modified, of course, by altering the JavaScript function timeout.
And if you wish to embed the JavaScript function in the html, just add it to the <head> tag:
Interesting…did you ever figure out the reason for the blank HTTP requests? I'm having the same problem. I'm using an ESP8266-12E in the Arduino 1.6.10 IDE with the following related libraries:
#include <SoftwareSerial.h>
#include <Wire.h>
#include <ESP8266WiFi.h> //ESP8266 Core WiFi Library (you most likely already have this in your sketch)
#include <DNSServer.h> //Local DNS Server used for redirecting all requests to the configuration portal
#include <ESP8266WebServer.h> //Local WebServer used to serve the configuration portal
#include <ESP8266HTTPClient.h> //HTTP Client to Post to Internet
#include <WiFiManager.h> //https://github.com/tzapu/WiFiManager WiFi Configuration Magic
Every now and then I get a an empt (e.g "") HTTP request. Here is the basic code I'm using:
WiFiClient client = server.available(); // Check if a HTTP client has connected.
if (client) { // If so, then process HTTP request.
String request = client.readStringUntil('\r');
request.trim();
String s = "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nAccess-Control-Allow-Origin: *\r\n\r\n<!DOCTYPE HTML>\r\n<html>\r\n";
// Process HTTP reqeust
if (request == "") { // Do nothing with empty request, was causing problem.
return;
}
else if (request == "GET /trigger/start HTTP/1.1") {
// Run my code here
}
s += "started"; // Add to string to be returned to client.
}
client.print(s);
Here is a link to the function where the problem MIGHT be: https://github.com/esp8266/Arduino/blob/889775c6fe3c23a390cfbb0b57e6de371292a361/libraries/ArduinoOTA/ArduinoOTA.cpp#L159
I'm not really thinking that .readStringUntil() is the problem, but I'm not sure where else to look.
When I get a valid response I'm it's taking like 60 ms to respond but when I get a blank request it's closer to 1.1 seconds, which is too frequent for my application.
Any suggestions on this?
Having a look at the http headers when a blank request is received might reveal what is going on. My gut tells me these might be 'favicon' requests initiated by the Web browser.
I should have access to my ESP8266 gadgets to check this out Saturday. Please share you findings here if you unmask the mystery first.
Actually, I do get favicon requests which process normally. The empty ones are different and cause the problem. Makes me wonder if it's hanging because the terminating character isn't being sent, which would make sense if it was an errant empty call.
FYI, as part of my debugging I printed out the request before it was processed. I got so many favicon requests that I'm tempted to respond to those with a link or file just to have it stop bothering me, but didn't bother because I'm not accessing it from a web browser but from an application running in Chrome. That application doesn't bother me with favicon requests but still has an occasional empty request.
By the way, I wonder if it could be a problem with the HTTP request having TWO terminating characters rather than one. I wonder if that would show up as a second request, even if I did a "flush".
I had a closer look at my original http server setup and what I have been using now after significant improvements. My conclusion is that there is a high probability that your problem is the result of using a polling implementation for your http server (checking for http requests in the loop() function). This is a very bad idea, I had experienced lost requests and request response delays as you have noted when polling. A much better approach is to start a separate thread that responds to requests in an event callback function. I compared the results using http server that polls vs one that uses an event callback. The polling server did exhibit response delays of 1-3 seconds every two or 3 requests while the event callback server responded instantly, every time. And this was using a simple browser to make the requests.
I recommend you revise your ESP8266 sketch to use an http server with an event callback, eliminating the polling in the loop() function and see if your problem goes away. Here is my implementation of this type of server. http://wp.me/p5NRQ8-ku"
My guess is that the ESP8266 http server does not finish processing a request when the polling loop detects another request (the same request, only now it has a blank payload). This will not happen with callbacks.
Or, what if there are additional set of characters after the terminating character?
Sorry for three posts on this…right after clicking Post I had additional thoughts.
Thanks for the reply! I went to Github and downloaded internetofhomethings/ESP8266-Arduino-Ide-Web-Server-Using-SDK-API and tried to compile web_server.ino (Arduino 1.6.10) but ended up with the error messages below. I checked to make sure 'Utility Library' and 'webserver' appeared in the IDE under Manage Libraries. Likely I am missing something, but I'm a bit intimidated to start playing with memory manager libraries. 🙂
Any suggestions on what I might be doing wrong? Possibly I don't need to get the example to compile, but I was using it as a test to make sure I had things set up properly. All I really want to do is to cover my web server over from polling to callbacks as you recommended.
Thanks!
Mark
========================
In file included from /Users/Mark/Dropbox/Programming/Arduino/Sketches/web_server/web_server.ino:19:0:
/Users/Mark/Dropbox/Programming/Arduino/Sketches/libraries/webserver/mem_manager.h:72:40: error: declaration of C function 'void* pvPortMalloc(size_t)' conflicts with
void *pvPortMalloc( size_t xWantedSize ) ;//ICACHE_FLASH_ATTR;
^
In file included from /Users/Mark/Library/Arduino15/packages/esp8266/hardware/esp8266/2.3.0/cores/esp8266/pgmspace.h:12:0,
from /Users/Mark/Library/Arduino15/packages/esp8266/hardware/esp8266/2.3.0/cores/esp8266/Arduino.h:240,
from sketch/web_server.ino.cpp:1:
/Users/Mark/Library/Arduino15/packages/esp8266/hardware/esp8266/2.3.0/tools/sdk/include/ets_sys.h:160:7: error: previous declaration 'void* pvPortMalloc(size_t, const char*, int)' here
void *pvPortMalloc(size_t xWantedSize, const char* file, int line) __attribute__((malloc, alloc_size(1)));
^
In file included from /Users/Mark/Dropbox/Programming/Arduino/Sketches/web_server/web_server.ino:19:0:
/Users/Mark/Dropbox/Programming/Arduino/Sketches/libraries/webserver/mem_manager.h:74:26: error: declaration of C function 'void vPortFree(void*)' conflicts with
void vPortFree( void *pv ) ;//ICACHE_FLASH_ATTR;
^
In file included from /Users/Mark/Library/Arduino15/packages/esp8266/hardware/esp8266/2.3.0/cores/esp8266/pgmspace.h:12:0,
from /Users/Mark/Library/Arduino15/packages/esp8266/hardware/esp8266/2.3.0/cores/esp8266/Arduino.h:240,
from sketch/web_server.ino.cpp:1:
/Users/Mark/Library/Arduino15/packages/esp8266/hardware/esp8266/2.3.0/tools/sdk/include/ets_sys.h:162:6: error: previous declaration 'void vPortFree(void*, const char*, int)' here
void vPortFree(void *ptr, const char* file, int line);
^
In file included from /Users/Mark/Dropbox/Programming/Arduino/Sketches/web_server/web_server.ino:20:0:
/Users/Mark/Dropbox/Programming/Arduino/Sketches/web_server/web_server.ino: In function 'void SdkWebServer_recv(void*, char*, short unsigned int)':
/Users/Mark/Library/Arduino15/packages/esp8266/hardware/esp8266/2.3.0/tools/sdk/include/mem.h:20:48: error: too many arguments to function 'void* pvPortZalloc(int)'
#define os_zalloc(s) pvPortZalloc(s, "", 0)
^
/Users/Mark/Dropbox/Programming/Arduino/Sketches/web_server/web_server.ino:197:31: note: in expansion of macro 'os_zalloc'
pURL_Param = (URL_Param *)os_zalloc(sizeof(URL_Param));
^
/Users/Mark/Dropbox/Programming/Arduino/Sketches/web_server/web_server.ino:25:8: note: declared here
void * pvPortZalloc(int size);
^
In file included from /Users/Mark/Dropbox/Programming/Arduino/Sketches/web_server/web_server.ino:20:0:
/Users/Mark/Library/Arduino15/packages/esp8266/hardware/esp8266/2.3.0/tools/sdk/include/mem.h:20:48: error: too many arguments to function 'void* pvPortZalloc(int)'
#define os_zalloc(s) pvPortZalloc(s, "", 0)
^
/Users/Mark/Dropbox/Programming/Arduino/Sketches/web_server/web_server.ino:206:19: note: in expansion of macro 'os_zalloc'
pRx = (char *)os_zalloc(512);
^
/Users/Mark/Dropbox/Programming/Arduino/Sketches/web_server/web_server.ino:25:8: note: declared here
void * pvPortZalloc(int size);
^
In file included from /Users/Mark/Dropbox/Programming/Arduino/Sketches/web_server/web_server.ino:20:0:
/Users/Mark/Dropbox/Programming/Arduino/Sketches/web_server/web_server.ino: In function 'void SdkWebServer_senddata(void*, bool, char*)':
/Users/Mark/Library/Arduino15/packages/esp8266/hardware/esp8266/2.3.0/tools/sdk/include/mem.h:20:48: error: too many arguments to function 'void* pvPortZalloc(int)'
#define os_zalloc(s) pvPortZalloc(s, "", 0)
^
/Users/Mark/Dropbox/Programming/Arduino/Sketches/web_server/web_server.ino:316:28: note: in expansion of macro 'os_zalloc'
pbuf = (char *)os_zalloc(length + 1);
^
/Users/Mark/Dropbox/Programming/Arduino/Sketches/web_server/web_server.ino:25:8: note: declared here
void * pvPortZalloc(int size);
^
In file included from /Users/Mark/Dropbox/Programming/Arduino/Sketches/web_server/web_server.ino:20:0:
/Users/Mark/Dropbox/Programming/Arduino/Sketches/web_server/web_server.ino: In function 'bool SdkWebServer_savedata(char*, uint16)':
/Users/Mark/Library/Arduino15/packages/esp8266/hardware/esp8266/2.3.0/tools/sdk/include/mem.h:20:48: error: too many arguments to function 'void* pvPortZalloc(int)'
#define os_zalloc(s) pvPortZalloc(s, "", 0)
^
/Users/Mark/Dropbox/Programming/Arduino/Sketches/web_server/web_server.ino:386:33: note: in expansion of macro 'os_zalloc'
precvbuffer = (char *)os_zalloc(headlength + 1);
^
/Users/Mark/Dropbox/Programming/Arduino/Sketches/web_server/web_server.ino:25:8: note: declared here
void * pvPortZalloc(int size);
^
In file included from /Users/Mark/Dropbox/Programming/Arduino/Sketches/web_server/web_server.ino:20:0:
/Users/Mark/Library/Arduino15/packages/esp8266/hardware/esp8266/2.3.0/tools/sdk/include/mem.h:20:48: error: too many arguments to function 'void* pvPortZalloc(int)'
#define os_zalloc(s) pvPortZalloc(s, "", 0)
^
/Users/Mark/Dropbox/Programming/Arduino/Sketches/web_server/web_server.ino:389:33: note: in expansion of macro 'os_zalloc'
precvbuffer = (char *)os_zalloc(dat_sumlength + headlength + 1);
^
/Users/Mark/Dropbox/Programming/Arduino/Sketches/web_server/web_server.ino:25:8: note: declared here
void * pvPortZalloc(int size);
^
In file included from /Users/Mark/Dropbox/Programming/Arduino/Sketches/web_server/web_server.ino:20:0:
/Users/Mark/Dropbox/Programming/Arduino/Sketches/web_server/web_server.ino: In function 'void SdkWebServer_parse_url_params(char*, URL_Param*)':
/Users/Mark/Library/Arduino15/packages/esp8266/hardware/esp8266/2.3.0/tools/sdk/include/mem.h:20:48: error: too many arguments to function 'void* pvPortZalloc(int)'
#define os_zalloc(s) pvPortZalloc(s, "", 0)
^
/Users/Mark/Dropbox/Programming/Arduino/Sketches/web_server/web_server.ino:440:26: note: in expansion of macro 'os_zalloc'
pbufer = (char *)os_zalloc(length + 1);
^
/Users/Mark/Dropbox/Programming/Arduino/Sketches/web_server/web_server.ino:25:8: note: declared here
void * pvPortZalloc(int size);
^
exit status 1
Error compiling for board NodeMCU 1.0 (ESP-12E Module).
I upgraded to Arduino IDE 1.6.12 which appears to be the latest at this time.
With this version of the IDE, there are two changes needed to the sketch in order to get a clean compile:
1: Near the top of the sketch, comment and change the line:
//void * pvPortZalloc(int size);
void * pvPortZalloc(int size,char *, int);
2: In the function "SdkWebServer_parse_url_params", comment and change the following line:
//pbuffer = (char *)os_strstr(precv, "Accept:");
pbuffer = (char *)os_strstr(precv, "Connection:");
After compiling and uploading the sketch to an ESP8266, prompt replies were repeatedly observed to the request:
192.168.0.132:9701/?request=GetSensors
Of course, you can set the IP and port to any value you want in the sketch.
Hope you get good results with this.
Dave
Dave,
I'm starting to think the problem is on my side. I made the changes above (in 1.6.10) but still get the error messages below. Where should I look next?
(FYI, about to be on vacation for a week so I will be out of pocket till then.)
Thanks again for your help!!!
Mark
*********ERRORS**********
In file included from /Users/Mark/Dropbox/Programming/Arduino/Sketches/web_server/web_server.ino:19:0:
/Users/Mark/Dropbox/Programming/Arduino/Sketches/libraries/webserver/mem_manager.h:72:40: error: declaration of C function 'void* pvPortMalloc(size_t)' conflicts with
void *pvPortMalloc( size_t xWantedSize ) ;//ICACHE_FLASH_ATTR;
^
In file included from /Users/Mark/Library/Arduino15/packages/esp8266/hardware/esp8266/2.3.0/cores/esp8266/pgmspace.h:12:0,
from /Users/Mark/Library/Arduino15/packages/esp8266/hardware/esp8266/2.3.0/cores/esp8266/Arduino.h:240,
from sketch/web_server.ino.cpp:1:
/Users/Mark/Library/Arduino15/packages/esp8266/hardware/esp8266/2.3.0/tools/sdk/include/ets_sys.h:160:7: error: previous declaration 'void* pvPortMalloc(size_t, const char*, int)' here
void *pvPortMalloc(size_t xWantedSize, const char* file, int line) __attribute__((malloc, alloc_size(1)));
^
In file included from /Users/Mark/Dropbox/Programming/Arduino/Sketches/web_server/web_server.ino:19:0:
/Users/Mark/Dropbox/Programming/Arduino/Sketches/libraries/webserver/mem_manager.h:74:26: error: declaration of C function 'void vPortFree(void*)' conflicts with
void vPortFree( void *pv ) ;//ICACHE_FLASH_ATTR;
^
In file included from /Users/Mark/Library/Arduino15/packages/esp8266/hardware/esp8266/2.3.0/cores/esp8266/pgmspace.h:12:0,
from /Users/Mark/Library/Arduino15/packages/esp8266/hardware/esp8266/2.3.0/cores/esp8266/Arduino.h:240,
from sketch/web_server.ino.cpp:1:
/Users/Mark/Library/Arduino15/packages/esp8266/hardware/esp8266/2.3.0/tools/sdk/include/ets_sys.h:162:6: error: previous declaration 'void vPortFree(void*, const char*, int)' here
void vPortFree(void *ptr, const char* file, int line);
^
exit status 1
Error compiling for board NodeMCU 1.0 (ESP-12E Module).
Somewhere along the Arduino IDE updates, the libraries included some of the declarations that I had to include manually in the mem_manager.h file. From the errors you have listed, it appears that some of the statements in mem_manager.h that had to be commented out to avoid a duplication with the IDE were never included with my GitHub repository. Things just keep changing!
To address your immediate concern, I will send you a private email message with the updated mem_manager.h file. Then, when I get a chance, I'll update the GitHub repository. Hope this corrects your errors that must be frustrating to experience.
Dave