4 reasons I abandoned NodeMCU/Lua for ESP8266

Like many, I became very excited when first introduced to the ESP8266. Imagine, a complete IoT system on a chip for a couple of USDs! It was very easy to bring a factory-new ESP8266 module up and running from the wealth of information available on-line. The “OK” response to the “AT” command was the “Hello World” for this device, and on you go from there.

With more education, it became obvious that a lot of folks were using NodeMCU to develop Lua scripts for the device. It looked simple enough. And the demo applications to control an LED from a web browser were easy to duplicate. Wow, this really works!

But then I started to develop a larger server, which responded to multiple requests.

And that’s when the problems started…

Initially, I blamed the problems I encountered on my coding. And a lack of experience working with the module. But it seemed that every time a problem was resolve, another few reared their ugly head. In the end, I have concluded that the NodeMCU/Lua development environment (at least at this time) is not suitable for ESP8266 application development.

Here is why…

#1. Insufficient memory provided for your application

This environment does not compile your application, but rather interprets it while running. And every byte counts, including code comments. They all reduce the available free heap. Anyone that has worked with the NodeMCU/Lua development has encountered the “not enough memory” wall. It does not take much code and you start scrounging for bytes just to get the script to run.

I tried to work around this by creating smaller scripts with limited functionality and switching between them with module resets to start a different lua script. Before resetting, the code would simply write the name of the next script to start in a file, which the lua start-up script would read and call upon reset. Each time the device reset, the free heap was freed to it’s maximum size. The code was restarted based on the request received from a web client. Besides being a clumsy way to code, it was also unreliable as the resets did not always start up correctly.

#2. Web Server stops replying to client

During repeat-ability testing, I noticed that at some point, usually within a few web browser requests, the ESP8266 simply stopped replying to “http GET” requests. I expected the application to respond to the same request in the same manner every time, but this proved not to be the case. Perhaps this might be related to the well known issue with the memory usage lingering, reducing the free heap until the system stops functioning.

#3. Crashes

Many times, I would start the application with a restart. The first time an “http GET” request was sent, the system would crash. Reset it again and it would work. Without changing anything! Ugh…not very repeatable. One can only endure this so long…

#4. Better options

After circling the wagons with NodeMCU/Lua for over a week, insanity set in. I really was expecting different results. But nooo – the problems persisted.

So I investigated other options.

The Espressif SDK offers native ESP8266 support. Your code is developed in C language and compiled.  There is plenty of space available for your application. It seemed to be a far more robust development option than using NodeMCU. With the SDK, you have direct control of the callback function registration and content. And the make file output provides a clear picture of the memory usage and partitioning. But, unfortunately, I discovered some issues with the ESP8266 SDK (Version 1.0) that forced me to abandon it as well, at least for now. I really did like this IDE and anxiously await the problem resolving updates.

I am currently using the Arduino IDE for ESP8266 application development. While not as robust as the SDK, this is the only development environment that I have found (at this time) to work. But here, too, this environment is still under development. The Arduino IDE for ESP8266 has unique issues of it’s own.

 

 

 

Share This:
Facebooktwittergoogle_plusredditpinterestlinkedintumblrFacebooktwittergoogle_plusredditpinterestlinkedintumblr

34 Responses to 4 reasons I abandoned NodeMCU/Lua for ESP8266

  1. Øyvind says:

    Regarding #1, node.compile() compiles the lua files to bytecode. Dunno if it was available when you wrote this, but that sounds like it’d reduce your troubles at least a little bit.

    • facebook-profile-picture Dave St. Aubin says:

      Thanks for that tip. I will check node.compile() out when I get a chance to compare the memory limits of the NodeMCU environment with the SDK and Arduino IDE. Compiling the node will definitely help optimize memory usage. We shall see how much memory is used with with a simple NodeMCU compiled script truly know how much extra space is available for your application.

      • Øyvind says:

        I’d like to hear the results of that.

        • facebook-profile-picture Dave St. Aubin says:

          If you are having good results with NodeMCU for ESP8266 software development, by all means keep using it. But my experience has been anything but acceptable. For my part, there would have to be major improvements to even consider using it again.

          I looked back on my notes and noticed that I had indeed been using node.compile() to reduce the script sizes. To ensure that the compiled (script.lc) script was run instead of the interpreted (script.lua) script, the .lua files were removed from the ESP after compiling with node.compile(). The heap usage was reduced by about 50% using the compiled files. However, optimum coding, with no indentation or commenting was still needed for successful loading of my application scripts.

          In the spirit of your comment, I just made an honest attempt to revisit the NodeMCU, to collect some metrics.

          After flashing NodeMCU to one of my ESP8266 setups again and attempting to “Send to ESP” a basic script I had used in the past, numerous errors were encountered, reminders to myself why I have abandon NodeMCU. Very frustrating. It just did not seem to be worth the effort to continue trying to bring back to life a script that used to work. But then that is the nature of the NodeMCU/lua platform for ESP8266 that I have experienced. Sometimes things work, sometimes you get issues.

          – panic messages
          – unexplained ESP resets
          – crashes

          Sorry, I cannot afford to squander another minute of precious time fiddling with NodeMCU. In my experience, the SDK and Arduino IDE have been far more stable.

          Allowing you to focus on your IoT visions.

          Here is just a sample of what occured when I attempted to send a working init.lua script to the ESP8266:

          > setup();
          stdin:1: attempt to call global ‘setup’ (a nil value)
          > local m=0;
          > PANIC: unprotected error in call to Lua API (stdin:2: attempt to perform arithmetic on global ‘it’ (a nil value))
          PANIC: unprotected error in call to Lua API (attempt to concatenate a table value)
          PANIC: unprotected error in call to Lua API (attempt to concatenate a romtable value)
          PANIC: unprotected error in call to Lua API (attempt to call a string value)
          ü! ¤„ÿ1ä)Mô1ä)}†1¤ñ…Cá

          – This loaded without error in the past.
          – “Setup()” was a locally defined function
          – PANIC? Really?
          – And then it crashed/reset with ü! ¤„ÿ1ä)Mô1ä)}†1¤ñ…Cá

          One thing I’ve noticed with the ESP8266 is that heap usage lingers, so it needs to be monitored carefully. I was able to retrieve the free heap with all three development platforms for comparison:

          1. NodeMCU (startup, no scripts loaded)………..23032 bytes
          2. EspressIf SDK(Web server running)…………..33856 bytes
          3: Arduino IDE(Web server,7 sensors running).34152 bytes

          While I have found the Arduino IDE to be acceptable, the EspressIf SDK (1.0.1 April 24, 2015) is my development platform of choice. Why? It offers:

          – The largest collection of API functions
          – Lowest level of callbacks for events
          – Detailed compilation report of memory allocation/usage
          – Stable performance

          ESP8266….have fun with it!

          • facebook-profile-picture Dave St. Aubin says:

            Note: EspressIf released SDK version 1.1 on 26-May.2015. I have experiences some problems with it that I have reported to EspressIf; currently awaiting their reply.

            As of today, I have found SDK Version 1.0.1 24-April-2015 to be the best version to work with.

          • Mike says:

            My experience somewhat mirrors yours – until I learned more about both the ESP and Lua

            I have an app that creates an AP, creates a web page to allow you to join the AP and enter your local SSID/password, it then reboots and joins the internet at which time it implements MQTT and starts logging data to and from the ESP.

            Not bad from a tiny little serial transceiver.

            If I want more, I just connect it to an Arduino pro mini – about 20% larger than the ESP and can do what an Arduino can do.

            Horses for courses…

            I will try the Arduino IDE, though… being able to write in C++ would be a boon.

          • Mike says:

            Damn! 10 minutes in and already I can do stuff I found tiresome in Lua… and my prog loads in 12% of available space.

            Thanks Dave!

      • Mr Wiggles says:

        You can build the Nodemcu from source and strip it of the stuff you dont use and freeing up a lot of space.

  2. Horacio says:

    Thanks for the heads up, however, I have been working with the NodeMCU Lua API and it works just fine. The memory limitations can be circumvented by being careful cleaning up. I am also combining NodeMCU Lua and Espressif SDK and coming up with new packaged lua APIs on top of those of NodeMCU and I am achieving really powerful applications.

    • facebook-profile-picture Dave St. Aubin says:

      So your having good results with the NodeMCU Lua API and pleased with the resulting applications? Nice. Sounds like your preferred development approach.

      Glad to hear you are using the EspressIf SDK. They have fixed all the bugs I had encountered along the way to the current (as of June 19,2015) version 1.1.1. The SDK may be somewhat intimidating for those used to using Arduino (in that case, the ESP8266 Arduino IDE is a good alternative), but the SDK is astounding powerful with callbacks exposed for many of the critical system events. Giving you more control and visibility of system events.

      And the Wifi, both in STATION and AP modes, are much more reliable, more robust.

      But the greatest improvement in the SDK for me was the memory management.

      The heap usage no longer lingers before being released. This vital improvement now enables you to perform periodic sensor readings and other tasks at a much higher rate. In the past, with a rapid task loop (timer of 1 second or less), the heap would shrink with every passing cycle (if you did not slow things way down) until the system would either reset or fatally crash.

      It’s nice to have the issues become a fading memory. So you can simply focus on your application development…

  3. Mac says:

    Hi,

    in my opinion the reason why you seams to be disappointed in NodeMCU is because you expect too much from esp8266 chipset it self. This chipset got less than 100kB of RAM, trying to run webserver on it means asking for troubles (to much connections in the same time and it will hang). That’s why I think that this device should only acquire the data and send it to the proper server. In my house I use a VM on my Dell T20 server to collect data and analyze it, serve to other devices, but Raspberry Pi should work as well.

    From my point of view much more important problem with this chipset it self is it power consumption. It’s ok when you’re powering it from mains, but on battery it need to much power. Nowadays everything is wireless or cableless and that mean that almost every portable device run on batteries. I know – I can do deep sleep (not on all ESP modules I think), but it reset the module it self… So it’s more like doing bypass rather than solving the problem. I think that there is a reason why those chips are so cheap. I think that the power management got bugs (on hardware level). So can’t wait to see successor of esp8266 😉

    • micromicro says:

      100k is plenty, talking as someone that used to develop in micros with as little as 1.2k – the problem is the desire to use excessively high level languages on small hardware. 100k of C or assembler can get a shitload done.

  4. xlaser says:

    The author is correct in my opinion.
    Results are very erratic and a lot of time has been wasted looking for solutions.
    I will wait until a stable version becomes available before wasting any more of my time.

  5. Marko Rizvic says:

    You might want to check this out. Im planning to write uart2websocket bridge soon.
    https://github.com/mrizvic/nodemcu-uart2http-exosite
    https://github.com/mrizvic/nodemcu-uart2mqtt

  6. karthikeyan says:

    Try SMING.
    Fast, Arduino compatible functions, C++ code, Performance and no memory issues.
    C++ code written will compile to firmware.

    • facebook-profile-picture InternetOfHomeThings says:

      I’ll put SMING in my “to check out” queue. But I am very pleased with the results I’ve had using the EspressIf SDK and Arduino IDE for ESP8266 code development.

  7. GuyBlom says:

    Whilst I was amazed by the built-in capabilities of such a cheaply priced device, I was quite horrified by the instability of Nodemcu/Lua. I have consistently had the exact same issues as the OP plus quite a few more.

    As others have mentioned, you can prob work around many of these issues and I dare say a portion are due to hardware factors but this doesn’t detract from the fact that NM/lua hogs too much of the heap for a device with such tight memory constraints.
    As a veteran programmer of 25+ yrs I am loathe to waste time working around inherent instability issues and have quickly moved to seeking better alternatives. Whilst the Arduino IDE is great for sketching quick prototypes even just to check proof of concept, even a mildly experienced coder will quickly become frustrated by its broad limitations. I am hoping Espressif have dealt with any outstanding issues as it sounds like a much more robust system. I”ll take a good look at Sming too and return with my (possibly useless) observations in case anyone may be interested.

    Does anyone have any more recent additions to the available options?

    I really hope something substantial becomes available as this device presents some tremendous opportunities and deserves a comprehensive & efficient interface.

    Would love hear of any new/recent entries into the market (and any first impressions).

  8. Marcel says:

    Things have changed quite dramatically with NodeMCU since this post was written:

    – NodeMCU dev branch is no longer based on <1.0 SDKs from Espressif but on 1.4.
    – Memory consumption has been greatly improved. It's much less likely you run out of heap space.
    – Many more modules were contributed by the community.
    – A new governance model and a few more collaborators ensure that development is now more structured and focused. See https://github.com/nodemcu/nodemcu-firmware/issues/719 for details.
    – I wrote a NodeMCU custom build service at http://nodemcu-build.com/ with which you can configure and build a firmware optimized for your use case.
    – I created a Docker image at https://hub.docker.com/r/marcelstoer/nodemcu-build/ with which you can very easily build your own firmware locally.

    So, I suggest you try a recent dev build and re-evaluate (do NOT download the old released binaries from GitHub). I'm quite positive NodeMCU will no give you a much more pleasant experience.

    • roman says:

      I’d like to provide you with a new datapoint:

      I’ve tried both the Espressif AT commands (but was quickly put away by the C) and then Arduino IDE. For what I wanted to do (wifi scale) the arduino IDE was first to work! That was great, however it kept crashing because of the too long cycles – even though I tried many things (watchdog). It is very frustrating but very clearly my own fault – they clearly state that you need to relinquish control to the processor, or you will encounter hardware resets – BUT mind you, the linear loop is implanted on top of the underlying event-based SDK. It was very hard to do right (not impossible, when you are aware of it, you get it right – but it just adds lots of “let’s put this watchdog/sleep call here because i don’t know whether it will fail on me”)

      If you are making decisions as to which project to choose – at minimum, you should understand the implications – there is a big difference between loop and event-based execution!

      I saw that and so I tried the NodeMCU – it was pain (and I’m planning to contribute code/documentation back to the project to make it easier for beginners) – but I’ve seen it is much better/natural way of coding for this device. In the end I’ve invested lots of time into writing a webserver, curl-like downloader, hx711, and humidity/sensor measurements. And they all work! I’m still using 0.96 and it is almost never failing on me because of insufficient heap. But I have to write code that is self-contained and does not leak objects (but if you are a self-respecting software engineer, you should do that anyway).

      And I appreciate the rapid prototyping abilities of NodeMCU and ESPlorer – it is a joy to write a new code and see it working with the sensor. Not the (mind-numbing) code-compile-upload-o-shit-recompile-reupload…. cycle that other projects have to put up with. So if you choose NodeMCU, it is fun! And not necessarily a fragile heap of shit.

      btw: you can also write code in C for nodemcu (and I’ve done that too – that option is there too! – but I’ve taken it only for bit-banging where speed was crucial; all the rest is just lua)

  9. Dave Houston says:

    Take a look at ZBasic for ESP8266. It’s a compiler. See…
    http://www.zbasic.net/doc/ZBasicESP8266.html
    and…
    http://www.zbasic.net/doc/ZBasicSysLib.html
    There are also PDF versions at…
    http://www.zbasic.net/downloads.php

  10. Zhao says:

    Hello, I had same experience with you when I use NodeMcu, but when I change to Arduino it is similar!
    When I try to control the LED by webserver, the response time take few seconds, some times no more response!

    Here is the code, changed from the example:
    /*
    * This sketch demonstrates how to set up a simple HTTP-like server.
    * The server will set a GPIO pin depending on the request
    * http://server_ip/gpio/0 will set the GPIO2 low,
    * http://server_ip/gpio/1 will set the GPIO2 high
    * server_ip is the IP address of the ESP8266 module, will be
    * printed to Serial when the module is connected.
    */

    #include

    const char* ssid = “EasyBox-150000”;
    const char* pass = “sKYFUWy7k”;

    // Create an instance of the server
    // specify the port to listen on as an argument
    WiFiServer server(80);

    void setup() {
    Serial.begin(115200);
    delay(10);

    // prepare GPIO2
    pinMode(2, OUTPUT);
    digitalWrite(2, 0);

    // Connect to WiFi network
    Serial.println();
    Serial.println();
    Serial.print(“Connecting to “);
    Serial.println(ssid);

    WiFi.begin(ssid, pass);

    while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(“.”);
    }
    Serial.println(“”);
    Serial.println(“WiFi connected”);

    // Start the server
    server.begin();
    Serial.println(“Server started”);

    // Print the IP address
    Serial.println(WiFi.localIP());
    }

    void loop() {
    // Check if a client has connected
    WiFiClient client = server.available();
    if (!client) {
    return;
    }

    if (client.connected()) {
    Serial.println(“Connected to client”);
    }

    // Wait until the client sends some data
    Serial.println(“new client”);
    while(!client.available()){
    delay(1);
    }

    // Read the first line of the request
    String req = client.readStringUntil(‘\r’);
    Serial.println(req);
    client.flush();

    // Prepare the response
    String buf = “Web based LED Controller.Turn LED “;
    String _on,_off;

    if (req.indexOf(“pin=ON”) != -1)
    {
    _on = ” selected=true”;
    _off = “”;
    digitalWrite(2, 1);
    Serial.println(“LED: ON”);
    }
    else if (req.indexOf(“pin=OFF”) != -1)
    {
    _on = “”;
    _off = ” selected=true”;
    digitalWrite(2,0);
    Serial.println(“LED: OFF”);
    }
    else
    {
    int st = digitalRead(2);
    if( st == 1 )
    _on = ” selected=true”;
    else if( st == 0 )
    _off = ” selected=true”;
    }
    String s = buf + “OFFON”;

    // Send the response to the client
    client.print(s);

    delay(1);
    Serial.println(“Client disonnected”);
    client.stop();
    // The client will actually be disconnected
    // when the function returns and ‘client’ object is detroyed
    }

    • facebook-profile-picture InternetOfHomeThings says:

      I have discovered an absolutely unacceptable behavior using “polling for connections” in loop(). It explains why the server stops responding. Here is the problem:

      Every time a server connection is made, memory is taken from the ESP8266 heap, which is NOT released when the connection is closed. This means that the heap gets smaller and smaller each time the server is accessed until the ESP8266 crashes or resets.

      Here is how to confirm this with the sketch you posted in your comment:

      1. Add this to the top of the sketch:

      extern “C” {
      #include “user_interface.h”
      }

      2. Add this after the line “String s = buf + “OFFON”;”

      s += “\r\n\r\nHeap available: “;
      s += system_get_free_heap_size();

      – The first add gives the Arduino IDE access to the SDK system_get_free_heap_size() function.
      – The second add displays the free heap in your web browser when the access is made.

      For this reason, I no longer use the standard Arduino IDE web server you find in many examples on-line.

      I have found using the SDK API, which uses callbacks instead of polling in the loop() function to be stable and does not reduce the free heap after a connection is made and closed. Here is my post with an example server using the Espressif SDK API:

      ESP8266 Arduino IDE Web Server Using Callbacks

  11. I also just “made the switch” away from NodeMCU and ESPlorer. Even simple prototype code behaves erratically and this thread, amongst other things I read, convinced that for now it is not a suitable approach for me.

    I do hope that things improve, though, because it is a great prototyping system and will bring people into the game that would otherwise stay away. The latter would, similar to what Arduino did, be a huge benefit for everybody IMHO.

    • facebook-profile-picture InternetOfHomeThings says:

      There are many that claim good results using NodeMCU and that many improvements have been made since I have last used it. But it sounds like your experience is more recent. Yet still unacceptable.

      Since I’ve moved on, the frustrations of resets, lock-ups, and insufficient memory are also gone. Leaving me with little motivation to ever consider using NodeMCU again. Time is way to precious. From my experience, using the Arduino IDE for ESP8266 prototyping very closely correlates with development of genuine Arduino board sketches.

      Having said that, my favorite ESP8266 development platform is the EspressIF SDK using the Eclipse IDE. Powerful, efficient, and very stable. But it is somewhat more difficult to initially setup which scares off a fair number of would-be users.

      So most of my blog post use the Arduino IDE. It is the platform that enjoys the largest audience. It also provides a mechanism to import many of the EspressIf SDK features.

      • Chris says:

        Very interested post… thanks.
        I’ve started with ESP8266 this week and as a old java/Eclipse programmer I got inclined to use CDT. But it was a bit hard to understand how to setup xtensa and other dependencies. The use of open-esp-sdk turns things more clear, but even if I’ve passed lot of time in order to make my first code.
        Perhaps you could write an article teaching us to setup a environment using linux + eclipse cdt + Espressif sdk ? 😉

        • facebook-profile-picture InternetOfHomeThings says:

          Thanks for your comment. Hmmm. Never tried using Linux with the the EspressIf SDK. Windows-Eclipse-EspressIf/SDK was too easy to setup for me to bother trying to use it with Linux. But thanks again for the suggestion, I may try it as another alternate envirnment.

  12. marcelo says:

    I agree with most of comments here.. I find the 8266 tool chain pretty unstable and erratic. Another big issue is OTA….. i found 8266basic.com which promises internet OTA ..installed it…and was deeply frustrated to see this is 10 times more unstable and unrelayable than Node MCU (don’t kow if aanyone else has tried it)….last but no least….forums and blogs are very poor..(to be honest..this one is the first time I take the time to read comments form first to last!)…guess we have a couple of years to go before we have stable platafomrs for the iot…

  13. Andy says:

    I think that it’s an easy way in to the hardware, but still needs work for use in final production. Most of the “panics” I’ve had are because of the buffer sizes but there are workarounds for these.

    You should check out this video by Angus Gratton – https://www.youtube.com/watch?v=AMMOrwqSPwY – He explains how the ESP8266 has gotten this far, and why some things are the way they are. Not great but I definitely think there’s hope after watching it.

    I’m looking forward to a proper MicroPython port now it’s got funding – https://www.kickstarter.com/projects/214379695/micropython-on-the-esp8266-beautifully-easy-iot

    ……and considering I got 5 of these chips for £10 recently, I’m prepared to put up with some quirks 🙂

    (There’s also the ESP32 coming soon!)

    • facebook-profile-picture InternetOfHomeThings says:

      Thanks for your comment and links. The video was great. I have enjoyed many successes developing projects with the ESP8266 over the past year. The progress was exponentially accelerated after dumping the NodeMCU platform. Both the EspressIf SDK and the Arduino IDE wrapper provide solid development environments for this powerful SoC.

  14. jorge blanco says:

    So what is the solution? any recommendations on other hardware?

    • facebook-profile-picture InternetOfHomeThings says:

      I have had great success using both the Arduino IDE and native EspressIF SDK for ESP8266 code development. The Arduino IDE is easier to use for those that are not comfortable getting too deep into the details. The SDK unleashes the full power of the ESP8266, but some find it too complicated. I recommend starting with the Arduino IDE. And as a personal plug, check out some of my articles with specific Arduino IDE example code. Have fun with your projects!

  15. Fraser says:

    I find NodeMCU/Lua a good firmware and have had none of the problems you list – Indeed I think the problems you are facing are simply due to 2 things.

    #1 Bad, clumsy, code
    #2 Not keeping it simple

    Most important to realise is that the ESP8266 IS NOT a general purpose device – “If you are trying to implement a user-interface or HTTP webserver in your ESP8266 then you are really abusing its intended purpose. When it comes to scoping your ESP8266 applications, the adage Keep It Simple Stupid truly applies.” – Terry Ellison, nodemcu-firmware maintainer,

    Finally if size/memory is an issue – try http://luaforge.net/projects/luasrcdiet/ – it reduces the size of Lua 5.1.x source files by aggressively removing all unnecessary whitespace and comments, optimizing constant tokens, and renaming local variables to shorter names.

    Have fun!

    • facebook-profile-picture InternetOfHomeThings says:

      I am glad you are enjoying your experiences using the ESP8266 with the NodeMCU/Lua firmware.

      If you have never developed code for the ESP8266 using the Arduino IDE, you should consider some test projects. I have successfully developed projects that behave reliably with massive capacity for expansion as an http server and can render responsive bootstrap pages without any memory issues whatsoever. Many possibilities are opened up that are unthinkable using NodeMCU/Lua firmware.

      And to fully unleash the power, the EspressIf SDK should be used. While it requires a stronger software background, the SDK provides the greatest level of control of the hardware.

Trackbacks/Pingbacks

  1. ESP8266 – Developer board for the ESP-01 and ESP-12 | Primal Cortex's Weblog
  2. ESP8266 and Stability of NodeMCU Firmware | Christoph Jahn

Leave a Reply