Archives for App Inventor configurable MQTT

MQTT for App Inventor – More Configurable Settings

26th July 2016 by InternetOfHomeThings in All Posts, ESP8266 : An IoT System on Chip, Internet of Things, MQTT | 9 Comments

configs

More MQTT Configurable Parameters via App Inventor

My last post on this subject introduced a few configurable setting to an MQTT App Inventor project. It was a no frills, bare bones version. But a few essential parameters were  left out. This article fills in the some of the gaps, including connection authentication, websocket port assignment, timeout, last will and testimonial, and the clientID. With this update, the following MQTT settings become configurable:

  • MQTT Broker domain name
  • Request Topic
  • Reply Topic
  • User Name (Leave Blank if not used)
  • Password (Leave Blank if not used)
  • MQTT Websocket Port
  • Client ID
  • Keep Alive Timeout (seconds)
  • Last Will Topic
  • Last Will Message
  • Last Will QoS

NOTE: If you want to skip the implementation details presented below and simply use this project now, it is available on GitHub here.

A New Client Library

My initial App Inventor MQTT project used the Mosquitto client library. While that was great for developing the “proof-of-concept”, demonstrating that you could indeed link an App Inventor project to an MQTT broker, some serious shortcomings soon became evident.

When attempting to refine the project to support broker connections with authentication, I discovered that the Mosquitto library did not support this basic feature. So a new library was needed. Fortunately, an existing open-source client library is available with full support for password-enabled logins.

The library is called Eclipse Paho. And just as with my initial application, the API provides a JavaScript interface. This interface supports all the features identified above, configurable through App Inventor.

Expanding The Configurable Parameters

Here is how the expanded App Inventor configuration screen looks. When user name/password credentials are not used, the fields should remain blank. Just as before, the configuration settings are stored in an Android device file. Upon start-up, the system initializes with the values stored in that file (cfg.txt).

mqtt_cfg

A last will message has been added to the set of App Inventor configurable parameters. As per the MQTT specification, the last will message is sent when the MQTT connection is closing.

Test Case

The HiveMQ broker “broker.mqttdashboard.com” was used to demonstrate the capabilities of this project. And the on-line client used is available at: http://www.hivemq.com/demos/websocket-client/

A simple test: Open a browser to the MQTT client: http://www.hivemq.com/demos/websocket-client/

Enter the Host broker as shown below and click “Connect”.

hiveMQ

Subscribe to the App Inventor MQTT Request and Last Will topics:

Request Topic: mqtt_request

Last Will Topic: lwt

subscribe

Open the updated MQTT App on your Android device. Click on the “Configure MQTT” icon.

new_cfg

Edit or take note of the last will topic and message. Close the App Inventor app and verify the HiveMQ client displays the last will message.

Note: Use a broker that supports username/password logins to verify that new configurable parameter pair. Refer to this post if you would like to setup your own MQTT broker with login authentication enabled.

JavaScript Updates

The App Inventor WebViewString is used to communicate between the application and the Paho MQTT JavaScript library. Here are the highlights of the revised JavaScript interface.

Paho MQTT API Fork

I started this project update with paho-javascript version 1.0.2. After debugging the changes using a web browser, the code was moved to the target Android device. Unfortunately, after much troubleshooting, an unsupported AppInventor “WebViewer” component feature was identified.  The  required feature for the Paho library is called “localStorage”.

Fortunately, there is an alternative to localStorage. I have modified the Paho library to use Cookies instead of localStorage. With this change, the Paho library/App Inventor communication has been verified to function properly.

Three sections of the library had to be modified:

  1. Verify localStorage is supported (Comment out this check)
 
 
  1. // Check dependencies are satisfied in this browser.
  2. if (!("WebSocket" in global && global["WebSocket"] !== null)) {
  3. throw new Error(format(ERROR.UNSUPPORTED, ["WebSocket"]));
  4. }
  5. //cookies used since localstorage not supported with appinventor
  6. /*
  7. if (!("localStorage" in global && global["localStorage"] !== null)) {
  8. throw new Error(format(ERROR.UNSUPPORTED, ["localStorage"]));
  9. }
  10. */

2. Replace localStorage with cookies

 
 
  1. //localStorage.setItem(prefix+this._localKey+wireMessage.messageIdentifier, JSON.stringify(storedMessage));
  2. setCookie(prefix+this._localKey+wireMessage.messageIdentifier, JSON.stringify(storedMessage), 1);
  3. };
  4. ClientImpl.prototype.restore = function(key) {
  5. //var value = localStorage.getItem(key);
  6. var value = getCookie(key);

3. Add Get/Set cookies functions

 
 
  1. function setCookie(cname, cvalue, exdays) {
  2.     var d = new Date();
  3.     d.setTime(d.getTime() + (exdays*24*60*60*1000));
  4.     var expires = "expires="+d.toUTCString();
  5.     document.cookie = cname + "=" + cvalue + "; " + expires;
  6. }
  7. function getCookie(cname) {
  8.     var name = cname + "=";
  9.     var ca = document.cookie.split(';');
  10.     for(var i = 0; i < ca.length; i++) {
  11.         var c = ca[i];
  12.         while (c.charAt(0) == ' ') {
  13.             c = c.substring(1);
  14.         }
  15.         if (c.indexOf(name) == 0) {
  16.             return c.substring(name.length, c.length);
  17.         }
  18.     }
  19.     return "";
  20. }

The AppInventor to PAHO MQTT API JavaScript

The MQTT broker is no longer automatically loaded after the webview page is loaded by the App Inventor application:

 
 
  1. //executes once after window is loaded --------------------------------------------->
  2. function windowloaded() {
  3. //client.connect(connectOptions);     // Connect to MQTT broker
  4.     AppInventorServer();                  // Start polling WebViewString
  5. }
  6. window.onload = windowloaded;  // Launch windowloaded()

The JavaScript must now receive a command to connect to the configured MQTT broker. The command received is called CFG, which loads the current connection settings stored in a file on the android device. The ensures the most current configuration is used for the connection.

There are 4 commands the JavaScript recognizes from the App Inventor App via the WebViewString.

  1. GET: Used to send an MQTT request message from the AppInventor App
  2. CFG: Update configurable parameters, disconnect and reconnect to MQTT broker
  3. CNX: Connect to MQTT broker
  4. KILLCNX: Disconnect from MQTT broker
 
 
  1. //GET: Send MQTT message -------------------------------------------------------------
  2. //------------------------------------------------------------------------------------
  3. if(request.substring(0, 4)=="GET:") {                // Validate request
  4. window.AppInventor.setWebViewString("");         // Reset String (process once)
  5. $("#request").val(request.substring(4, request.length)); //set request html textbox
  6. SendMqttRequest();                               // Send Mqtt Request
  7. }
  8. //CFG: Update configurable files in Android Device File ------------------------------
  9. //------------------------------------------------------------------------------------
  10. if(request.substring(0, 4) == "CFG:") { // Validate request
  11. window.AppInventor.setWebViewString(""); // Reset String (process once)
  12. if(typeof(client) !== 'undefined') { // Disconnect if connected
  13. client.disconnect();
  14. }
  15. var cfgpar = JSON.parse(request.substring(4, request.length));
  16. txtopic = cfgpar.mqtt_txtopic;
  17. rxtopic = cfgpar.mqtt_rxtopic;
  18. mqtt_url = cfgpar.mqtt_broker;
  19. mqtt_port = Number(cfgpar.mqtt_port);
  20. mqtt_clientId = cfgpar.mqtt_clientId;
  21. mqtt_keepalive = cfgpar.mqtt_keepalive;
  22. mqtt_lastwilltopic = cfgpar.mqtt_lastWillTopic;
  23. mqtt_lastwillmessage = cfgpar.mqtt_lastWillMessage;
  24. mqtt_lastwillqos = cfgpar.mqtt_lastWillQoS;
  25. connectOptions.userName = cfgpar.mqtt_un;
  26. connectOptions.password = cfgpar.mqtt_pw;
  27. connectOptions.keepAliveInterval= Number(mqtt_keepalive);
  28. // Create a Last-Will-and-Testament
  29. var lwt = new Paho.MQTT.Message(mqtt_lastwillmessage);
  30. lwt.destinationName = mqtt_lastwilltopic;
  31. lwt.qos = Number(mqtt_lastwillqos);
  32. lwt.retained = false;
  33. connectOptions.willMessage = lwt;
  34. client = new Paho.MQTT.Client(mqtt_url, mqtt_port, mqtt_clientId);
  35. client.onConnectionLost = onConnectionLost;
  36. client.onMessageArrived = onMessageArrived;
  37. client.connect(connectOptions); // Connect to MQTT broker
  38. }
  39. //CNX: Connect to MQTT broker --------------------------------------------------------
  40. //------------------------------------------------------------------------------------
  41. if(request.substring(0, 4)=="CNX:") { // Validate request
  42. window.AppInventor.setWebViewString(""); // Reset String (process once)
  43. if(typeof(client) == 'undefined') {
  44. // Create MQTT client instance -------------------------------------------------->
  45. client = new Paho.MQTT.Client(mqtt_url, mqtt_port, mqtt_clientId);
  46. // set callback handlers
  47. client.onConnectionLost = onConnectionLost;
  48. client.onMessageArrived = onMessageArrived;
  49. }
  50. client.connect(connectOptions); // Connect to MQTT broker
  51. }
  52. //KILLCNX: Diconnect from MQTT broker ------------------------------------------------
  53. //------------------------------------------------------------------------------------
  54. if(request.substring(0, 7)=="KILLCNX") { // Validate request
  55. window.AppInventor.setWebViewString(""); // Reset String (process once)
  56. conn_kill = "Y";
  57. client.disconnect();
  58. }
  59. hAppInvSvr = setTimeout(AppInventorServer, 100); // run AppInventorServer() in 100 ms

The callbacks executed upon MQTT broker connection, disconnection, and receipt of subscribed message can easily be understood through a review of the JavaScript file (mqtt_appinventor_paho.html).

App Inventor Updates

Changes to the App Inventor code simply expand the scope of the configurable parameters. Again, a review of the code should make these changes self-evident.

appinventor_update

Here is the code

The AppInventor and JavaScript code is available on GitHub here. Installation instructions are included in the readme.md file.

In Conclusion

There you have it. It is with great pleasure to present this update which supports username/password MQTT connections using the AppInventor. I hope you find this information useful.

Loading

Share This:
FacebooktwitterredditpinterestlinkedintumblrFacebooktwitterredditpinterestlinkedintumblr

MQTT for App Inventor – Adding Configurable Settings

21st January 2016 by InternetOfHomeThings in All Posts, Arduino, ESP8266 : An IoT System on Chip, Internet of Things | 5 Comments

App Inventor to MQTT Communication

In my last post, a basic MQTT client portal for App Inventor was introduced. As result, there has been some interest expressed for TCP socket support. Note that the implementation presented relies on WebSockets. Due to security vulnerabilities, pure TCP sockets are simply not possible with this design. If TCP sockets are needed for MQTT communication at your IoT device, a broker capable of both WebSockets and TCP sockets should be used – Websockets on the App Inventor side, and TCP sockets on the IoT device end.

Adding Configurable Setting

My next step was to refine the App Inventor project to support configurable MQTT setting.

NOTE: If you just want to review and use the updated App Inventor project without the walk-through that follows, simply use the files provided in this Github repository.

The first challenge while adding this capability was the inability to grant file system access to the JavaScript code. Native Android Apps can overcome this with a change to the App’s manifest file. But App Inventor does not expose this application setting.

So the App Inventor file component had to be used to read and write to the file system. The configuration file settings are then passed on to the JavaScript to update the MQTT connection parameters.

mqtt_cfg

App File Location

This App now uses two files stored on the Android device. These files have been moved to a common location which should be the same on most Android devices:

File Description
/sdcard/mqtt/mqtt_appinventor.html The JavaScript MQTT App Inventor Server
/sdcard/mqtt/cfg.txt MQTT Configuration

Configuration file

For this example, 3 parameters are configurable:

  1. Broker: This can be any on-line broker, on your own network or either publicly or privately provided.
  2. Request Topic: MQTT topic used to send a client request
  3. Reply Topic: MQTT topic used to reply results to the client.

Since this example uses the publicly available broker ‘test.mosquitto.org’, username/password authentication is not provided. You could easily be add this to the example provided here, however, if you are using a broker requiring authentication.

The file is formatted as a JSON string. This makes it very easy for the JavaScript to parse the values. The first time the App is run, these configurable parameters are set to the values shown in this JSON string:

{
“mqtt_broker”:”ws://test.mosquitto.org:8080/mqtt”,
“mqtt_txtopic”:”MyMqttSvrRqst”,
“mqtt_rxtopic”:”MyMqttSvrRply”
}

App Inventor File Read/Write

When the screen used to edit the configurable parameters is open, it first reads the cfg.txt file, extracts the parameter setting, and uses those values to populate the screen fields.

But when using App Inventor, there are no (at least none that I am aware) libraries available for parsing JSON. So I created the procedure “GetParam”, which extracts the cfg.txt values with 3 calls, one for each parameter.

readfile

Two values are determined initially in order to extract the value. First is the position in the string that the value starts. The other value is the string length. These values are determined by using App Inventor’s string search (starts at) feature to find something unique (StartPiece and LenPiece) around the value wanted, and an offset from the parameter value’s exact position in the string.

getparam

Once extracted, the “Edit Configuration” screen is populated with the values from this file.

cfg_gui

Saving the values is just the opposite. The values from the screen are formatted into a JSON string and then saved back to the same file, cfg.txt.

save

When this screen is exited, the new configuration values are sent to the JavaScript code. The JavaScript uses these new values and closes, then opens the MQTT connection with the new values. Here is how that has been implemented…

Updated Main Screen

As I have gained knowledge coding with the App Inventor, it became obvious that meaningful names were needed for project components. Much to my surprise, this does not extend to the main screen. It is called “Screen1” by default, and cannot be changed.

The screen GUI was changed somewhat for this update. Two new buttons were added. One to open the configuration screen and one to exit the application. The components were also move to the bottom of the screen (don’t want to cover the mosquito, do we?).

main

Every time the main screen is open, it’s 100 ms timer restarts. That is where the command is sent to the JavaScript to update the configuration settings.

Once.

It is sent 1/2 second (500 ms) after the screen is open. An important note: I found that the command did not execute if initiated concurrently with the opening of the screen. So the 500 ms delay (5 timer iterations) was introduced using a timer, which works every time.

timer

At 500 ms, the cfg.txt file is read. “CFG:” is inserted at the beginning of the string before being sent to JavaScript via the WebViewString.

sendcfg

A handler is also included for the first time this application is run. In that case, the cfg.txt file does not exists. As result, an exception is thrown when attempting to “ReadFrom” the file. An error handler was added for this condition.

error2

Error 2101 occurs when the file does not exist. In this case, a file is created with default values for the configurable parameters.

JavaScript for Configurable MQTT

The JavaScript also required revisions to support MQTT configurable parameters. The original version only handled IoT requests. These were identified with ‘GET:’ as the first 4 characters. Similarly, the added configuration command is now recognized as WebViewStrings beginning with “CFG:”

 
 
  1. // 10 Hz WebViewString polling timer function -------------------------------------->
  2. function AppInventorServer() {
  3. var request = window.AppInventor.getWebViewString(); // Get WebViewString
  4.     if(request.substring(0, 4)=="GET:") {                // Validate request
  5.      window.AppInventor.setWebViewString("");         // Reset String (process once)
  6.         $("#request").val(request.substring(4, request.length)); //set request html textbox
  7.         SendMqttRequest();                               // Send Mqtt Request
  8.     }
  9.     if(request.substring(0, 4)=="CFG:") {                // Validate request
  10.      window.AppInventor.setWebViewString("");         // Reset String (process once)
  11.         var cfgpar = JSON.parse(request.substring(4, request.length));
  12.         txtopic = cfgpar.mqtt_txtopic;
  13.         rxtopic = cfgpar.mqtt_rxtopic;
  14.         mqtturl = cfgpar.mqtt_broker;
  15.         client.disconnect();
  16.     }
  17.     setTimeout(AppInventorServer, 100);   // run AppInventorServer() in 100 ms
  18. }

As you can see, parsing and extracting values from a JSON string is very simple and straight-forward with JavaScript. Once the new values are set, the MQTT broker connection is updated with a call to ‘client.disconnect()’.

 
 
  1. // Callback executed upon MQTT broker disconnection -------------------------------->
  2. client.ondisconnect = function(rc){
  3. client.connect(mqtturl);
  4. };
  5. // Callback executed upon MQTT broker connection ----------------------------------->
  6. client.onconnect = function(rc){
  7. client.subscribe(rxtopic, 0);
  8. };

Improving the Arduino Digital Commands

The original project used hard-coded values for the Digital Get and Set commands. Ok for an example, but not so good if you need access to a channel outside the card-coded value. That was rectified with this update, which now supports digital channel selection.

dig_new2

Select “Set Arduino Digital Channel” and a new screen is open. This screen allows you to select the channel to set.

digital_sel

This screen is open with a “start value” received from the main screen that called it.

select_digit

The value returned when OK is clicked depends on the startValue. If it is a “Set” command, an Arduino SetDigital request string is returned. The selected channel and logic state from the screen’s selections are also returned with the request.

Notice that a call to close the screen is made before the main screen (Screen1) is re-opened. That is necessary when returning a startValue. If the screen is not closed, it remains open for the duration of the App execution.

Another call to this window will open another instance of the window. This is effectively a memory leak which will crash you Android device eventually. Takeaway: Close the screen before opening a new screen when passing a start value to avoid memory leaks.

As you can see, a similar value is returned for “Get” requests. But when “Get” requests are received, the channel state is not selected, it is returned from the request. When this screen is open with a “Get” request, the channel state selection GUI is suppressed.

get_digit

Sending the Arduino Digital Request

When the Digital channel selection screen is closed, the main screen is reopened. But how does it know to send the Arduino command to MQTT? After all, the screen is opened fresh, just like when the app is started.

Not quite. Remember the ‘startValue’ returned from the digital selection? That is checked to determine whether an arduino request is needed. It is done in the 100ms timer callback.

timer

At 500 ms after the screen is open, the MQTT configuration is sent to the JavaScript. But then, at 1000 ms (10 100 ms intervals), the startval string is check for a non-null value. If a string is present, it is sent to the JavaScript via the WebViewString, by calling the SendRequest procedure.

send_request

Exiting the Application

Just one more thing and we are done with this update. I thought it would be nice to add an ‘Exit’ button to close the app. What I found was that it required several clicks before the app would actually close. What was happening was that every time a different screen returned to the main screen, the previous main screen remained open and a new main screen instance was created.

Not exactly what I had in mind. Another memory leak!

This was corrected by adding a ‘close screen’ before each time one of the two added screens was open. This guarantees only one screen is open at a time while the app is running.

cfg_mqttt

With this correction, the Exit button works as intended, with only one click required.

In Closing

This example should provide a framework for anyone needing to use MQTT with an App Inventor project. The main caveat is that a broker that supports WebSockets is required. I leave it up to you as an exercise to add username and password credentials to the mqtt connection settings. The basic structure is here. And it should also be a simple task to add “GetAnalog” to the Arduino request options. Go ahead, you can do it! If you need it.

Again, here are the project files.

I hope you find this information useful…

Loading

Share This:
FacebooktwitterredditpinterestlinkedintumblrFacebooktwitterredditpinterestlinkedintumblr
Press Ctrl+C to copy the following code.
"