Your Own MQTT Broker

mqtt

Like many folks, I too started out using the public MQTT broker at test.mosquitto.org. It’s a great way to get started – simple, easy to get working, and FREE! But it does not take long to realize it is unsecured. Anyone can listen in on your topics and there are no logon credentials required or offered as an option.

So I got to searching for a better broker…

One with security. And all the options available with the MQTT standard. Things like:

  • Security Authentication (passwords,certificates)
  • Simultaneous websockets and mqtt (tcp) listeners
  • Persistent Messages

But I did not want to pay for the service. The obvious solution was to host your own broker, either on a host server, or on your local network with broadband access via a router.

Since my host server does not permit continuously running scripts or programs, was limited to a local network solution. But with a broadband connection, it would be on-line and accessible anywhere.

After some research, the most promising options included:

  • PC Broker with Windows OS
  • PC Broker with Linux OS
  • Flash Driver Linux Distro Broker
  • Raspberry Pi2 Broker
  • Android Device Broker
  • Embedded micro-controller

I have read numerous comments about poor (slow) performance using a Raspberry Pi, and since I do not own one, that option was ruled out. For the same reason, I thought about hosting a broker on the trusty ESP8266 but decided against it, at least for now.

And while it would be great to use an old Android phone as an MTTQ broker, the path to get there was a bit murkier than using a Linux hosted server.  It can be done, but few have gone this path. That is, there is little in the way of guidance so this would require significant development.

Windows? Maybe with Windows Server OS running. But that’s not what I got. No.

So looking around at my inventory, I decided to use an unused Window 7 netbook. But, rather than overwriting the hard-drive, a USB flash drive installation was done.

Linux running an MQTT broker when booting to the flash drive.

Windows 7 with the flash drive removed.

While there were a few challenges along the way, it turned out to be a great solution. It has been running continuously now for over a week – flawlessly.

Here is how to set it up…

Linux Installation

Looking for a small, clean Linux distribution, I selected Puppy Linux.  The choice was easy to make, since it had already been setup and running. This post provides step-by-step instructions to configure your flash drive. Follow all the instructions as you will need the development environment to build the MQTT application.

Building the MQTT Broker Application

First thing needed is a copy of the Mosquitto 1.4.7 broker. You can get it here. Then, with the flash drive in a Windows PC, copy the unzipped contents of the folder org.eclipse.mosquitto-1.4.7 to the flash drive in a new folder in the path:

<flash drive>/MyPrograms/mosquitto

You can now install the flash drive in your target computer and reboot. It should start in Puppy Linux.

puppy-start

Before we can build the application, a couple of steps are needed.

  • Install Mosquitto Package
  • Install libwebsockets
  • Edit build configuration file

Let’s go…

Installing the Mosquitto Package

  1. From the Desktop, click on the blue “install” icon.
  2. Click on the “Install Applications” tab.
  3. Click on the Puppy Package Manager icon.
  4. Enter “mqtt” into the search box and hit the “Enter” key.
  5. Click on the mosquitto_0.15 Package.
  6. Click “Install” (Upper right of windows).

Installing the libwebsockets Library

  1. From the Desktop, click on the blue “install” icon.
  2. Click on the “Install Applications” tab.
  3. Click on the Puppy Package Manager icon.
  4. Enter “libwebsockets” into the search box and hit the “Enter” key.
  5. Click on the libwebsockets3_1.2.2.1 Package.
  6. Click on the libwebsockets-dev_1.2.2.1 Package.
  7. Click “Install” (Upper right of windows).

Edit build configuration file

  1. From the Desktop, click on the green “edit” icon.
  2. Click Open, then under “Places”, click “File System”.
  3. Under “Name”, click “mnt”. Then click open.
  4. Under “Name”, click “home”. Then click open.
  5. Under “Name”, click “MyPrograms”. Then click open.
  6. Under “Name”, click “mosquitto”. Then click open.
  7. Under “Name”, click “config.mk”. Then click open.
  8. Scroll down to “WITH_WEBSOCKETS:=no and change to “yes”
  9. Save the file and exit.

Building the broker application

We are now ready to build the application. This is really simple. First, open the console window by clicking on the “console” icon from the desktop. Now switch to the directory that contains the mosquitto source code by entering:

cd /mnt/home/myprograms/mosquitto/src

now build the application by entering:

make

Configuring the MQTT Broker Application

Edit mosquitto run-time configuration file

  1. From the Desktop, click on the green “edit” icon.
  2. Click Open, then under “Places”, click “File System”.
  3. Under “Name”, click “mnt”. Then click open.
  4. Under “Name”, click “home”. Then click open.
  5. Under “Name”, click “MyPrograms”. Then click open.
  6. Under “Name”, click “mosquitto”. Then click open.
  7. Under “Name”, click “mosquitto.conf”. Then click open.
  8. From the edit menu, select”Save as” and save this file to the src folder. The full file path should now be /mnt/home/myprograms/mosquitto/src/mosquitto.conf.
  9. Scroll down or search for “#allow_anonymous true”. Change this to “allow_anonymous false”. Remember to delete the # so this is not commented out. This will force the broker to require usernames and passwords to connect.
  10. Scroll down or search for “#user mosquitto”. Change this to “user nobody”. Remember to delete the # so this is not commented out.Puppy linux does not have a user named “mosquitto” but it does have one named “nobody”. Since Puppy Linux is a single user distribution, it does not allow you to add users.
  11. Scroll down or search for “#password_file”. Change this to “password_file pw.txt”. Remember to delete the # so this is not commented out.
  12. Scroll down or search for the text “#protocol mqtt”. Just after this line, add the following 4 new lines:
    1. listener 11883
    2. protocol mqtt
    3. listener 18080
    4. protocol websockets
  13. Save and exit the file

What step 9 does is configure the broker with two listeners, one with standard mqtt (tcp) protocol and one with websockets.

While any port can be used, a one was added to the standard port numbers so our broker is not in conflict with the “well-known” mqtt ports. This could be important in the complicated event that your client is connected to two different brokers at the same time. In this case, the ports can only be open once. This eliminates potential conflicts.

Creating a password file

Using the file editor (edit icon from the desktop), save a blank file in the src folder:

/mnt/home/myprograms/mosquitto/src/pw.txt

Add a few username/password entries in this file in the format, for example:

username:password
user2:password2

Save the file. Also make a copy of this file for off-line storage.

Now run the password utility. This will change the plain text passwords in the file pw.txt to a hashed value. Run the password utility from the console:

cd /mnt/home/myprograms/mosquitto/src
./mosquitto_passwd -U pw.txt

If you open pw.txt, you will find the plain text passwords have been replaced with a hashed value.

Opening the Linux Firewall

network-firewall-icon

Are we ready to run the broker yet? Almost. But there is one more thing needed in order to access the broker from another device. We need to configure Linux to allow external connections.

Open the file /etc/hosts.allow

It should contain one line:

ALL:LOCAL

Change this to:

ALL:ALL

Save and exit the file.

If you want a more restrictive environment, it is suggested that you research configuration settings for the Linux hosts.allow file. For the purpose of this example, we are opening the MQTT Broker to anyone with proper username/password credentials.

Starting the Broker

Now to start the broker, just go to the src folder and enter the following:

cd /mnt/home/myprograms/mosquitto/src
./mosquitto -c mosquitto.conf

The startup console should display:

start_mosq

The warning occurs because ipv6 is not supported. But this is of no concern for the typical ipv4 addresses. While the application was initially build in the ../src directory, it can be moved and executed from any location of your choosing. just remember to also include the password file utility, the config file and the password file. These are the files needed to run the application:

mosquitto
mosquitto_passwd
mosquitto.conf
pw.txt

Testing the Broker

You probably have your own devices to connect to and test the broker. With the configuration of this broker, you will need to include a username and password when connecting. I like to use the Google Chrome MQTTlens extension and the Android MyMQTT App when making quick MQTT connection checks. I’ve provided additional details for using these MQTT tools in this post.

Port Forwarding and DDNS 

icon-port-forwarding

While you can access this MQTT  broker from any device on your LAN subnet, world-wide broadband access will require configuring your router to forward MQTT port requests to the device running the broker. This is called port forwarding. Please refer to this post for more information about configuring port forwarding and DDNS.

In Closing

This broker setup provides you with all the features of the MQTT specification. One of the best features is the ability to operate a mixed protocol system. This provides an connection to devices using either mqtt (tcp) and websocket protocol together. And you can enable any level of security needed, from simple passwords to security certificates. And since it is running locally, you are in full control.

Next up. I am planning to update my MQTT App inventor template application to support the basic security credential provided by this broker.

I hope you find this information useful…

 

Loading

Share This:
FacebooktwitterredditpinterestlinkedintumblrFacebooktwitterredditpinterestlinkedintumblr
Social tagging: >

10 Responses to Your Own MQTT Broker

  1. Vinod rawat says:

    Hi,
    I visited your repository on Github regarding broker for Android, but was unable to find any.
    Is this structure according to Android studio?
    And where is the broker library to be implemented in android studio?

    • facebook-profile-picture InternetOfHomeThings says:

      The broker is for a Linux distribution. The referenced repository was tested specifically with Puppy Linux installed on a USB Memory stick. Since Android is built on the Linux OS, it might be possible to install an MQTT broker, I have not explored this possibility and have no information to share in this regard,

  2. Tanmay says:

    Is it possible to make a microcontroller or even my ESP8266 as the MQTT broker? I'd love to implement it!

    • facebook-profile-picture InternetOfHomeThings says:

      Raspberry Pi can be programmed to be an MQTT Broker. But the reviews I have read suggest significant performance issues with it. I looked into the possibility of using the ESP8266 as an MQTT broker in the past but concluded that there was no easy method of porting existing broker code such as Mosquito, which is geared towards the Linux OS, to the ESP environment.

      I am currently implementing an MQTT broker on my VPS (Virtual Private Server). I believe this is a superior solution, making the broker available anywhere without the need to having a pc running 24-7./

  3. Carl Gifford says:

    Dave thanks for this instructable very well done! I have a couple of questions.
    I downloaded puppy ver 6.3.2 and the development sfs, all works well, Mosquitto and Libwebsockets, however the Mosquitto Package manager doesn't find them.
    1)do you know if, there is a specific folder that these zips should be downloaded to in order for the PPM to see them?
    2) if I clone them from git would I clone to /usr or /root/my-applications
    plan B will to download your identical setup and retry.
    when I compile Mosquitto after some compiler actions I get a failure notice it can't find Libwebsockets.h. I've tried some of the other suggestions on the net and have installed libwebsockets in the /lib folder. but not getting recognized by compiler.
    did you go through a similar process? , if so, I'll just hunker down and try a few more hunches. Just didn't want to force feed with a lenghty makefile if default paths exist, new to linux, not new to command line development I really enjoy the hunt but I'm off track from my original distributed esp sequencer code which uses yout webserver MQTT framework (again thanks, learned a lot: )
    Any suggestions on the questions would be appreciated,

  4. Carl Gifford says:

    I should follow instructions verbatim.
    new to linux so Each one of these packages are similarish. I downloaded Puppy-tahr as indicated, the package manager responded as indicated.
    please disregard above comment, other than the Thankyou!

  5. Carl Gifford says:

    I have managed to get the broker and clients made and installed.
    testing broker inside of puppy with mosquitto_pub and mosquitto_sub work in multiple shells so I know clients and broker function, the ESP test code (with a slight mod) and MQTTlens publish to the broker. Broker side mosquitto_sub does pick up publishes from multiple external sources.

    A broker side publish does not get received by any callback functions I have tried, or MQTTlens. but does get picked up on the broker side Mosquitto_sub client.I noticed in your next post that you mentioned there were some issues.
    Did you see similar problems?
    is this a mosquitto issue or a puppy/ mosquitto issue?

  6. Mark says:

    Excellent Excellent post… This post will go a long way in helping me understand how to set up mqtt communication for my project. thank you

  7. Prashant says:

    How to do the same on Windows 10?
    Thanks!

Leave a Reply