Skip to main content

Recursively watch a directory for modifications and publish file content to an MQTT broker

Project description

This simple Python program portably watches a directory recursively and publishes the content of newly created and modified files as payload to an MQTT broker. Files which are deleted are published with a NULL payload.

The path to the directory to watch recursively (default .), as well as a list of files to ignore (*.swp, *.o, *.pyc), the broker host (localhost) and port number (1883) must be specified in via environment variables , together with the topic prefix to which to publish to (watch/).

Installation

git clone https://github.com/jpmens/mqtt-watchdir.git
cd mqtt-watchdir
python3 -mvenv watchdir
source watchdir/bin/activate
pip install -e .

Configuration

Set the following optional environment variables before invoking mqtt-watchdir.py:

  • MQTTHOST (default "localhost") is the name/address of the MQTT broker.

  • MQTTPORT (default 1883) is the TCP port number of the broker.

  • MQTTUSERNAME is the username to connect to the broker.

  • MQTTPASSWORD is the password to connect to the broker.

  • MQTTWATCHDIR (default: ".") is the path to the directory to watch.

  • MQTTQOS (default: 0) is the MQTT Quality of Service (QoS) to use on publish. Allowed values are 0, 1, or 2.

  • MQTTRETAIN (default: 0) specifies whether the “retain” flag should be set on publish. Set to 1 if you want messages to be retained.

  • MQTTPREFIX (default: "watch") is the prefix to be prepended (with a slash) to the MQTT topic name. The topic name is formed from this prefix plus the path name of the file that is being modified (i.e. watched). You can set this to an empty string ("") to avoid prefixing the topic name.

  • MQTTFILTER (default None) allows modifying payload (see below). Takes path to a Python file (e.g. "/path/to/example-filter.py".

  • MQTTFIXEDTOPIC (default None) sets a MQTT topic to which all messages are published. If set, the MQTTPREFIX setting is overruled and ignored.

  • Set WATCHDEBUG (default: 0) to 1 to show debugging information.

Testing

Launch mosquitto_sub:

mosquitto_sub -v -t 'watch/#'

Launch this program and, in another terminal, try something like this:

echo Hello World > message
echo JP > myname
rm myname

whereupon, on the first window, you should see:

watch/message Hello World
watch/myname JP
watch/myname (null)

Filters

Without a filter (the default), mqtt-watchdir reads the content of a newly created or modified file and uses that as the MQTT payload. By creating and enabling a so-called filter, mqtt-watchdir can pass the payload into said filter (a Python function) to have a payload translated.

Consider the included example-filter.py:

def mfilter(filename, topic, payload):
    '''Return a tuple [pub, newpayload] indicating whether this event
       should be published (True or False) and a new payload string
       or None'''

    print "Filter for topic %s" % topic

    if filename.endswith('.jpg'):
        return False, None

    if payload is not None:
        return True, payload.replace("\n", "-").replace(" ", "+")
    return True, None

The mfilter function is passed the fully qualified path to the file, the (possibly prefixed) MQTT topic name and the payload. In this simple example, spaces and newlines in the payload are replaced by dashes and plusses.

The function must return a tuple with two elements:

  1. The first specifies whether the payload will be published (True) or not (False)

  2. The second is a string with a possibly modified payload or None. If the returned payload is None, the original payload is not modified.

Possible uses of filters include

  • Limiting payload lengths

  • Conversion to JSON

  • Ignore certain file types (e.g. binary data)

  • Process content of files, say, YAML or JSON, and extract elements returning as string

Requirements

  • watchdog, a Python library to monitor file-system events.

  • Paho-MQTT’s Python module

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mqtt-watchdir-2.4.tar.gz (8.0 kB view details)

Uploaded Source

File details

Details for the file mqtt-watchdir-2.4.tar.gz.

File metadata

  • Download URL: mqtt-watchdir-2.4.tar.gz
  • Upload date:
  • Size: 8.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: Python-urllib/3.8

File hashes

Hashes for mqtt-watchdir-2.4.tar.gz
Algorithm Hash digest
SHA256 4c081b8c4b72a16f2377f784f6d579ba1e6a389cc749e7e9e4d2bacb7d02143e
MD5 1faf87751547c38b1beecd36c3b8d4ab
BLAKE2b-256 9be60db90ebc537148221eb6cf63552a084e74684edff04c9965320c7ba7c576

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page