node-blue 0.0.0

Node-RED without a mouse / Node-RED as a library / Write Node-RED flows in YAML / Efficient test harnesses for Node-RED / Headless Node-RED / Improved development cycle times / Now it all makes sense

Node-BLUE

Node-BLUE is a friendly wrapper around Node-RED.

Node-RED without a mouse / Node-RED as a library / Write Node-RED flows in YAML / Write Node-RED user-defined functions in Python / Efficient test harnesses for Node-RED / True headless Node-RED / Improved development iteration times / Node-RED without needing to take the red pill / Node-BLUE is Node-RED on rails / Now it all makes sense

About

The idea is to embed Node-RED into Python programs, in order to leverage it for a number of use cases, like system automation, software testing, parallel execution, etc.

The project has similar intentions like node-red-embedded-start, offering to interact programmatically with the Node-RED API, with a few bells and whistles.

The other idea is to extend the Node-RED ecosystem by leveraging other programming languages and their ecosystems natively, beyond what JavaScript/NPM can do. We made a start with Python, by using the excellent JSPyBridge package.

Synopsis

Command-line use

Install Node-BLUE and HTTPie, and their prerequisites.

pip install httpie node-blue

There is currently a little manual post-setup procedure.

node-blue setup
wget https://github.com/daq-tools/node-blue/raw/refs/heads/main/package.json
wget https://github.com/daq-tools/node-blue/raw/refs/heads/main/node.importmap
npm install

Start Node-BLUE with a Node-RED flow defining an HTTP/HTML endpoint/responder.

# Launch Node-BLUE/Node-RED.
node-blue launch --flow=examples/flows/http-html-templating.json
node-blue launch --flow=https://github.com/daq-tools/node-blue/raw/main/examples/flows/http-html-templating.json

# Run an example HTTP request.
http http://localhost:1880/hello-form name=Hotzenplotz

Launch a flow which defines a pipeline to converge data from MQTT to [CrateDB].

# Launch Node-BLUE/Node-RED.
node-blue launch --flow=examples/flows/mqtt-to-cratedb.yaml

Library use

import asyncio
from node_blue.core import NodeBlue

async def launch_blue():
    """
    Launch a Node-BLUE instance.
    """
    
    # Configure Node-BLUE instance with Node-RED flow.
    blue = NodeBlue(flow="https://github.com/daq-tools/node-blue/raw/main/examples/flows/http-html-templating.json")

    # Start Node-RED instance.
    blue.start().wait_started()

    # Wait until termination.
    await blue.forever()

if __name__ == "__main__":
    asyncio.run(launch_blue())

Examples

Introduction

Unless otherwise noted, all subsequent examples can be exercised using MQTT. You will start the Mosquitto MQTT broker, subscribe to a topic, and publish a JSON message, which will be processed by a Node-RED flow definition.

All flow definitions implement the same rule, and as such, can be triggered by using the same MQTT messages, outlined below. For demonstration purposes, the transformation rule is very simple: It applies a unit conversion formula to a numeric input parameter, effectively converting value units, from Fahrenheit to Celsius. In order to not mix things up, the outcome will be published to a different MQTT topic.

This effectively demonstrates message routing and republishing capabilities in both terms of content transformation and topic rewriting. The procedures can be applied to any message broker system, we just used MQTT here, because it is so convenient to operate.

docker run --name=mosquitto -it --rm --publish=1883:1883 eclipse-mosquitto:2.0 \
  mosquitto -c /mosquitto-no-auth.conf

Subscribe to the broker to observe the outcome.

mosquitto_sub -t 'testdrive/#' -v

Run an example MQTT publish.

echo '{"temperature": 42.42}' | mosquitto_pub -t 'testdrive/imperial' -l

JSON5 flow format

Start Node-BLUE with a Node-RED flow defining an MQTT topic rewrite and message transformation step. Note that this example uses the JSON5 format, which has a few advantages over regular JSON, like inline comments, and multi-line strings.

wget https://github.com/daq-tools/node-blue/raw/main/examples/flows/mqtt-unit-rewriting.json5
node-blue launch --flow=mqtt-unit-rewriting.json5

This flow snippet demonstrates JSON5 capabilities on behalf of a sensible example.

// The transformation node element uses a function written in JavaScript to convert from
// Fahrenheit to Celsius, and to rewrite the MQTT topic from `imperial` to `metric`.
// The outcome will be the amended MQTT message object, which can be passed to the `mqtt out`
// data sink node without further ado.
{
  "id": "transformation.55460a",
  "type": "function",
  "func": "\
    if (msg.topic.endsWith('imperial')) { \
      msg.payload.temperature = Number.parseFloat(((Number.parseFloat(msg.payload.temperature) - 32) * 5 / 9).toFixed(2)); \
      msg.topic = msg.topic.replace('imperial', 'metric'); \
      return msg; \
    } \
  ",
  "wires": [
    [
      "sink.3539af",
    ]
  ]
}

YAML flow format

You can also use the YAML format for writing flow recipes. The authors believe it offers the best conciseness and convenience, with even better multi-line code blocks, without needing any sorts of line-continuation characters. YAML is already used by the node-red-contrib-flow-manager and node-red-contrib-yaml-storage plugins.

node-blue launch --flow=https://github.com/daq-tools/node-blue/raw/main/examples/flows/mqtt-unit-rewriting.yaml

With this example flow snippet, you will immediately recognize how convenient it is, especially for user-defined functions.

# The transformation node element uses a function written in JavaScript to convert from
# Fahrenheit to Celsius, and to rewrite the MQTT topic from `imperial` to `metric`.
# The outcome will be the amended MQTT message object, which can be passed to the `mqtt out`
# data sink node without further ado.
- id: transformation.55460a
  type: function
  func: |-
    if (msg.topic.endsWith('imperial')) {
      msg.payload.temperature = Number.parseFloat(((Number.parseFloat(msg.payload.temperature) - 32) * 5 / 9).toFixed(2));
      msg.topic = msg.topic.replace('imperial', 'metric');
      return msg;
    }
  wires: [
    [
      "sink.3539af",
    ]
  ]

Python user-defined functions

This time, let's use the Python language, to define a user-defined function within the flow step.

wget https://github.com/daq-tools/node-blue/raw/main/examples/flows/mqtt-routing-python.yaml
node-blue launch --flow=examples/flows/mqtt-routing-python.yaml

# The transformation node element uses a function written in Python to convert from
# Fahrenheit to Celsius, and to rewrite the MQTT topic from `imperial` to `metric`.
# The outcome will be the amended MQTT message object, which can be passed to the `mqtt out`
# data sink node without further ado.
- id: transformation.66571b
  type: python-function
  func: |-
    if msg.topic.endswith("imperial"):
      msg.payload.temperature = round((float(msg.payload.temperature) - 32) * 5 / 9, 2)
      msg.topic = msg.topic.replace("imperial", "metric")
      send(msg)
    done()
  wires: [
    [
      "sink.3539af",
    ]
  ]

References

See also the IBM Data management article about Implementing ETL flows with Node-RED by Ondrej Lehota.

Etymology

To use the name »Node-BLUE« for this project was obvious. However, we discovered that there has been another project called Node-BLUE, referenced below. On the other hand, because it has been archived two years ago already, we think there will not be much harm to reuse that name now.

• https://www.npmjs.com/package/node-blue
• https://github.com/node-blue/node-blue

Acknowledgements

This project bundles a few significant pieces of software and technologies, which a few bright minds invented the other day, and countless authors contributed to. We are only listing inventors of the major application-level components, remember there may be several layers of operating systems beneath, and silicon either.

• Andy Stanford-Clark, Arlen Nipper, and co-authors for MQTT.
• Brendan Eich and contributors for JavaScript.
• @extremeheat and contributors for JSPyBridge.
• Guido van Rossum and contributors for Python.
• Lars Bak and contributors for V8.
• Nick O'Leary and contributors for Node-RED.
• Roger Light and contributors for Mosquitto.
• Ryan Dahl and contributors for Node.js and Deno.

Thank you!