deephaven_plugin_node_editor

A Deephaven element plugin that edits configuration data as an interactive node graph.

The node_editor component renders a configuration as a graph of nodes. Objects, arrays, and scalar values each get a node, and the graph supports full structure editing: renaming keys, adding and deleting entries, changing value types, and converting between objects, arrays, and scalars. Every edit produces the updated configuration as a plain JSON-serializable dict.

Input can be a dict, a ConfigTree, or a HOCON string. HOCON is parsed on the Python side with pyhocon, so substitutions and includes are resolved before the graph is built.

Plugin Structure

The src directory contains the Python and JavaScript code for the plugin.
Within the src directory, the deephaven_plugin_node_editor directory contains the Python code, and the js directory contains the JavaScript code.

The Python files have the following structure:
node_editor.py defines the node_editor component and normalizes dicts, ConfigTree instances, and HOCON strings into plain JSON-serializable data.
register.py registers the plugin with Deephaven. This file will not need to be modified for most plugins at the initial stages, but will need to be if the package is renamed or JavaScript files are moved.

The JavaScript files have the following structure:
DeephavenPluginNodeEditorPlugin.ts registers the plugin with Deephaven and maps the deephaven_plugin_node_editor.node_editor element name to the React view.
DeephavenPluginNodeEditorView.tsx renders the React Flow canvas and owns the controlled and uncontrolled editing behavior.
editorTree.ts holds the immutable tree model and the path-based edit operations.
jsonToGraph.ts derives the nodes and edges from the configuration and lays them out with dagre.
EditorNodes.tsx defines the object, array, and value node components.
NodeEditorContext.ts passes the edit callbacks down to the node components.

Using plugin_builder.py

The plugin_builder.py script is the recommended way to build the plugin. See Building the Plugin for more information if you want to build the plugin manually instead.

To use plugin_builder.py, first set up your Python environment and install the required packages.
To build the plugin, you will need npm and python installed, as well as the build package for Python. nvm is also strongly recommended, and an .nvmrc file is included in the project. The script uses watchdog and deephaven-server for --watch mode and --server mode, respectively.

cd deephaven-plugin-node-editor
python -m venv .venv
source .venv/bin/activate
cd src/js
nvm install
npm install
cd ../..
pip install --upgrade -r requirements.txt
pip install deephaven-server watchdog

First, run an initial install of the plugin: This builds and installs the full plugin, including the JavaScript code.

python plugin_builder.py --install --js

After this, more advanced options can be used. For example, if only iterating on the plugins with no version bumps, use the --reinstall flag for faster builds. This adds --force-reinstall --no-deps to the pip install command.

python plugin_builder.py --reinstall --js

If only the Python code has changed, the --js flag can be omitted.

python plugin_builder.py --reinstall

Additional especially useful flags are --watch and --server. --watch will watch the Python and JavaScript files for changes and rebuild the plugin when they are modified. --server will start the Deephaven server with the plugin installed. Taken in combination with --reinstall and --js, this command will rebuild and restart the server when changes are made to the plugin.

python plugin_builder.py --reinstall --js --watch --server

If interested in passing args to the server, the --server-arg flag can be used as well Check deephaven server --help for more information on the available arguments.

python plugin_builder.py --reinstall --js --watch --server --server-arg --port=9999

See Using the Plugin for more information on how to use the plugin.

Manually Building the Plugin

To build the plugin, you will need npm and python installed, as well as the build package for Python. nvm is also strongly recommended, and an .nvmrc file is included in the project. The python venv can be created and the recommended packages installed with the following commands:

cd deephaven-plugin-node-editor
python -m venv .venv
source .venv/bin/activate
pip install --upgrade -r requirements.txt

Build the JavaScript plugin from the src/js directory:

cd src/js
nvm install
npm install
npm run build

Then, build the Python plugin from the top-level directory:

cd ../..
python -m build --wheel

The built wheel file will be located in the dist directory.

If you modify the JavaScript code, remove the build and dist directories before rebuilding the wheel:

rm -rf build dist

Installing the Plugin

From PyPI

The released plugin is published to PyPI as deephaven-plugin-node-editor, so the easiest way to install it is with pip:

pip install deephaven-plugin-node-editor

To install a specific version, pin it:

pip install deephaven-plugin-node-editor==0.1.2

The plugin needs to be installed into the same Python environment as the Deephaven server. If you are running the server from a venv, the plugin and server can be installed with the following commands:

pip install deephaven-server deephaven-plugin-node-editor
deephaven server

If you are running Deephaven in Docker, add the package to the image or set it in the START_OPTS/requirements used to build your image.

From a local wheel

A locally built plugin can be installed with pip install <wheel file>. The wheel file is stored in the dist directory after building the plugin. Exactly how this is done will depend on how you are running Deephaven. If using the venv created above, the plugin and server can be created with the following commands:

pip install deephaven-server
pip install dist/deephaven_plugin_node_editor-0.1.2-py3-none-any.whl
deephaven server

See the plug-in documentation for more information.

Using the Plugin

Once the Deephaven server is running, the plugin should be available to use.

The editor is uncontrolled when given a default_value. The client owns the configuration after the initial render, and on_change is called with the updated dict after every edit.

from deephaven_plugin_node_editor import node_editor

editor = node_editor(
    default_value={"name": "prod", "port": 8080, "db": {"host": "localhost", "ssl": True}},
    on_change=print,
)

value and default_value also accept a HOCON string, which is parsed on the server. Invalid HOCON raises, and substitutions and includes are resolved before the graph is built, so mirror.port below is 9000.

editor = node_editor(default_value="app { port = 9000 }, mirror { port = ${app.port} }")

The editor is controlled when given a value. The server owns the configuration, and it is up to the caller to update value in response to on_change. Passing both value and default_value raises a ValueError.

from deephaven import ui
from deephaven_plugin_node_editor import node_editor


@ui.component
def config_editor():
    config, set_config = ui.use_state({"name": "prod", "port": 8080})
    return node_editor(value=config, on_change=set_config)


editor = config_editor()

Props are automatically converted from snake_case to camelCase, so default_value becomes defaultValue and on_change becomes onChange on the JavaScript side.

Examples

examples/trade_filter.py builds a dashboard where the config graph drives the filters on a live ticking table. Run it in a Deephaven console and open the trade_monitor dashboard.

examples/algo_matrix.py edits an algo matrix: phases holding nodes and the transitions between them. The graph feeds a nodes table, a transitions table, and a live feed of the transitions that have fired, so changing a threshold or rewiring a node immediately changes the tables. Run it in a Deephaven console and open the algo_matrix dashboard.

Testing the Plugin

The end to end tests drive a real Deephaven server with the plugin installed and assert on the configuration that reaches the server after each edit.

tests/node_editor.spec.ts covers the editing behavior: renaming keys, adding and deleting entries, changing value types, converting between objects, arrays and scalars, and the difference between a controlled and an uncontrolled editor.
tests/editor_examples.spec.ts runs the shipped examples and checks that editing the graph re-filters and rebuilds the tables they derive.
tests/app.d is loaded by the server in application mode, so every fixture is in the Panels menu when the page loads.

Install the plugin first, then run the suite. Playwright starts the server itself, so the virtual environment holding deephaven-server and the plugin must be active:

python plugin_builder.py --install --js
npm install
npx playwright install --with-deps chromium
npx playwright test

Set DH_PORT to run against a different port. To regenerate the images in this README:

UPDATE_SCREENSHOTS=1 npx playwright test tests/screenshots.spec.ts

Debugging the Plugin

It's recommended to run through all the steps in Using plugin_builder.py and Using the Plugin to ensure the plugin is working correctly.
Then, make changes to the plugin and rebuild it to see the changes in action. Checkout the Deephaven plugins repo, which is where this template was generated from, for more examples and information.
The plugins folder contains current plugins that are developed and maintained by Deephaven.
Below are some common issues and how to resolve them as you develop your plugin.
If there is an issue with the process while following the Installation and Usage steps on the originally generated plugin, please open an issue.

The Panel is Not Appearing

Checking if the Plugin is Registered

If the panel is not appearing or an error is thrown that the import is not found, the plugin may not be registered correctly. To verify the plugin is registered, check either the console logs or the versions in the settings panel.

• In the console logs, there should be a messaging saying Plugins loaded: with a map that includes this plugin.
  

• To get to the settings panel, click on the gear icon in the top right corner of the Deephaven window. Towards the bottom this plugin should be listed.
  

• If the plugin is not listed, attempt to rebuild and reinstall the plugin and check for errors during that process.

Checking if the Python Package is Installed

• Running pip list in the .venv environment should show the Python package installed, but this is not a guarantee that the plugin is registered properly.
• The version can also be checked directly from the Python console with:

from importlib.metadata import version
print(version("deephaven_plugin_node_editor"))

The Panel is Appearing but with Errors or Not Functioning Correctly

Check both the Python and JavaScript logs for errors as either side could be causing the issue.

Distributing the Plugin

To distribute the plugin, you can upload the wheel file to a package repository, such as PyPI. The version of the plugin can be updated in the setup.cfg file.

There is a separate instance of PyPI for testing purposes. Start by creating an account at TestPyPI. Then, get an API token from account management, setting the “Scope” to “Entire account”.

To upload to the test instance, use the following commands:

python -m pip install --upgrade twine
python -m twine upload --repository testpypi dist/*

Now, you can install the plugin from the test instance. The extra index is needed to find dependencies:

pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ deephaven_plugin_node_editor

For a production release, create an account at PyPI. Then, get an API token from account management, setting the “Scope” to “Entire account”.

To upload to the production instance, use the following commands. Note that --repository is the production instance by default, so it can be omitted:

python -m pip install --upgrade twine
python -m twine upload dist/*

Now, you can install the plugin from the production instance:

pip install deephaven_plugin_node_editor

See the Python packaging documentation for more information.