dsconfig 1.9.0

Library and utilities for Tango device configuration.

dsconfig

This is a set of command line tools for managing configuration of Tango (https://www.tango-controls.org/) device servers.

The tools are centered around a simple JSON configuration format that represents Tango database information. It's possible to dump an existing database into the dsconfig format, as well as configure an existing database from it.

Installation

dsconfig is available on PyPI and conda-forge. Install it with your preferred tool:

• pip install dsconfig in a virtualenv
• or conda create -n dsconfig -c conda-forge dsconfig

If you are a uv or pixi fan, you can install it globally with either:

• uv tool install dsconfig
• or pixi global install dsconfig

Usage

tango2json - dumping Tango database information as JSON

Dumps database (as usual controlled by TANGO_HOST) information into JSON:

$ tango2json
{
    "servers": {
        ...
        "TangoTest": {
            "test": {
                "TangoTest": {
                    "sys/tg_test/1": {}
            }
        },
        ...
    }
}

That dumps all the content of the current Tango database into a JSON file. tango2json writes its output to stdout so you should redirect to a file. Or, you could pipe it to e.g. less.

$ tango2json > db.json

This is useful e.g. to inspect the current configuration, or as a backup that can be restored later (see below).

Instead of including the entire database, you can add filters that select some subset:

$ tango2json class:TangoTest

For more help on the available options, try the -h (--help) argument. See below for more details on the JSON format.

json2tango - modifying the Tango database (in a safe way)

By making a database dump of the relevant stuff, modifying the JSON output, and then feeding it back, you can change the database.

Let's say we want to add a device property:

$ tango2json class:TangoTest > tangotest.json

Then we open the JSON file in an editor. It might look something like this:

    {
        "servers": {
            "TangoTest": {
                "test": {
                    "TangoTest": {
                        "sys/tg_test/1": {}
                }
            }
        }
    }

To add a property, we change it something like this (make sure to adhere to the JSON format, it is very picky about extra commas and such):

    {
        "servers": {
            "TangoTest": {
                "test": {
                    "TangoTest": {
                        "sys/tg_test/1": {
                            "properties": {
                                "MyNewProp": ["foo", "bar"]
                            }
                        }
                    }
                }
            }
        }
    }

Then, in order to update the Tango DB:

    $ json2tango tangotest.json

    = Device: sys/tg_test/1
      Properties:
        + MyNewProp
          foo
          bar

    Summary:
    Add/change 1 device properties in 1 devices.

    *** Nothing was written to the Tango DB (use -w) ***

This output tells us that the tool has recognized that we're adding the property to an existing device, and asking us to verify that this is indeed what we intended. It's not actually doing anything yet.

Since this looks fine, let's apply it for real, with the --write flag:

    $ json2tango tangotest.json --write

    = Device: sys/tg_test/1
      Properties:
        + MyNewProp
          foo
          bar

    Summary:
    Add/change 1 device properties in 1 devices.

    *** Data was written to the Tango DB ***
    The previous DB data was saved to /tmp/dsconfig-ajtow9w0.json

The output is the same apart from the last two lines, that tell us that the operations were applied, and that a backup file was created. We're done!

The automatic backup can always be used to get back to the database state before applying the configuration, by applying it with json2tango /tmp/dsconfig-ajtow9w0.json. You can try it and verify that it would remove the new property again.

See below for more details on how to use json2tango.

Some other use cases

• Automatic configuration for many Tango devices, by building JSON with e.g. a python script and sending it to json2tango
• Creating "snapshots" of a working configuration so that you can easily go back
• An easy way for developers to store and exchange Tango device setups.
• Compare a dump with the current database to see what has changed.

Caveats

There are a few things to be aware of before using this tool.

• TANGO is case insensitive for names, for example of devices and properties. But there are some cases where this causes confusing results. For example, TANGO keeps whatever casing was used when written, which means that the same name may exist in different places with different cases. Dsconfig tries to handle this gracefully, but it is complex (for example, all relevant string comparisons need to be done in a case insensitive way) and there are bound to be corner cases where the behavior is unexpected. Please report such cases if you run into them.

• Dsconfig only deals with Tango database stuff. Writing attributes and running commands, restarting servers, etc, is out of scope.

• Dsconfig is a fairly "low level" tool, and the JSON format is not extremely friendly to human editing. But it is easy to create JSON programmatically, so other tools can build on top of it. DSconfig takes care of all the tedious Tango database communication stuff.

JSON format

This is an example of the format, with comments (comments are not actually supported by JSON so don't copy-paste this!):

{
    // these lines are meta information and are ignored so far
    "_version": 1,
    "_source": "ConfigInjectorDiag.xls",
    "_title": "MAX-IV Tango JSON intermediate format",
    "_date": "2014-11-03 17:45:04.258926",

    // here comes the actual Tango data
    // First, server instances and devices...
    "servers": {
        "some-server": {
            "instance": {
                "SomeDeviceClass": {
                    "some/device/1": {
                        "properties": {
                            "someImportantProperty": [
                                "foo",
                                "bar"
                            ],
                            "otherProperty": ["7"]
                        },
                        "attribute_properties": {
                            "anAttribute": {
                                "min_value": ["-5"],
                                "unit": ["mV"]
                            }
                        }
                    }
                }
            }
        }
    },
    // Here you can list your class properties
    "classes": {
        "SomeDeviceClass": {
            "properties": {
                "aClassProperty": ["67.4"]
            }
        }
    }
}

Properties

All properties must be given as lists of strings. This is how the Tango DB represents them so it gets a lot easier to compare things if we do it too.

Leaving out the "properties" field in a device will mean that json2tango just ignores any existing properties when applying the configuration. Otherwise, properties not specified in the new configuration will get cleaned away. But an empty properties object ("properties": {}) means that any existing properties will get cleaned up. Same goes for "attribute_properties".

If you don't want to remove any existing properties, try the --update flag (see below).

Protected properties

Some properties are considered "protected", meaning they will not be removed by json2tango if they are absent in the new config. This covers properties that are used by Tango for configuring internal features, such as polling, events and logging. The assumption is that these may be configured in other ways and should not cleaned up automatically. Individual protected properties can still be explicitly removed by specifying an empty list as value. There is also a flag, --cleanup-protected-props, which means protected properties are handled just like normal ones.

The lists of protected properties can be found in dsconfig/tangodb.py.

json2tango

This tool reads a JSON file (or from stdout if no filename is given), validates it and, optionally, configures a Tango database accordingly. By default, it will only check the current DB state, compare, and print out what actions would be performed, without changing anything. This should always be the first step, in order to catch errors before they are permanently written to the DB.

json2tango is idempotent. The idea is that if the database already contains the stuff specified in the config file, the tool should do nothing. Therefore, the tool tries to figure out the smallest set of database operations needed to get to the intended state. This also means that json2tango can be used to check the differences between a config file and a Tango database.

json2tango config.json

Inspect the output of this command carefully. Things in red means removal, green additions and yellow changes. Note that properties are stored as lists of strings in the DB, so don't be confused by the fact that your numeric properties turn up as strings.

[Pro-tip: if you're unsure of what's going on, it's a good idea to inspect the output of the -d argument (see below) before doing any non-trivial changes. It's usually less readable than the normal diff output, but garanteed to be accurate.]

A summary of the numbers of different database operations is printed at the end. This should be useful to double check, usually you have a good idea of e.g. how many devices should be added, etc.

Once you're convinced that the actions are correct, add the "-w" flag to the command line (this can be at the end or anywhere). Now the command will actually perform the actions in the Tango DB.

For safety and convenience, the program also writes the previous DB state that was changed into a temp JSON file (this is the same as the output of the -d flag). It should be possible to undo the changes made by swapping your input JSON file with the temp file.

Note that the tool in principle only concerns itself with the server instances defined in your JSON file. All other servers in the DB are left untouched. The exception is if your JSON contains devices that already exist in the DB, but in different servers. The devices will be moved to the new servers, and if any of the original servers become empty of devices, they will be removed. There is currently no other way to remove a server with dsconfig.

Some useful flags (see --help for a complete list):

• --write (-w) is needed in order to actually do anything to the database. This means that the command will perform the actions needed to bring the DB into the described state. If the state is already correct, nothing is done.

• --update (-u) means that "nothing" (be careful, see caveats below) will be removed, only changed or added. Again the exception is any existing duplicates of your devices. Also, this only applies to whole properties, not individual lines. So if your JSON has lines removed from a property, the lines will be removed from the DB as the whole property is overwritten, regardless of the --update flag.

• --include (-i) [Experimental] lets you filter the configuration before applying it. You give a filter consisting of a "term" (server/class/device/property) and a regular expression, separated by colon. E.g. "--include=device:VAC/IP.*01". This will cause the command to only apply configuration that concerns those devices matching the regex. It is possible to add several includes, just tack more "--include=..." statements on.

• --exclude (-x) [Experimental] works like --include except it removes the matching parts from the config instead.

Some less useful flags:

• --no-validation (-v) skips the JSON validation step. If you know what you're doing, this may be useful as the validation is very strict, while the tool itself is more forgiving. Watch out for unexpected behavior though; you're on your own! It's probably a better idea to fix your JSON.

• --dbcalls (-d) prints out all the Tango database API calls that were, or would have been, made to perform the changes. This is mostly handy for debugging problems. Since this is the real list of commands that are performed, it is guaranteed to correspond to reality.

• --sleep (-s) tweaks the time to wait between db calls. The default is 0.01 s. This is intended to lighten the load on the Tango DB service a bit, but it can be set to 0 if you just want the config to be done as fast as possible.

• --input (-p) tells the command to simply print the configuration file, but after any filters have been applied. It can be useful in order to check the result of filtering. If no filters are used, it will just (pretty) print whatever file you gave as input. This flag skips all database operations so it can be used "offline".

• --json (-j) [Experimental] prints a JSON format representation of the diff, instead of the default "human friendly" output. This can be more convenient if the output is to be consumed by another program, as JSON is easy to parse. For an example of the JSON format, see the json2tango tests. Since this feature is considered "experimental" the format may change in future versions.

Progress bar

The command should normally not take a very long time, but if there are many actions to perform it may need to work for a while. If you want some visual feedback while it is working, you can install the tqdm package from PyPI. When dsconfig detects that it's installed, it will use it to show a nice "progress bar" to visualize how far it has come.

It can also be automatically installed when installing dsconfig:

pip install dsconfig[progress]

Other features

Viewing JSON files

Reading a large, nested JSON file can be painful, but dsconfig has a solution; a hierarchical, terminal based JSON viewer! If you install the python packages urwid and urwidtrees, you can interactively view any JSON file by running

python  -m dsconfig.viewer something.json

From the start, everything is "folded" but you can navigate the structure by using the arrow keys and return to fold/unfold nodes.

Making a release of DSConfig

When it's time to release a new package, the procedure is currently like this:

• Make sure all the new stuff is in main, and pipelines are green.
• Add an entry in CHANGELOG.md (try to use "semver" logic for the version number).
• Create a tag in the repo, with the same version number.
• There should be a new build pipeline in CI, ending with a manually triggered step to upload to PyPI.

Done! Your new version should very shortly be available on PyPI.