PyTango DatabaseDS

This is a "fork" of some internal code from pytango, that runs a Tango database device server. This code was originally written by Tiago Coutinho. It uses sqlite for storing data and therefore doesn't have any dependencies beyond python and PyTango.

It's currently under heavy development, but is pretty usable. Please report any problems!

The repo also contains a test suite that runs on both this device and a C++ database, for compatibility checking.

For discussion of the future of this device, see https://gitlab.com/tango-controls/pytango/-/issues/626

Usage

Create a Python environment and install the package:

pip install pytango-db
TANGO_HOST=localhost:11000 PyDatabaseds 2

On Windows Command Prompt, use set instead of export:

pip install pytango-db
set TANGO_HOST=localhost:11000
PyDatabaseds 2

To work from source, check out this repository, create a Python environment, and run pip install -e . before starting the server with the same command.

Now you should be able to use any normal Tango stuff like Jive etc, as long as you set TANGO_HOST=localhost:11000. You can pick any port, but 10000 might be taken if you already have a Tango installation locally.

SQLite database

The SQLite database is saved in the current working directory. It's possible to have any number of databases running on the same host, with different ports, but be careful about sharing the same database file, it will likely cause problems. You can configure where it is stored by setting the environment variable PYTANGO_DATABASE_NAME. For a pure in-memory database, use :memory:.

Schema versions

New SQLite databases use schema version 1. Existing files with another schema version are rejected without being modified. Migrate a version 0 file explicitly:

pytango-db-migrate-sqlite tango_database.db

The migration is atomic and idempotent. If duplicate rows prevent a required unique index from being created, it rolls back without changing the database.

For debugging, you can enable logging:

TANGO_HOST=localhost:11000 PyDatabaseds --logging_level=2 2

pytest fixture

This project also includes two pytest fixtures to start a database:

• pytango_db, a function-scoped fixture (database is started and stopped after each test)
• session_pytango_db, a session-scoped fixture (database is started only once and stopped at the end of the test session)

To use one of this fixture in your project, add pytango-db to your tests requirements.

These fixtures will start a database on a random port and return the used TANGO_HOST variable, that has been set in the environment. The database runs in memory.

WARNING: While you can mix both fixtures in your tests, be careful if you create a DeviceProxy as cpptango is caching the database connection. You might point to the wrong database. Pass the full TRL including the tango_host returned by the fixture.

Custom server fixtures

It's common to use pytango-db to run your Tango devices in a realistic way for tests.

run_device_server is a helper to configure and run a device server. You can use it directly in tests, like so:

from databaseds.fixtures import run_device_server

def my_test(pytango_db):
    my_config = dict(
        server_instance="MyDevice/1",
        device_classes={
            MyDevice: {
                "test/mydevice/1": {
                    "properties": {"SomeProperty": ["hello"]}
                }
            }
        }
    )
    with run_device_server(**my_config) as (proxy,):
        # Within this block, the device is available via the proxy
        proxy.read_attribute("hello")
        ...

Note that MyDevice here is the actual device class. When specified this way, the server will be created from the class(es) specified. The class can also be specified as a string, in which case we'll start the server by running e.g. MyDevice, which must be in the $PATH.

If your device is not installed or doesn't have an entry point corresponding to the server name, you can also supply the commandline argument to tell the function how to start it.

run_device_server(
    server_instance="MyDevice/1",
    device_classes={
        "MyDevice": {
            "test/mydevice/1": {}
        }
    },
    commandline="python mydevice.py 1 -v4",
)

If you use the same device server in many tests, you can save some code duplication by turning it into a "fixture":

@pytest.fixture
def my_device():
    with run_device_server(...) as (proxy,):
        yield proxy

Tests using the my_device fixture will now always have the device running, and my_device will contain a proxy to it.

For more documentation see the docstring and tests.

Use cases

• Local Tango development. Easy setup, possibility of having several databases and switching between them, etc.
• Continuous integration pipelines. Simpler setup, no service depencencies.
• Small lab setups?

Drawbacks

• Still buggy
• Likely lower performance than the C++ device
• Tango serializes commands addressed to one database device. SQLite uses one worker and one writer transaction at a time.

Tests

The test suite is implemented as two very simple tests, one runs agains the pytango database device (provided by a fixture) and the other assumes there is a C++ database server running at the configured TANGO_HOST.

Run the self-contained suite with:

pytest

Run the local bulk-write benchmark separately with:

pytest -m benchmark -s

The test are run multiple times, with data provided by a JSON "spec". The spec contains separate definitions for each test run, e.g.

    {
        "name": "DbAddDevice",
        "setup": [
            {
                "command": "DbAddDevice",
                "argument": [
                    "Dummy/{name}",
                    "test/dummy/{name}",
                    "Dummy"
                ]
            },
        ],
        "check": [
            {
                "command": "DbGetDeviceInfo",
                "argument": "test/dummy/{name}",
                "result": [
                    [
                        0,
                        0
                    ],
                    [
                        "test/dummy/{name}",
                        "nada",
                        "0",
                        "Dummy/{name}",
                        "nada",
                        "?",
                        "?",
                        "Dummy"
                    ]
                ]
            }
        ],
        "teardown": [
            {
                "command": "DbDeleteServer",
                "argument": "Dummy/{name}"
            }
        ]
    },

Here, the test is named after "DbAddDevice" since this is the command that is being tested.

The "check" section contains several command runs that will be executed on the database device under test. "command" and "argument" describe what to run, and the optional "result" describes the expected outcome. It will be compared against the actual result, to determine if the test was successful. Multiple commands are run in sequence.

The "teardown" section is optional, and will be run after the "check" section. If any modification is made to the database, this should be implemented in order to restore the database as far as possible, so that the test doesn't interfere with any other tests. At least in the case of the C++ database, we can't wipe it completely between tests so make sure to remove anything that was added by the test.

Protip: getting the "teardown" right and not forgetting something in the database can be tricky. However you can check this by setting the env var PYTANGO_DATABASE_NAME=/tmp/test.sqlite when running the tests. That will cause the test database to be stored on disk instead of memory, and allows you to inspect the contents of the database afterwards, e.g. using sqlite3. Note that some tables such as *_history aren't possible to clean up via Tango.

There is another optional section, "setup" that is run before "check" and can be used in order to create prerequisites for the checks, where the result doesn't matter.

The test may also contain a "comment" field for extra information, and a boolean "skip" field which if true will cause the test to be skipped. At the time of writing, many command tests are still empty "stubs", set to be skipped.

If a test is failing on one of the implementations, it can be marked as xfail by adding the field expected_to_fail_on. This field shall be a list where each entry is either:

• a backend string: cppdb or pydb
• a dict with: backend set to cppdb or pydb, and an optional boolean strict

Examples:

"expected_to_fail_on": ["cppdb"]

"expected_to_fail_on": [
    {
        "backend": "cppdb",
        "strict": false
    }
]

If strict is omitted, it defaults to true. Don't forget to add a comment explaining why this test is expected to fail and/or a link to an issue.

Check

The check section also has a few more features:

result is a list or a string, depending on the type returned by the command. Items in the list are usually strings, but sometimes it can be useful to match them in a less exact way, when we only care about the "format" of the item:

• null means we don't care what the item is, it matches anything.
• {"regexp": "a.*"} is an example of a regular expression matching any string starting with "a".
• {"timestamp": "%Y-%m-%d"} matches any date in ISO format e.g. "2024-10-01". The matcher uses ordinary strptime syntax.

Ordinary string values may contain special template syntax e.g. {name}. Before matching, it will be substituted with the name of the current test. This makes it easy to name things uniquely, which simplifies tracking down cases of stuff being left in the database by mistake (which should have been cleaned up in the "teardown" step). For now, only "name" is available.

result_slice takes a list of arguments, corresponding to the python slice builtin; "start", "stop" and "step". Each is an integer or None. This can be used to get only part of the result, in case it contains things we aren't interested in.