Skip to main content

aio-sensorthings

An asyncio Python library and SDK for OGC SensorThings API (STA) servers such as FROST-Server.

Summary

aio-sensorthings copies data from one SensorThings server to another and keeps it up to date in near real time:

  • Bootstrap (HTTP): reads Locations, Things, Sensors, ObservedProperties and Datastreams from the source in dependency order and creates them on the target. Source IDs are rewritten to target IDs, and the mapping is kept in a state store.
  • Scoped sync: limit the copy to a subgraph via an STA navigation path (ObservedProperties(5674)/Datastreams) or an OData filter. Dependencies are resolved automatically.
  • Live data (MQTT): new Observations, entity changes and newly created entities are streamed from the source's MQTT broker to the target.
  • Pluggable parts: state store (in-memory or Redis), transformers, target (STA HTTP or Kafka), and Basic/OAuth authentication. Multiple instances can share the work through a Redis-based federation coordinator.
  • Observability: optional OpenTelemetry spans and metrics (API only, no SDK dependency).

Historical Observations are not copied during bootstrap; only Observations published after the MQTT subscription starts are forwarded.

The code follows a hexagonal layout (models, ports, adapters, pipeline, config, transforms) under src/aio_sensorthings/.

Installation

Requires Python 3.13 or 3.14.

pip install "aio-sensorthings[all]"   # or pick extras: http, mqtt, redis, kafka, otel

Usage

Scoped copy of all Datastreams of one ObservedProperty from a remote server to a local FROST-Server, followed by live MQTT forwarding (state kept in Redis):

import asyncio

import redis.asyncio as aioredis

from aio_sensorthings import (
    EntitySyncer,
    IdentityTransformer,
    ProcessRunner,
    RedisStateStore,
    STAClient,
    STAEntityType,
    STAHttpSource,
    STAHttpTarget,
    STAMqttSubscriber,
    StandaloneCoordinator,
)
from aio_sensorthings.config import MqttSettings


async def main() -> None:
    redis_client = aioredis.from_url("redis://localhost:6379/0", decode_responses=True)
    mqtt = MqttSettings(broker_host="iot.hamburg.de", broker_port=1883, qos=1, clean_session=False)

    async with (
        STAClient("https://iot.hamburg.de") as source_client,
        STAClient("http://localhost:8080/FROST-Server") as target_client,
    ):
        syncer = EntitySyncer(
            source=STAHttpSource(source_client),
            target=STAHttpTarget(target_client),
            state=RedisStateStore(redis_client, key_prefix="demo"),
            transformer=IdentityTransformer(),
            entity_types=[
                STAEntityType.LOCATION,
                STAEntityType.THING,
                STAEntityType.SENSOR,
                STAEntityType.OBSERVED_PROPERTY,
                STAEntityType.DATASTREAM,
            ],
            sync_by="name",
            source_mqtt=STAMqttSubscriber(mqtt),
            coordinator=StandaloneCoordinator(),
        )

        runner = ProcessRunner(
            syncer,
            bootstrap_kwargs={
                "entry_point": STAEntityType.DATASTREAM,
                "nav_path": "ObservedProperties(5674)/Datastreams",
            },
        )
        await runner.run()  # bootstrap, then subscribe until SIGINT/SIGTERM


asyncio.run(main())

More examples (federation, OpenTelemetry) are in examples/.


Entity mapping DSL

MappingTransformer maps fully expanded STA payloads to typed Pydantic models. The mapping is a YAML or JSON document: one section per entity type, one named block per output, one rule per target field.

version: 1
DATASTREAM:
  CountingField:
    when: {from: properties.art, equals: "Zählfeld"}   # output condition
    name: Thing.properties.assetID                      # shorthand for {from: ...}
    unit: properties.aggregationMethod
    method: {const: infrared}
    traffic_node_arm.direction:
      from: Thing.properties.richtung
      regex: '^Richtung (?P<num>\d+)$'
      group: num
    id: {is_id: true, id_source: [Thing.properties.assetID, properties.knotenName]}
    vehicle_class:
      from: properties.typ
      when: {from: properties.typ, regex: "^(Kfz|Rad)$"}  # field condition

Field rules

  • A scalar is a source path (Thing.Locations[0].location) and is shorthand for from.
  • const sets a literal value.
  • regex post-processes the source value; group picks a named or numbered group; match_all returns all matches as a list.
  • is_id with id_source (list of paths) and id_separator (default -) joins several sources into one ID.
  • Nested blocks and dotted keys are equivalent (a: {b: x} is a.b: x). A target field named like a rule key (from, const, when, ...) must use the dotted form.

Conditions (when)

when is optional on an output and on a single field. Without it, the output or field is always mapped.

Key Meaning
from Source path to test (required, never inherited from the field).
equals Exact match against a scalar (str, int, float, bool).
regex Regex searched in the value.
negate Optional, inverts the result.

Exactly one of equals or regex is required. Source values are converted with str() before matching.

  • Output condition: evaluated first. If it is false, the output is skipped and none of its field rules are evaluated.
  • Field condition: if false, the field is left out, so model defaults apply. A required field that is left out fails validation, and the output is skipped with a warning.
  • Missing source: if the source path resolves to nothing, the condition is false, even with negate. A warning is logged once per payload and path.
  • Source paths in conditions are included in the derived $expand (required_expands).

Prerequisites

For development, install one of the following package managers once on your machine before proceeding:

Package manager Install
pixi (default) curl -fsSL https://pixi.sh/install.sh | bash — pixi.sh
poetry pipx install poetry — python-poetry.org

For Docker workflows you also need Docker.


Development setup

pixi

Pixi manages Python, all conda-forge and PyPI dependencies, and the virtual environment in a single tool. The environment lives at .pixi/envs/ and is not committed to git.

# Install all dependencies (creates the environment from pixi.lock)
pixi install

# Open a shell with dev tools available
pixi shell -e dev

# Or run a single task without activating the shell
pixi run -e dev test
pixi run -e dev lint
pixi run -e dev format
pixi run -e dev typecheck

Add a new dependency:

pixi add numpy                     # conda-forge, runtime
pixi add --pypi requests           # PyPI, runtime
pixi add --feature dev pytest-mock # dev-only

Update all dependencies and regenerate the lockfile:

pixi update
git add pixi.lock && git commit -m 'chore: update pixi lockfile'

Common development tasks

All tasks are available via make. Run make with no arguments to execute lock → build-image → run-local.

Command What it does
make test Run the test suite
make lint Lint with ruff
make format Auto-format with ruff
make typecheck Type-check with mypy
make lock Update the dependency lockfile
make build-image Build the Docker image
make push-registry Push the image to the container registry
make run-local Start the app locally via docker-compose
make build-docs Build the Jupyter Book documentation
make clean-docs Remove the documentation build output

Pre-commit hooks

Code quality checks run automatically on every git commit. Install the hooks once after cloning:

pre-commit install

Run all hooks manually at any time:

pre-commit run --all-files

The hooks cover: ruff (lint + auto-fix), ruff-format, mypy, and standard file hygiene (trailing whitespace, YAML validity, merge-conflict markers, large files).


Docker

Copy .env.example to .env and fill in the values (copier does this automatically on first scaffold):

cp .env.example .env
Command What it does
make build-image Builds ${CONTAINER_REGISTRY}/${IMAGE_NAME}:${IMAGE_TAG}
make run-local Runs the image via docker-compose-dev.yaml
make push-registry Pushes to the container registry

Key Docker files:

File Purpose
Dockerfile Multi-stage build (adapts to the chosen package manager)
docker-compose-build.yaml CI/CD image build
docker-compose-dev.yaml Local dev/test run with secrets and env vars
docker-compose.yaml Production deployment spec

Project structure

├── src/aio_sensorthings/                    # importable package
├── tests/                                   # pytest test suite
├── docs/                                    # Jupyter Book documentation
├── data/                                    # local data
├── assets/                                  # images, figures
├── reports/                                 # generated reports and outputs
├── examples/                                # usage examples
├── scratch/                                 # throwaway experiments (not tested)
└── secrets/                                 # local credentials — never committed

VS Code setup

Recommended extensions (install via the Extensions panel or .vscode/extensions.json):

Extension Purpose
ms-python.python + ms-python.pylance Python language support
charliermarsh.ruff Linting and formatting (replaces black + isort)
ms-python.mypy-type-checker Type checking

Formatting on save and import sorting are pre-configured in .vscode/settings.json.


Updating the template

To pull in improvements from the project template into this project:

copier update . --trust

Copier merges template changes with your local edits. Review the diff carefully before committing.

Metadata

Release files for aio-sensorthings 0.8.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for aio-sensorthings 0.8.1
File Size Uploaded Provenance Actions
aio_sensorthings-0.8.1.tar.gz 704.2 kB Without provenance Details for aio_sensorthings-0.8.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for aio-sensorthings 0.8.1
File Interpreter ABI Platform Actions
aio_sensorthings-0.8.1-py3-none-any.whl Python 3 none any Details for aio_sensorthings-0.8.1-py3-none-any.whl

Total release size: 791.4 kB

Release files / aio_sensorthings-0.8.1.tar.gz

Download URL aio_sensorthings-0.8.1.tar.gz
Size 704.2 kB
Tags Source
SHA-256 checksum
How to use checksums
44b3a7293c191fa1591a786976204d9a1f957a1b2fb69ceb8446dda939850ba8
BLAKE2b-256 checksum
How to use checksums
d69018a37266d083ce689b69cb5e54d391721b51defa2d0536eeb7c69732c986
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.11.8 {"installer":{"name":"uv","version":"0.11.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / aio_sensorthings-0.8.1-py3-none-any.whl

Download URL aio_sensorthings-0.8.1-py3-none-any.whl
Size 87.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
20fd49f067b3376ea1b8e7bcda26e4214d0000b9253f4189f327c6eb194d36e3
BLAKE2b-256 checksum
How to use checksums
c6f742aa2d8fd1d57cc5544a94cba3aef5dc1d528911afc22a4270c91674dfc2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.11.8 {"installer":{"name":"uv","version":"0.11.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for aio-sensorthings 0.8.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page