opensemantic.base

Python models and controllers for the world.opensemantic.base page package.

Builds on oold-python (BaseController, LinkedBaseModel, cast(), _types registry) and opensemantic (OswBaseModel, compute_scoped_uuid).

Overview

• Auto-generated Pydantic models (v1 and v2): Database, WebService, DataTool, DataChannel, Person, Organization, etc.
• DataToolController - generic controller for any DataTool
• TimeSeriesDatabaseController - SQLite and PostgREST backends for time series storage

Architecture

opensemantic.base/
  _model.py            # auto-generated v2 Pydantic models (DO NOT EDIT)
  _controller_mixin.py # DataToolMixin, TimeSeriesDatabaseController mixins
  _controller.py       # v2 controllers
  __init__.py           # re-exports model + controller classes
  v1/                   # same structure for Pydantic v1

Data model

A DataTool instance is typed by the DataTool category, carries its data channels (each typed by a characteristic), and references its storage:

graph LR
    probeA["probeA<br/>(DataTool instance)"]
    probeA -- HasType --> DataTool["DataTool"]
    probeA -- HasStorageLocation --> archive["archive<br/>(Database)"]
    probeA -- HasDataChannel --> temp[":temp"]
    probeA -- HasDataChannel --> pressure[":pressure"]
    temp -- HasCharacteristic --> Temperature["Temperature"]
    pressure -- HasCharacteristic --> Pressure["Pressure"]

DataToolController

Extends DataTool with channel management, subdevice traversal, and data archiving.

from opensemantic.base import DataToolController

tool = DataToolController(
    name="sensor",
    label=[...],
    data_channels=[ch1, ch2],
    storage_locations=[db],
    auto_archive=True,  # auto-creates archive DB from storage_locations[0]
)

tool.get_all_channels()       # recursive across subdevices
tool.get_channel_owner(ch)    # find which device owns a channel
tool.to_json()                # only model fields (controller fields stripped)

Auto-archive from storage_locations

When auto_archive=True and no explicit archive_database is set, the controller auto-creates a LocalTimeSeriesDatabaseController from the first storage_locations entry (resolved via oold backend).

Typed read/write

Channels with a characteristic IRI enable typed serialization:

# Write: converts to base unit, strips defaults for compact storage
await tool.store_typed_data(DataToolMixin.StoreTypedDataParams(
    tool_osw_id=tool.get_osw_id(),
    rows=[DataToolMixin.TypedDataRow(ts=now, channel=ch, value=Temperature(value=300.0))],
))

# Read: resolves characteristic class via oold's _types registry
results = await tool.read_typed_data(DataToolMixin.ReadTypedDataParams(
    tool_osw_id=tool.get_osw_id(), channel=ch,
))
# results[0] is a Temperature instance with defaults restored

Subobject ID auto-computation

Inline subobject osw_id fields are auto-prefixed with the parent's osw_id:

Parent:  OSW<parent_uuid>
Channel: OSW<parent_uuid>#OSW<channel_uuid>

Fields with range in json_schema_extra are references to separate entities and are not prefixed.

Unloaded characteristic warning

On init, DataToolController checks if channel characteristic IRIs are present in oold's _types registry. Missing entries produce a warning with guidance to import the corresponding package.

TimeSeriesDatabaseController

Abstract base for time series storage, with SQLite and PostgREST implementations.

from opensemantic.base import LocalTimeSeriesDatabaseController

db = LocalTimeSeriesDatabaseController(name="archive", label=[...], db_path="./data.sqlite")
await db.create_tool(params)
await db.write_tool_channel_raw(params)
await db.read_tool_channel_raw(params)

DataToolView (Dashboard UI)

Interactive dashboard for visualizing archived time series data from DataToolControllers.

Features:

• Wunderbaum TreeGrid for channel selection with characteristic metadata
• Stacked Bokeh plots grouped by characteristic (temperature, pressure, etc.)
• Unit conversion via dropdown (e.g. K to C, Pa to hPa)
• Composite channel splitting (e.g. AirQuality into temperature + pressure sub-plots)
• Text channel log console with timestamped entries
• Native Plot Controls with a widget for every config field (grouping, auto-fetch, row limit, cache, downsampling, units, time range)

Interactive demo: channel selection, plotting, unit switching

Channel selection with stacked plots grouped by characteristic

Unit conversion via dropdown (K to C)

Text channel log console with timestamped entries

from opensemantic.base.view import (
    DataToolView,
    DataToolViewConfig,
    DataToolPlotControlsConfig,
)

view = DataToolView(
    controllers=[ctrl],
    config=DataToolViewConfig(
        lang="en", plot=DataToolPlotControlsConfig(auto_fetch=True)
    ),
    url_sync=True,  # optional: persist the full state in the URL
    title="My Dashboard",
)
view.servable()  # for panel serve

See examples/datatool_dashboard.py for a full working example.

To regenerate the screenshots after UI changes, see docs/generate_screenshots.py.

Export

A toolbar at the top of the plot card exports the currently plotted data and plot (requires the export extra):

• Download CSV / XLSX - a unit-aware pint-pandas table (one column per series, its unit written in a dedicated header row via dequantify()), matching exactly what is plotted (unit conversion + normalization). Rows are capped at EXPORT_MAX_ROWS (default 1,000,000) via uniform downsampling.
• Download plot (HTML) - a standalone interactive Bokeh document (bokeh.embed.file_html). PNG is available from the Bokeh toolbar's save tool.

The rendered figures are exposed via view.figures so a host app can add its own annotations; view.export_series() returns the plotted series as tidy records.

Export covers every checked channel, numeric and text-log alike, while the plot HTML is numeric-only. Column names must stay unique: composite sub-fields share their parent channel label, and a duplicate name makes df[label] return a DataFrame instead of a Series, which breaks the per-column dequantify(). Composite series are therefore labelled <channel>/<field>, with a dedupe safety net in _series_to_dataframe.

HTML export builds a fresh set of figures rather than reusing the live ones: bokeh.embed.file_html raises RuntimeError: Models must be owned by only a single document for a figure already rendered in a pn.pane.Bokeh, and copy.deepcopy on a figure fails too. Both views therefore split figure construction into a pure _make_figures(), used for the live panes and reused by _export_figures().

Configuration and URL sync

The config is the complete, JSON-serializable state of a view, composed to mirror the panels: component configs (TreeConfig, PlotControlsConfig, DownsampleConfig) make up a view's config (DataToolViewConfig, ProcessObjectViewConfig), and a host composes several of those into one aggregate model.

• Each view owns its config class (view.config_cls) and exposes get_config() / set_config(cfg) / on_config_change(cb). User edits write through into the config; set_config applies a config back to the widgets.
• TreeConfig.source reuses Wunderbaum's native node JSON, so the selection round-trips without a parallel model.
• UrlConfig[T] (Pydantic <-> pn.state.location) persists a config in the URL in COMPRESSED_BASE64 (default) or human-readable PLAIN_KEYS mode. Opt in per view with url_sync=True (and url_mode=), or a host URL-syncs one aggregate parent config. See examples/composed_dashboard.py for the multi-view composition + URL-sync pattern.

Two views composed under one aggregate config, URL-synced in PLAIN_KEYS mode (readable ?app.reactor... params in the address bar).

Implementation notes

Behaviour of the Panel/Bokeh stack that the view code depends on. Read this before touching figure building, zoom/reset or layout CSS in src/opensemantic/base/view/.

Linked x-axes need one shared range at creation. Create a single DataRange1d and pass it to every figure while building it:

shared_x = DataRange1d()
fig = bk_figure(x_range=shared_x, ...)   # every figure

Reassigning f.x_range afterwards races the render and leaves some plots unlinked, so panning moves only part of them. The y-ranges stay per figure, one scale per characteristic.

Reset events do not reach the server. figure.on_event(Reset, cb) never fires under pn.pane.Bokeh: Bokeh events are not forwarded, while model property changes sync reliably. Bridge it through a property:

self._reset_bridge = ColumnDataSource(data={"n": [0]})
self._reset_bridge.on_change("data", self._on_reset_bridge)
fig.js_on_event(Reset, CustomJS(args={"bridge": self._reset_bridge},
    code="bridge.data = {n: [(bridge.data.n[0] || 0) + 1]};"))

m.reset.emit() does fire js_on_event, but client-side only, so it is useful in tests and not a substitute for the bridge.

Shadow-DOM CSS. Page-level pn.config.raw_css does not reliably reach elements inside a component's shadow root. For a Panel AnyWidgetComponent such as Wunderbaum, pass stylesheets=[css], which injects into the component's own shadow root. That is how the tree grid's hardcoded inline width: 800px is overridden.

Content scrolling is owned by panelini. .content lacked the overflow-y: auto that .sidebar has, so tall stacked plots spilled past the wallpaper; fixed upstream in panelini#37. With that in place, leave _plot_col uncapped, otherwise a max_height plus scroll=True adds a second, nested scrollbar.

Stale Bokeh models. After a pane rebuild, old figures and ranges linger in doc._all_models. When inspecting through JS, filter to the live ones or you will read a phantom range.

Server-side downsampling

Large time series are downsampled on the server so the plot only transports the points the current zoom level can show. This is driven by PlotControlsConfig.downsample:

from opensemantic.base.view import (
    DataToolViewConfig,
    DataToolPlotControlsConfig,
    DownsampleConfig,
)

config = DataToolViewConfig(
    plot=DataToolPlotControlsConfig(
        downsample=DownsampleConfig(
            enabled=True,      # downsample when the backend supports it
            max_points=2000,   # target points per channel
            method="auto",     # auto | sample | average | minmax
            edge_anchors=True, # keep the window's first/last real datapoints
        )
    )
)

It only engages on a PostgREST/TimescaleDB backend (the downsample_tool_channel RPC). On a SQLite/local backend, the RPC being absent, or any error, the read silently falls back to the full-resolution path - downsampling never breaks a read. The DataToolView plot also reloads at a finer resolution when you zoom in.

Strategies (N = number of buckets):

• sample (default): one real datapoint nearest each bucket center. Schema agnostic; works for scalar and composite channels. N rows.
• average: structure-preserving deep average per bucket (every numeric leaf averaged, non-numeric keys carried), bucket-center timestamp. N rows.
• minmax: the real min and max datapoint of every numeric sub-characteristic per bucket. Scalar: 2N real rows; composite: the real per-leaf extremes. Best at preserving spikes.
• auto: minmax for numeric channels, sample for text channels.

average/minmax skip non-numeric leaves (comments, labels) and fall back to sample when a channel has no numeric leaf at all.

Unit normalization caveat: average and minmax compare and combine the bare stored numbers per leaf, so they are only correct when all stored values of a leaf share the same unit. The archive stores base-unit-normalized values, but data ingested without normalization (mixed units in one channel) will produce wrong average/minmax results. sample returns whole real rows and is unaffected.

The RPC lives in pgstack's postgres/config/optional/100_init_tsdb_schema.sql (uses only core, Apache-2 TimescaleDB; no toolkit dependency). It is created at database init; apply it manually (psql -f / pgAdmin) on an already-running cluster. To measure the speedup, see benchmarks/bench_downsample.py.

examples/downsample_demo.py is an interactive demo: four channels carry the same 100k-point signal (with narrow spikes), one per strategy - the channels are named raw, sample, average, minmax. Selecting raw loads slowly with full detail; sample/average load fast but drop the spikes; minmax keeps them. Box-zoom into a flat stretch and click "Load current range" to re-fetch that window at finer resolution - the hidden spikes reappear on sample/average. Needs a running pgstack with the RPC applied; seed once with python examples/downsample_demo.py, then panel serve examples/downsample_demo.py.

Full window, all four channels: raw and minmax keep the spikes; sample and average smooth them away at the coarse full-window resolution.

Box-zoomed around a spike on the sample channel: only the coarse full-window points are shown, so the spike is still hidden.

After "Load current range": the window is re-fetched at finer buckets and the hidden spike reappears. The toolbar reset returns to the full window.

To regenerate these, see docs/generate_downsample_screenshots.py.

ProcessObjectView (Process/Object Dashboard UI)

Where DataToolView is centered on data tools, ProcessObjectView is centered on the objects (samples, specimens, ... - any Item) that pass through processes. It answers "how did measurement X compare across the runs my objects went through?" by overlaying repeated runs on a common, time-normalized axis.

The view walks these relations - an object is a process input, a process uses data tools, and a data tool has channels:

graph LR
    process[":process"]
    process -- HasInput --> sample["sample (Item)<br/>the 'object'"]
    process -- HasTool --> probeA["probeA<br/>(DataTool)"]
    process -- HasStartDateAndTime --> startTime["start_date_time"]
    process -- HasEndDateAndTime --> endTime["end_date_time"]
    probeA -- HasDataChannel --> channels[":temp, :pressure"]

start_date_time / end_date_time define each run's plot window; the view normalizes every run to its first data point (t=0).

Pick objects (tree 1) and a channel under a process type (tree 2); each run is overlaid from t=0.

Two trees drive the plot:

1. Objects - the Item instances to compare.
2. Process types -> channels - for each process type the objects went through, the channels of the data tools used in those runs.

Tree 2 is aggregated for selection only (tick once instead of ticking the same channel on every run and tool). The aggregation is co-presence aware:

• data tools of the same type that run together in a process stay as separate entries (distinct measurement points);
• data tools that only ever appear in different runs are treated as drop-in replacements and merged into one … [n channels] entry (the actual channels are listed in its tooltip).

Evacuation ran both probes together (separate per-instance entries); Heating swapped the probe between runs (merged DataTool/… [2 channels] entries).

Selecting an object + a channel entry plots every real channel that object has data on - fanning out across process runs and co-present tools. Each line is normalized to its own run (first data point at t=0, x-axis in seconds), grouped by characteristic, and gets a distinct legend entry (object / process / data tool / channel).

One channel selection fans out to two runs of the same sample, overlaid from t=0 for comparison.

Adding Heating/DataTool/temp and …/pressure on top of the Evacuation selection overlays both processes, grouped into separate Temperature and Pressure plots; the merged Heating entries resolve to probe A for Sample 1 and probe B for Sample 2 (drop-in replacements compared across objects).

from opensemantic.base.view import ProcessObjectView, ProcessObjectViewConfig

view = ProcessObjectView(
    objects=objects,        # list[Item]
    processes=processes,    # list[Process] (filtered to those with start+end
                            # time and >=1 DataTool whose data you can load)
    controllers=controllers,  # list[DataToolController], matched to process tools
    config=ProcessObjectViewConfig(lang="en"),
    title="Process / Object Archive View",
)
view.servable()  # for panel serve

DataToolView and ProcessObjectView share their plot/unit/config machinery via BaseDataView, and both support embeddable=True (exposing sidebar_cards / main_cards) so a host app can combine them.

See examples/process_dashboard.py for a full working example, and docs/generate_process_screenshots.py to regenerate these screenshots.

Installation

pip install opensemantic.base            # models only
pip install opensemantic.base[controller] # + aiosqlite, postgrest
pip install opensemantic.base[view]       # + panel, bokeh, panelini, pint
pip install opensemantic.base[export]     # + pandas, pint-pandas, openpyxl (data/plot export)

Running the examples

git clone https://github.com/OpenSemanticWorld-Packages/opensemantic.base-python.git
cd opensemantic.base-python
pip install -e .
pip install opensemantic.base[controller] opensemantic.base[view] opensemantic.base[export]
pip install opensemantic.characteristics.quantitative   # typed characteristics
pip install nest-asyncio                                 # examples/datatool_dashboard.py
pip install PyJWT                                        # examples/downsample_demo.py (needs a database)

panel serve examples/datatool_dashboard.py --dev

Testing

pytest tests/test_controller.py

PostgREST integration tests require a running pgstack instance. To enable them:

1. Start pgstack: docker compose -f docker-compose.yml -f docker-compose.example-tsdb.override.yml up -d
2. Copy tests/.env.example to tests/.env and fill in TEST_PGRST_URL and TEST_PGRST_JWT_SECRET
3. Run tests - PostgREST tests are skipped unless both env vars are set

Browser tests

tests/test_export_browser.py drives panel serve with Playwright. Four things it has to get right:

• examples/datatool_dashboard.py pins its SQLite database next to the script, so the test serves a copy in a temp directory rather than contending with a running demo on the shared file.
• Bokeh 3 renders widgets in a shadow root, so find buttons with get_by_role("button", name=...), which pierces it, not a light-DOM querySelectorAll.
• Readiness is a raw TCP connect, not an HTTP GET: a corporate proxy may divert localhost GETs.
• Port cleanup parses netstat by column, not by the state word, which is localized (German Windows prints ABHOEREN, not LISTENING).
• FileDownload generates its payload lazily on click, so wrap the click in expect_download.