eTiKeT Sync Agent - QUAlibrate Connector

Connector for synchronizing QUAlibrate calibration data with the eTiKeT platform. This connector scans a QUAlibrate data directory for node/workflow snapshots and syncs them to the cloud.

Installation

Install the QUAlibrate connector using the eTiKeT Sync SDK:

from etiket_sdk.sync import Connectors

# Install the latest version
Connectors.install_from_pypi("etiket-sync-agent-qualibrate")

The package is automatically discovered by etiket_sync_agent through the entry-point system. Once installed, you can verify with:

# List installed connectors
print(Connectors.list())

# Get details for the QUAlibrate connector
connector = Connectors.get("etiket_sync_agent_qualibrate")
print(connector)

Updating the Connector

# Update to the latest version
Connectors.update_from_pypi("etiket-sync-agent-qualibrate")

What Gets Synchronized

Each QUAlibrate snapshot is a directory containing a node.json (metadata) and a data.json (results, with references to .npz/.h5/.png/.json files). When a snapshot is synced, the following data is extracted and uploaded:

Data Processing

• Reference resolution: data.json reference strings (./arrays.npz#a.b.c) are resolved into real numpy arrays, xarray datasets, and images via the vendored loader.
• Raw-data extraction: Only the raw_data entry of the results tree is converted to netCDF. Other entries (e.g. fit_results) are intentionally excluded.
• Array-to-dataset convention (npz format): this is a best-effort reconstruction. Within a group of sibling numpy arrays, the leading 1-D arrays are treated as the coordinate axes in order (x, y, z, ...) and the remaining arrays as the measured values over the grid those axes span. If shapes don't fit this convention, arrays fall back to standalone variables with independent dimensions.
• Combining: per-measurement datasets are merged into one flat netCDF. Data variables are prefixed by their results-path; shared sweep axes stay shared (join="exact"). On conflict, every dataset is fully namespaced (lossless, collision-proof).
• Workflows: workflow snapshots typically carry no array data, in which case no netCDF file is uploaded.
• Thumbnails: with add_images_as_thumbnails on (the default), every image of the snapshot folder is added a second time as __thumbnail_<n>, which is what makes dataQruiser show it next to the dataset in the dataset list. The images are numbered most recent first, so the last picture the node produced is __thumbnail_1; images written in the same operation share a modification time and keep the order the files are walked in. These copies are hidden files (ranking = -1), so the file list of the dataset is unchanged: the images remain visible under their raw/ name. The sync agent downscales and re-encodes each image to keep a thumbnail around 100 kB, so large images are no longer skipped -- see sync_utilities.add_thumbnail.

Configuration

The QUAlibrate connector requires a QualibrateConfigData configuration with the following fields:

add_images_as_thumbnails can be changed after the sync source has been created, but only applies to snapshots synced from then on -- already synchronized snapshots are not revisited.

A scope is required for this connector (scope_requirement = REQUIRED); scope mapping is not supported.

The configuration is validated on creation: data_directory must exist and be a directory, and it must not overlap (be identical to, a parent of, or a subdirectory of) the path of an existing QUAlibrate sync source.

Example Configuration

Example using the etiket-sdk package:

from etiket_sdk.sync import SyncSources

SyncSources.create(
    name="my_qualibrate_source",
    connector_identifier="etiket_sync_agent_qualibrate",
    config_data={
        "data_directory": "~/.qualibrate/user_storage",
        "is_server_folder": False,
        "add_images_as_thumbnails": True
    },
    default_scope="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
)

Live Sync

Live synchronization is not supported by this connector. Snapshots are synced once they are complete (i.e. once a node.json is present in the snapshot directory).

Requirements

• Python >= 3.10
• numpy
• xarray
• h5netcdf
• Pillow

License

Copyright © 2025 QHarbor. All Rights Reserved. See LICENCE for details.