eTiKeT Sync Agent - Quantify Connector

Connector for synchronizing Quantify datasets with the eTiKeT platform. This connector scans Quantify data directories and syncs HDF5 datasets to the cloud.

Installation

pip install etiket_sync_agent_quantify

The package is automatically discovered by etiket_sync_agent through the entry-point system.

What Gets Synchronized

When a Quantify dataset is synced, the following data is extracted and uploaded:

Data Processing

• Grid conversion: Datasets are automatically converted to gridded format for efficient visualization
• Standard deviation detection: Variables ending with _u are marked as uncertainties for their base variable
• TUID extraction: The unique identifier follows the format YYYYMMDD-HHMMSS-fff-xxxxxx

Configuration

The Quantify connector requires a QuantifyConfigData configuration with the following fields:

Example Configuration

Example using the etiket-sdk package:

from etiket_sdk.sync import SyncSources

SyncSources.create(
    name="my_quantify_source",
    connector_identifier="etiket_sync_agent_quantify",
    config_data={
        "quantify_directory": "~/quantify-data",
        "set_up": "dilution_fridge_1",
        "is_server_folder": False
    },
    default_scope="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
)

Live Sync

The Quantify connector supports real-time synchronization of running measurements. We currenly do not support non-gridded datasets.

How Live Detection Works

A dataset is considered "live" (still being written to) if:

1. Recent modification: The HDF5 file was modified within the last 2 minutes
2. Contains NaN values: Quantify pre-allocates arrays with NaN values that get filled during measurement
3. No newer datasets: No other dataset directories have been created (indicating the measurement is still active)

How Live Sync Works

The XArrayReplicator class performs real-time replication:

1. Creates a qdrive dataset with SWMR (Single-Writer Multiple-Reader) HDF5 mode
2. Monitors file modifications by checking the source file's modification time
3. Tracks data using NaN cursor: Follows the position of NaN values to detect new data
4. Polls for updates every 0.5 seconds.
5. Completes when: All NaN values are replaced OR the 2-minute timeout is reached OR a newer dataset directory is created

Timeout Behavior

File Snapshot Safety

To avoid conflicts with concurrent file writes, the connector creates temporary snapshots of HDF5 files before reading:

with with_dataset_snapshot(file_path) as safe_file:
    dataset = xr.open_dataset(safe_file, engine='h5netcdf')
    # Process dataset safely

This is necessary because Quantify continuously overwrites the HDF5 file during measurement. Creating a temporary copy avoids file corruption or read errors when the source file is being written to.

Features

• Direct Quantify integration: Reads HDF5 datasets written by Quantify's data management system
• Live sync support: Real-time monitoring of running measurements via SWMR HDF5
• Automatic TUID parsing: Extracts timestamps and identifiers from Quantify's TUID format
• Grid conversion: Converts irregular datasets to gridded format for efficient visualization
• Auxiliary file upload: Syncs JSON, text, and other files alongside the main data

Requirements

• Python >= 3.10
• xarray
• h5py
• h5netcdf

License

Copyright © 2025 QHarbor. All Rights Reserved. See LICENCE for details.