etiket-sync-agent-quantify 0.3.0b1

Quantify backend for eTiKeT sync agent

eTiKeT Sync Agent - Quantify Backend

Backend for synchronizing Quantify datasets with the eTiKeT platform. This backend scans Quantify data directories and syncs HDF5 datasets to the cloud.

Installation

pip install etiket_sync_agent_quantify

The package is automatically discovered by etiket_sync_agent through the entry-point system.

What Gets Synchronized

When a Quantify dataset is synced, the following data is extracted and uploaded:

Quantify Data	eTiKeT Field	Description
TUID (first 26 chars)	alt_uid	Unique identifier extracted from the folder name
Folder name (after TUID)	name	Name of the measurement
TUID timestamp	collected	Measurement timestamp parsed from TUID format
Variable long_name/name attrs	tags	Extracted from HDF5 dataset attributes
Config set_up	attributes.set-up	Experimental setup from configuration
HDF5 files	Data files	xarray datasets converted and uploaded
JSON/text files	Auxiliary files	Additional files in the dataset folder

Data Processing

• Grid conversion: Datasets are automatically converted to gridded format for efficient visualization
• Standard deviation detection: Variables ending with _u are marked as uncertainties for their base variable
• TUID extraction: The unique identifier follows the format YYYYMMDD-HHMMSS-fff-xxxxxx

---

Configuration

The Quantify backend requires a QuantifyConfigData configuration with the following fields:

Field	Type	Required	Description
quantify_directory	Path or str	Yes	Path to the Quantify data directory
set_up	str	Yes	Name of the experimental setup (added as set-up attribute)
is_server_folder	bool	No	Whether this is a server folder (default: False), e.g. on network drive of the university.

Example Configuration

Example using the etiket-sdk package:

from etiket_sdk.sync import SyncSources

SyncSources.create(
    name="my_quantify_source",
    backend_identifier="etiket_sync_agent_quantify",
    config_data={
        "quantify_directory": "~/quantify-data",
        "set_up": "dilution_fridge_1",
        "is_server_folder": False
    },
    default_scope="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
)

Live Sync

The Quantify backend supports real-time synchronization of running measurements. We currenly do not support non-gridded datasets.

How Live Detection Works

A dataset is considered "live" (still being written to) if:

1. Recent modification: The HDF5 file was modified within the last 2 minutes
2. Contains NaN values: Quantify pre-allocates arrays with NaN values that get filled during measurement
3. No newer datasets: No other dataset directories have been created (indicating the measurement is still active)

How Live Sync Works

The XArrayReplicator class performs real-time replication:

1. Creates a qdrive dataset with SWMR (Single-Writer Multiple-Reader) HDF5 mode
2. Monitors file modifications by checking the source file's modification time
3. Tracks data using NaN cursor: Follows the position of NaN values to detect new data
4. Polls for updates every 0.5 seconds.
5. Completes when: All NaN values are replaced OR the 2-minute timeout is reached OR a newer dataset directory is created

Timeout Behavior

Condition	Timeout	Action
No file modification	120 seconds (2 min)	Dataset marked as complete
Newer directory created	Immediate	Live sync stops, dataset marked complete
All NaN values filled	Immediate	Dataset marked complete

File Snapshot Safety

To avoid conflicts with concurrent file writes, the backend creates temporary snapshots of HDF5 files before reading:

with with_dataset_snapshot(file_path) as safe_file:
    dataset = xr.open_dataset(safe_file, engine='h5netcdf')
    # Process dataset safely

This is necessary because Quantify continuously overwrites the HDF5 file during measurement. Creating a temporary copy avoids file corruption or read errors when the source file is being written to.

---

Features

• Direct Quantify integration: Reads HDF5 datasets written by Quantify's data management system
• Live sync support: Real-time monitoring of running measurements via SWMR HDF5
• Automatic TUID parsing: Extracts timestamps and identifiers from Quantify's TUID format
• Grid conversion: Converts irregular datasets to gridded format for efficient visualization
• Auxiliary file upload: Syncs JSON, text, and other files alongside the main data

Requirements

• Python >= 3.10
• xarray
• h5py
• h5netcdf

License

Copyright © 2025 QHarbor. All Rights Reserved. See LICENCE for details.