eTiKeT Sync Agent

A Python library for synchronizing scientific data from various connector sources to the eTiKeT platform.

Overview

The eTiKeT Sync Agent provides a unified interface for automatically discovering and synchronizing experimental datasets from different data acquisition systems. It supports both database-backed sources (e.g., QCoDeS, Core-tools) and file-based sources (e.g., Quantify, Labber, generic folder structures).

Key Features

• Plugin-based connector system — Easily extensible architecture using Python entry points
• Multiple data source support — Native eTiKeT, QCodes, Quantify, Labber, Core-tools, and generic folder-based sources
• Live dataset synchronization — Real-time sync for ongoing measurements
• File converters — Transform data formats during synchronization
• Persistent tracking — SQLite database for tracking the sync state of each dataset
• Manifest-based file monitoring — Efficient change detection for file-based sources

Supported Connectors

Connector	Type	Description
etiket_sync_agent_native	Native	eTiKeT native datasets
etiket_sync_agent_folderbase	File-based	Generic folder structures
etiket_sync_agent_qcodes	Database	QCoDeS measurement databases
etiket_sync_agent_quantify	File-based	Quantify scheduler data
etiket_sync_agent_labber	File-based	Labber measurement files
etiket_sync_agent_core_tools	Database	Core-tools measurements

Installation

We recommend using the qdrive package to install and manage the sync agent as a background service. See the QHarbor documentation for setup instructions.

For standalone installation:

pip install etiket_sync_agent

Install connectors as needed:

pip install etiket_sync_agent_qcodes
pip install etiket_sync_agent_quantify

Usage

The sync agent is typically run as a background service managed by the qdrive package. Once configured, it continuously monitors your data sources and synchronizes new datasets to eTiKeT.

Running Manually

To run the sync agent manually:

python -m etiket_sync_agent

For more control (e.g., enabling debug logging), you can run the sync loop directly:

import logging
import sys

# Configure logging
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
    handlers=[logging.StreamHandler(sys.stdout)]
)

from etiket_sync_agent.run import sync_loop

# Run sync loop: 0 = run indefinitely, N = run N iterations
sync_loop(0)

CLI Tools

The package provides CLI tools to scaffold new connectors and converters. These scaffolds give you a ready-to-use package structure with all the boilerplate code in place—just fill in your implementation logic.

How It Works

The generated packages include pre-configured entry points in their pyproject.toml. Once installed, the sync agent automatically discovers and registers these connectors or converters via Python's entry points mechanism.

Tip: For working examples, check out the existing connector packages like etiket-sync-agent-qcodes on PyPI.

Need help? Contact us at support@qharbor.nl.

---

Generate Connector Scaffold

Creates a new sync agent connector package for synchronizing data from a specific source (e.g., a database, file system, or instrument).

generate_sync_agent_scaffold <connector> [options]

Arguments:

Argument	Description
connector	Identifier for your connector (e.g., qcodes, labber, my-instrument)

Options:

Option	Default	Description
--version	0.0.1	Initial package version
--dependency	—	Add a dependency (repeatable). You can also edit pyproject.toml manually.
--base	database	Base class template: database or file (see below)
--default-scope	REQUIRED	Scope assignment policy (see below)
--live-sync	False	Set to True if you plan to implement live synchronization
--level	-1	File-based only. Directory level for dataset detection (see below)

Understanding --base:

• database — Use when syncing from a database (e.g., SQLite from QCoDeS). The sync agent queries the database for new datasets.
• file — Use when syncing folder-based datasets where each folder or file represents a dataset.

Understanding --default-scope:

• REQUIRED — Users must specify the target scope (project/folder) for each sync source.
• OPTIONAL — A default scope can be inferred, but users may override it.
• DISABLED — The connector determines the scope automatically; users cannot change it.

Understanding --level (file-based only):

• -1 — Every folder containing a _QH_dataset_info.yaml file becomes a dataset.
• 1 — Each immediate subfolder of the root is a dataset.
• 2+ — Folders at that depth level become datasets.

Example — Database Connector:

# Create a connector for a custom SQLite database
generate_sync_agent_scaffold my-connector \
    --base database \
    --dependency "sqlalchemy>=2.0" \
    --default-scope REQUIRED

This generates:

etiket-sync-agent-my-connector/
├── pyproject.toml                      # Package config with entry points
└── etiket_sync_agent_my_connector/
    ├── __init__.py                     # Exports sync and config classes
    ├── my_connector_config_class.py   # Configuration dataclass
    └── my_connector_sync_class.py     # Sync logic (implement this)

Example — File-based Connector:

# Create a connector for experiment folders at depth level 2
generate_sync_agent_scaffold lab-data \
    --base file \
    --level 2 \
    --live-sync

---

Generate Converter Scaffold

Creates a new file converter package for transforming file formats during synchronization (e.g., CSV → HDF5). Converters are used by default in the folderbase connector, but you can also use them in your own custom connectors.

generate_converter_scaffold <converter> [options]

Arguments:

Argument	Description
converter	Identifier for your converter (e.g., csv-to-hdf5, mat-to-zarr)

Options:

Option	Default	Description
--version	0.0.1	Initial package version
--dependency	—	Add a dependency (repeatable)
--entry-name	—	Custom entry point name (defaults to sanitized converter name)

Example:

# Create a converter for CSV to HDF5
generate_converter_scaffold csv-to-hdf5 \
    --dependency h5py \
    --dependency pandas

This generates:

etiket-sync-agent-csv-to-hdf5-converters/
├── pyproject.toml                           # Package config with entry points
└── etiket_sync_agent_csv_to_hdf5_converters/
    ├── __init__.py                          # Exports converter class
    └── csv_to_hdf5_converter.py             # Converter logic (implement this)

After implementing your converter, install it and the sync agent will automatically use it for matching file types.

License

See LICENCE for details.