Pre-release

This release is a pre-release and may not be stable for production use.

Dataerai Python SDK

Python client for the Dataerai transfer daemon.
Supports authenticated uploads, downloads, and metadata operations. The daemon is started automatically if it is not already running.

Requirements

Python ≥ 3.10
The dataerai binary must be installed and on PATH (or pass binary_path explicitly)
The user must be logged in via dataerai auth login before calling auth_status() / upload() / download()

Installation

pip install --pre "dataerai-sdk>=0.2.0b1,<0.3"
# or, from source:
pip install -e sdk/python/

Optional ML adapters are installed separately so the base SDK stays small:

pip install --pre "dataerai-sdk[ml]>=0.2.0b1,<0.3"          # PyTorch Zarr datasets
pip install --pre "dataerai-sdk[ml-keras]>=0.2.0b1,<0.3"    # Keras batches
pip install --pre "dataerai-sdk[ml-hls4ml]>=0.2.0b1,<0.3"   # HLS4ML conversion
pip install --pre "dataerai-sdk[notebook]>=0.2.0b1,<0.3"    # IPython %dataerai magic

Quick start

from dataerai import DataeraiClient

with DataeraiClient(binary_path="/usr/local/bin/dataerai") as client:
    # Check auth
    status = client.auth_status()
    print(f"Logged in as {status.user_email} ({status.user_id}), token expires {status.expires_at}")
    if status.user_id is None:
        raise RuntimeError("Upgrade the dataerai CLI to use user-owned uploads")

    # Upload a user-owned file. owner_id is a UUID, not an email address.
    result = client.upload(
        "/path/to/data.csv",
        title="My dataset",
        owner_type="user",
        owner_id=status.user_id,
        record_type="dataset",
        on_progress=lambda p: print(f"  {p.percent:.0f}%  {p.rate_mbps:.1f} MB/s"),
    )
    print(f"Uploaded  asset_id={result.asset_id}  content_id={result.content_id}")

    # Download it back
    dl = client.download(result.asset_id, dest_dir="/tmp/downloads")
    for f in dl.files:
        print(f"  {f.local_path}  ({f.size:,} bytes)")

    # Read / update metadata
    meta = client.get_metadata(result.asset_id)
    updated = client.set_metadata(
        result.asset_id,
        title="My dataset v2",
        record_type="analysis",
        tags=["csv", "demo"],
    )

    # Link a derived result to the source asset that produced it
    relationship = client.create_relationship(
        result.asset_id,
        "source-asset-id",
        relationship_type="derived_from",
        analysis_mode="non_destructive",
        qualifiers={"tool": "pycroscopy"},
    )
    related_id = relationship.related_asset["id"] if relationship.related_asset else "source-asset-id"
    print(f"Linked via {relationship.type} to {related_id}")

Notebook magic

After signing in with the Dataerai CLI, a notebook only needs a destination path. The first component names an existing visible project; missing collection components are created beneath it:

DESTINATION_COLLECTION_PATH = "Research / Experiments / July"

%load_ext dataerai.magics
%dataerai $DESTINATION_COLLECTION_PATH

The magic publishes a dataerai_session variable. Its uploads automatically target the selected project and collection:

dataset = dataerai_session.find_asset(
    "did:dataerai:beta:asset:...",
    title="input.csv",
)
dataerai_session.download(dataset, "source-data")
result = dataerai_session.upload(
    "analysis.csv",
    metadata={"component": "derived-output"},
)

Use %dataerai --as run $DESTINATION_COLLECTION_PATH to choose a different session variable. Regular Python code can call connect_notebook(DESTINATION_COLLECTION_PATH) instead.

Trace a notebook run

Add --trace to capture every subsequent cell's source, timestamps, stdout, stderr, structured Python logging records, rich display outputs, returned value, and error. Uploads and downloads through the session join the same run automatically, following the run-centered provenance pattern used by the QICK integration:

%dataerai --trace --notebook analysis.ipynb --title "July analysis" $DESTINATION_COLLECTION_PATH

# Run analysis cells and upload their products through dataerai_session.
result = dataerai_session.upload(
    "analysis.csv",
    record_type="analysis",
    metadata={"component": "derived-output"},
)

%dataerai --finish

%dataerai --finish waits until its cell completes, then uploads a JSON execution-log asset with record_type="log". Every uploaded asset carries a shared notebook_run_id plus dataerai-notebook-trace and notebook-run:<run-id> tags. The log is linked to each recorded input and product with records_telemetry, and the published result is available as dataerai_trace. Run identity is merged onto an existing same-title asset, so re-running a fixed notebook or output filename keeps both run tags searchable.

Tracing is deliberately opt-in because cell source and output can contain secrets or sensitive research data. Review the notebook before enabling it. The recorder does not read environment-variable values or scan/upload arbitrary files; a file becomes a recorded product only when the notebook uploads it through the traced session. Finish an active trace before starting another one in the same Python kernel.

API reference

`DataeraiClient(*, socket_path, binary_path, auto_start, start_timeout_s, request_timeout_s)`

Parameter	Default	Description
`socket_path`	`$DATAERAI_SOCKET` or `/run/user/<uid>/dataerai-transfer.sock`	Unix socket path
`binary_path`	`None`	Path to the `dataerai` binary (required for `auto_start`)
`auto_start`	`True`	Spawn the daemon if the socket does not exist
`start_timeout_s`	`10.0`	Seconds to wait for the daemon socket to appear
`request_timeout_s`	`30.0`	Per-request timeout in seconds

Use as a context manager (with DataeraiClient(...) as client:) for automatic cleanup, or call client.connect() / client.close() manually.

Methods

Method	Returns	Description
`auth_status()`	`AuthStatus`	Logged-in user ID, email, and token expiry; `user_id` is `None` with older daemons
`list_tree()`	`list[TreeNode]`	List visible workspaces, projects, and collections
`create_collection(title, *, owner_type, owner_id, parent_id=None)`	`Collection`	Create a collection
`ensure_collection_path(path)`	`CollectionDestination`	Resolve `Project / Collection / ...`, creating missing collection components
`search_assets(query, ...)`	`AssetSearchPage`	Search one page of visible assets
`find_assets(query, ...)`	`list[AssetSummary]`	Search and collect every result page
`create_project(name, *, description="")`	`Project`	Create a project (+ root collection, owner membership, default allocation); requires write scope
`upload(local_path, *, title, owner_type, owner_id, record_type=None, ...)`	`UploadResult`	Upload a file with an optional server-validated record type; blocks until complete
`download(asset_id, dest_dir, ...)`	`DownloadResult`	Download latest asset content; blocks until complete
`get_metadata(asset_id)`	`AssetMetadata`	Retrieve asset metadata
`set_metadata(asset_id, **fields)`	`AssetMetadata`	Update metadata fields, including `record_type`
`create_relationship(from_asset_id, to_asset_id, rel_type=None, *, relationship_type=None, ...)`	`Relationship`	Create a directed provenance link between two assets

For headless notebooks, pass record_type to upload() or set_metadata(). Both operations use the authenticated transfer daemon, so integrations do not need to read the CLI credential store or expose access tokens to Python.

ML adapters

The dataerai.ml package reads Dataerai-hosted Zarr image stores directly from the short-lived /zarr-access/ bundle. PyTorch users can keep using ZarrAssetDataset / ZarrOriginalsDataset; Keras users get matching batch datasets:

from dataerai.ml import access
from dataerai.ml.keras_dataset import ZarrOriginalsSequence

creds = access.load_credentials()
train = ZarrOriginalsSequence(
    creds,
    asset_id="dataset-asset-id",
    size=64,
    batch_size=32,
    shuffle=True,
    label_mode="categorical",
)

model.fit(train, epochs=5)

For HLS4ML conversion, keep inputs in Keras' channels-last layout:

from dataerai.ml.hls4ml import convert_keras_model

hls_model = convert_keras_model(
    model,
    output_dir="hls-project",
    project_name="dataerai_model",
    data=train,              # writes input/output .npy testbench arrays
    testbench_batches=2,
)

`create_relationship()` arguments

Authors a directed provenance edge from_asset_id → to_asset_id. You need write access to the source and read access to the target. rel_type is a free-form verb describing the source's role, e.g. "analysis_of" or "acquired_with". Notebook integrations can pass the same value with the keyword-only alias relationship_type.

Argument	Type	Description
`from_asset_id`	`str`	Source (dependent) asset — the edge starts here (required)
`to_asset_id`	`str`	Target (origin) asset — the edge points here (required)
`rel_type`	`str \| None`	Free-form relationship type, ≤255 chars (required unless `relationship_type` is provided)
`relationship_type`	`str \| None`	Keyword-only alias for `rel_type`
`analysis_mode`	`str \| None`	`non_destructive`, `altering`, `destructive`, `in_situ`, `ex_situ`, `invasive`, `non_invasive`
`qualifier_note`	`str \| None`	Free-text note on the relationship
`qualifier_time`	`str \| None`	ISO-8601 timestamp
`qualifiers`	`dict \| None`	JSON-serializable extra qualifiers

# Link a processed result back to the raw data it came from.
client.create_relationship(analysis.asset_id, raw.asset_id, "analysis_of",
                           analysis_mode="non_destructive")

`upload()` keyword arguments

Argument	Type	Description
`title`	`str`	Asset title (required)
`owner_type`	`str`	`"project"` or `"user"` (required)
`owner_id`	`str`	Owner entity ID (required)
`description`	`str \| None`	Free-text description
`alias`	`str \| None`	Short identifier
`tags`	`list[str] \| None`	Tag list
`metadata`	`dict \| None`	Arbitrary key-value metadata
`collection_id`	`str \| None`	Collection to add the asset to
`chunk_size_mb`	`int \| None`	Override default 64 MiB chunk size
`on_progress`	`Callable[[ProgressEvent], None] \| None`	Progress callback
`transfer_timeout_s`	`float`	Max seconds to wait for completion (default 3600)

Progress events

@dataclass
class ProgressEvent:
    transfer_id: str
    bytes_done: int
    bytes_total: int
    chunk_index: int
    chunk_count: int
    rate_bps: float
    file_index: int
    file_name: str

    @property
    def percent(self) -> float: ...   # 0–100

    @property
    def rate_mbps(self) -> float: ...

Error types

Exception	When raised
`DaemonError(code, message)`	Daemon returned a coded error (see `code` attribute)
`DaemonTimeoutError`	Request or transfer exceeded the configured timeout
`ConnectionError`	Daemon disconnected unexpectedly

dev notebook bambu ml-io ml croissant ml-gpu-cu12 ml-gpu ml-keras ml-hls4ml ml-stream ml-stream-video ml-stream-train runner metaextract metaextract-converters metaextract-viz metaextract-checksums metaextract-utils metaextract-containers metaextract-microscopy metaextract-medical metaextract-bio metaextract-spectroscopy metaextract-all

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dataerai_sdk-0.2.0b15.tar.gz (598.3 kB view details)

Uploaded Jul 19, 2026 Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

dataerai_sdk-0.2.0b15-py3-none-any.whl (541.4 kB view details)

Uploaded Jul 19, 2026 Python 3

File details

Details for the file dataerai_sdk-0.2.0b15.tar.gz.

File metadata

Download URL: dataerai_sdk-0.2.0b15.tar.gz
Upload date: Jul 19, 2026
Size: 598.3 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: twine/6.2.0 CPython/3.11.2

File hashes

Hashes for dataerai_sdk-0.2.0b15.tar.gz
Algorithm	Hash digest
SHA256	`cf9dffe52a0643f65523343efaee4868e5e0974f863be90d0c4c2586aeb3f70f`
MD5	`86d092d44aee3e4670c53785d162f902`
BLAKE2b-256	`e2a4404704a96059ac83a537f8250c510d5e30495d38a7456a6fd66826f35eaa`

See more details on using hashes here.

File details

Details for the file dataerai_sdk-0.2.0b15-py3-none-any.whl.

File metadata

Download URL: dataerai_sdk-0.2.0b15-py3-none-any.whl
Upload date: Jul 19, 2026
Size: 541.4 kB
Tags: Python 3
Uploaded using Trusted Publishing? No
Uploaded via: twine/6.2.0 CPython/3.11.2

File hashes

Hashes for dataerai_sdk-0.2.0b15-py3-none-any.whl
Algorithm	Hash digest
SHA256	`db473a49d703e95c602b5bbefe313d60cccf359afad28056ef4c14126fe32baa`
MD5	`413d753e5aa1124f1e478a0391d76919`
BLAKE2b-256	`029c23b5c9446256add9ec44c7fe4ca6f409611d2199cc276d3105cffcc34a12`