Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

Dataerai Python SDK

Python client for the Dataerai transfer daemon.
Supports authenticated uploads, downloads, and metadata operations. The daemon is started automatically if it is not already running.

Requirements

  • Python ≥ 3.10
  • The dataerai binary must be installed and on PATH (or pass binary_path explicitly)
  • The user must be logged in via dataerai auth login before calling auth_status() / upload() / download()

Installation

pip install --pre "dataerai-sdk>=0.2.0b1,<0.3"
# or, from source:
pip install -e sdk/python/

Optional ML adapters are installed separately so the base SDK stays small:

pip install --pre "dataerai-sdk[ml]>=0.2.0b1,<0.3"          # PyTorch Zarr datasets
pip install --pre "dataerai-sdk[ml-keras]>=0.2.0b1,<0.3"    # Keras batches
pip install --pre "dataerai-sdk[ml-hls4ml]>=0.2.0b1,<0.3"   # HLS4ML conversion

Quick start

from dataerai import DataeraiClient

with DataeraiClient(binary_path="/usr/local/bin/dataerai") as client:
    # Check auth
    status = client.auth_status()
    print(f"Logged in as {status.user_email} ({status.user_id}), token expires {status.expires_at}")
    if status.user_id is None:
        raise RuntimeError("Upgrade the dataerai CLI to use user-owned uploads")

    # Upload a user-owned file. owner_id is a UUID, not an email address.
    result = client.upload(
        "/path/to/data.csv",
        title="My dataset",
        owner_type="user",
        owner_id=status.user_id,
        record_type="dataset",
        on_progress=lambda p: print(f"  {p.percent:.0f}%  {p.rate_mbps:.1f} MB/s"),
    )
    print(f"Uploaded  asset_id={result.asset_id}  content_id={result.content_id}")

    # Download it back
    dl = client.download(result.asset_id, dest_dir="/tmp/downloads")
    for f in dl.files:
        print(f"  {f.local_path}  ({f.size:,} bytes)")

    # Read / update metadata
    meta = client.get_metadata(result.asset_id)
    updated = client.set_metadata(
        result.asset_id,
        title="My dataset v2",
        record_type="analysis",
        tags=["csv", "demo"],
    )

    # Link a derived result to the source asset that produced it
    relationship = client.create_relationship(
        result.asset_id,
        "source-asset-id",
        relationship_type="derived_from",
        analysis_mode="non_destructive",
        qualifiers={"tool": "pycroscopy"},
    )
    related_id = relationship.related_asset["id"] if relationship.related_asset else "source-asset-id"
    print(f"Linked via {relationship.type} to {related_id}")

API reference

DataeraiClient(*, socket_path, binary_path, auto_start, start_timeout_s, request_timeout_s)

Parameter Default Description
socket_path $DATAERAI_SOCKET or /run/user/<uid>/dataerai-transfer.sock Unix socket path
binary_path None Path to the dataerai binary (required for auto_start)
auto_start True Spawn the daemon if the socket does not exist
start_timeout_s 10.0 Seconds to wait for the daemon socket to appear
request_timeout_s 30.0 Per-request timeout in seconds

Use as a context manager (with DataeraiClient(...) as client:) for automatic cleanup, or call client.connect() / client.close() manually.

Methods

Method Returns Description
auth_status() AuthStatus Logged-in user ID, email, and token expiry; user_id is None with older daemons
create_project(name, *, description="") Project Create a project (+ root collection, owner membership, default allocation); requires write scope
upload(local_path, *, title, owner_type, owner_id, record_type=None, ...) UploadResult Upload a file with an optional server-validated record type; blocks until complete
download(asset_id, dest_dir, ...) DownloadResult Download latest asset content; blocks until complete
get_metadata(asset_id) AssetMetadata Retrieve asset metadata
set_metadata(asset_id, **fields) AssetMetadata Update metadata fields, including record_type
create_relationship(from_asset_id, to_asset_id, rel_type=None, *, relationship_type=None, ...) Relationship Create a directed provenance link between two assets

For headless notebooks, pass record_type to upload() or set_metadata(). Both operations use the authenticated transfer daemon, so integrations do not need to read the CLI credential store or expose access tokens to Python.

ML adapters

The dataerai.ml package reads Dataerai-hosted Zarr image stores directly from the short-lived /zarr-access/ bundle. PyTorch users can keep using ZarrAssetDataset / ZarrOriginalsDataset; Keras users get matching batch datasets:

from dataerai.ml import access
from dataerai.ml.keras_dataset import ZarrOriginalsSequence

creds = access.load_credentials()
train = ZarrOriginalsSequence(
    creds,
    asset_id="dataset-asset-id",
    size=64,
    batch_size=32,
    shuffle=True,
    label_mode="categorical",
)

model.fit(train, epochs=5)

For HLS4ML conversion, keep inputs in Keras' channels-last layout:

from dataerai.ml.hls4ml import convert_keras_model

hls_model = convert_keras_model(
    model,
    output_dir="hls-project",
    project_name="dataerai_model",
    data=train,              # writes input/output .npy testbench arrays
    testbench_batches=2,
)

create_relationship() arguments

Authors a directed provenance edge from_asset_id → to_asset_id. You need write access to the source and read access to the target. rel_type is a free-form verb describing the source's role, e.g. "analysis_of" or "acquired_with". Notebook integrations can pass the same value with the keyword-only alias relationship_type.

Argument Type Description
from_asset_id str Source (dependent) asset — the edge starts here (required)
to_asset_id str Target (origin) asset — the edge points here (required)
rel_type str | None Free-form relationship type, ≤255 chars (required unless relationship_type is provided)
relationship_type str | None Keyword-only alias for rel_type
analysis_mode str | None non_destructive, altering, destructive, in_situ, ex_situ, invasive, non_invasive
qualifier_note str | None Free-text note on the relationship
qualifier_time str | None ISO-8601 timestamp
qualifiers dict | None JSON-serializable extra qualifiers
# Link a processed result back to the raw data it came from.
client.create_relationship(analysis.asset_id, raw.asset_id, "analysis_of",
                           analysis_mode="non_destructive")

upload() keyword arguments

Argument Type Description
title str Asset title (required)
owner_type str "project" or "user" (required)
owner_id str Owner entity ID (required)
description str | None Free-text description
alias str | None Short identifier
tags list[str] | None Tag list
metadata dict | None Arbitrary key-value metadata
collection_id str | None Collection to add the asset to
chunk_size_mb int | None Override default 64 MiB chunk size
on_progress Callable[[ProgressEvent], None] | None Progress callback
transfer_timeout_s float Max seconds to wait for completion (default 3600)

Progress events

@dataclass
class ProgressEvent:
    transfer_id: str
    bytes_done: int
    bytes_total: int
    chunk_index: int
    chunk_count: int
    rate_bps: float
    file_index: int
    file_name: str

    @property
    def percent(self) -> float: ...   # 0–100

    @property
    def rate_mbps(self) -> float: ...

Error types

Exception When raised
DaemonError(code, message) Daemon returned a coded error (see code attribute)
DaemonTimeoutError Request or transfer exceeded the configured timeout
ConnectionError Daemon disconnected unexpectedly

Metadata

Release files for dataerai-sdk 0.2.0.dev2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for dataerai-sdk 0.2.0.dev2
File Size Uploaded Provenance Actions
dataerai_sdk-0.2.0.dev2.tar.gz 569.1 kB Without provenance Details for dataerai_sdk-0.2.0.dev2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for dataerai-sdk 0.2.0.dev2
File Interpreter ABI Platform Actions
dataerai_sdk-0.2.0.dev2-py3-none-any.whl Python 3 none any Details for dataerai_sdk-0.2.0.dev2-py3-none-any.whl

Total release size: 1.1 MB

Release files / dataerai_sdk-0.2.0.dev2.tar.gz

Download URL dataerai_sdk-0.2.0.dev2.tar.gz
Size 569.1 kB
Tags Source
SHA-256 checksum
How to use checksums
53033ade177b1fb19c4d5713a55fe5a4e96259064038c80a484c227627578c10
BLAKE2b-256 checksum
How to use checksums
1d82fd634cb6eb4006983a1b51418e560a9958ea96a04bc4e5f577c0a36bca9f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.2

Release files / dataerai_sdk-0.2.0.dev2-py3-none-any.whl

Download URL dataerai_sdk-0.2.0.dev2-py3-none-any.whl
Size 521.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
7396150a65cb4267217de0d19a1be136fad45a273edf76d41bfc2b3210bce6d2
BLAKE2b-256 checksum
How to use checksums
a1f88505737f5af75f9867aefe96e166e75e00dd55aa8af80dc626e38159bcd8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.2

Security for dataerai-sdk 0.2.0.dev2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release
Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page