Signal Dataset

Immutable, indexed storage for multidimensional signal records on local filesystems, GCS, and S3. Numerical fields use SafeTensors; ArrayRecord provides random access.

Status: experimental 0.x software. The Python API may make documented breaking changes between minor releases. Persisted-format compatibility is versioned separately.

import numpy as np
import signal_dataset as sds

record = sds.Record(
    id="capture-0042",
    fields={
        "iq": sds.Field(
            np.zeros((4, 4096), dtype=np.complex64),
            axes=(sds.Axis("channel", 4), sds.Axis("time", 4096)),
        )
    },
    metadata={"sample_rate_hz": 20_000_000},
)

shard = sds.write_shard([record], "captures.sds", work_id="worker-000")
dataset = sds.publish(
    "captures.sds",
    [shard],
    dataset_id="captures",
    snapshot_id="run-001",
)

assert dataset[0].id == "capture-0042"
assert dataset.record_metadata[0]["iq"].shape == (4, 4096)

for descriptor in dataset.iter_record_metadata():
    print(descriptor.id)

for full_record in dataset.iter_records():
    assert full_record["iq"].data.shape == (4, 4096)

dataset.record_metadata[index] reads one aligned metadata entry without fetching signal tensors. dataset.iter_record_metadata() streams all metadata in logical order with bounded shard-store requests and keeps memory bounded to StorageOptions.records_per_read_batch records. The shipped ArrayRecord store serves each request with one underlying reader lifetime; custom stores control their own read_many() implementation. dataset.iter_records() provides the same ordered, shard-batched traversal for full records, including tensor payloads and normal record validation. Readers follow generation-pinned manifests and never list directories or GCS prefixes.

See the quickstart, annotation example, and distributed-writing guide.

Install

uv add signal-dataset
uv add 'signal-dataset[gcs]'  # GCS transport
uv add 'signal-dataset[s3]'   # S3 transport

Equivalent pip commands are pip install signal-dataset, pip install 'signal-dataset[gcs]', and pip install 'signal-dataset[s3]'.

Python 3.11–3.13 on Linux and macOS are supported. Windows is not currently supported because the local atomic-publication implementation uses POSIX filesystem primitives.

Design

• Immutable, root-last publication: readers observe no dataset or a complete snapshot.
• Lazy ordinal random access without directory or bucket-prefix discovery.
• Tensor-free aligned metadata reads.
• Storage and shard-container extension contracts.
• Profile-free records: no modality, training framework, or split policy is embedded in the core.

The project does not provide mutable datasets, distributed scheduling, authorization, retention, or batching policy. See the architecture, format compatibility contract, and roadmap.

Signal Dataset is use-case agnostic. It has no training-framework, modality, or split-policy dependency.

Community

Read CONTRIBUTING.md before proposing changes. Use GitHub Issues for reproducible bugs and design discussion, and follow SECURITY.md for private vulnerability reports. Participation is governed by the Code of Conduct.