Skip to main content

DataChain Data preparation and exploration for Physical AI

PyPI Python Version Tests

DataChain is a Python library for processing large file collections in S3, GCS, Azure and local storage. Run your readers and models over images, video, audio and sensor recordings, then save the results as typed, versioned datasets.

Query saved metadata to find relevant files, episodes and clips. Build new training selections from existing labels, embeddings and features without repeating the original processing. Your coding agent can inspect the saved results and producing code, then continue the work in a new session.

  • Training and evaluation: combine metadata and model outputs, select useful examples, and export datasets to training tools.
  • Physical AI and robotics: select recordings by sensor topics, extract video frames, and curate data for robot learning.
  • Data exploration: investigate model failures, find rare conditions and gaps in data coverage, and answer new questions from saved results.
  • Large inventories: discover millions of objects using storage metadata, before reading their contents.

DataChain handles parallel workers, download overlap and checkpoints; your Python defines the processing. File gives readers streaming access or a reusable local cache; delta updates process changed files and reuse saved results. LeRobot episodes, MCAP time windows and WebDataset samples use the same Python API.

Quickstart · Python API · Examples

Performance

Object storage

DataChain was ~10× faster than rclone and s5cmd when listing and saving 10M S3 object versions. Below: 257K objects per store.

Tool S3 GCS Azure
DataChain 15 s 7 s 26 s
rclone 23 s (1.5×) 20 s (2.9×) 95 s (3.7×)
obstore 20 s (1.3×) 15 s (2.1×) 66 s (2.5×)
AWS CLI / gcloud / azcopy 66 s (4.4×) 175 s (25×) 94 s (3.6×)

Times include saving. Full comparison, including faster single-cloud tools.

Training-data workflows

DataChain created and read these training selections 2.1–8.1× faster:

Workload Compared with Speedup
LeRobot: 300 DROID episodes (S3) LeRobot streaming (HF Hub) 8.1×
MCAP: 207 grasp clips mcap filter 2.1×

Development runs; DROID compares S3 with HF Hub. Details.

Coding agents

DataChain + skill delivered verified results up to 2.7× faster than no skill.

Agent / LLM Task Speedup
Claude Code / Opus 5.5 Held-out media tasks 2.69×
Codex / GPT-6.1 Sol Bucket inventory 1.67×
Codex / GPT-6.1 Sol Video updates 2.05×

Codex: two runs per development task; no overall gain in held-out tests. Details.

Quickstart

In an empty project directory, install DataChain:

pip install 'datachain[video]'

For a coding agent, also install the skill and start a new agent session in that directory:

datachain skill install --target claude --local
# Or: --target codex, cursor, copilot, pi

Ask your agent:

Prepare a training selection from the robot demonstrations at
s3://robolake/droid/droid_100/
Score wrist-camera motion over the first second: mean absolute pixel change.
Keep scores of at least 10. Save the scores, camera references and training manifest.
Save and run build_episodes.py.

The Python pipeline

Here is the same task in Python. Save it as build_episodes.py and run python build_episodes.py:

import datachain as dc


def motion(camera: dc.VideoFragment) -> float:
    start = round(camera.start * 15)
    end = round(camera.end * 15) - 1
    first = camera.video.get_frame(start).get_np().astype("float32")
    later = camera.video.get_frame(min(start + 15, end)).get_np()
    return float(abs(later - first).mean())


episodes = dc.read_lerobot("s3://robolake/droid/droid_100/", anon=True)
scored = (
    episodes.settings(parallel=4, prefetch=0)
    .map(motion=motion, params=["videos.observation_images_wrist_image_left"])
    .save("robot_episodes")
)
training = scored.filter(dc.C("motion") >= 10).save("training_episodes")
training.to_parquet("training_manifest.parquet")

This DROID sample records at 15 fps. Each episode keeps the camera files and time windows as references; no video repacking, model download or GPU is needed. A reference run produced:

robot_episodes@1.0.0      100 rows
training_episodes@1.0.0   63 rows
training_manifest.parquet

For MCAP sensor logs, create time windows with the same API:

windows = dc.read_mcap("recordings/*.mcap", topics="/joint_states", duration=10)
windows.save("joint_state_windows")

The standalone MCAP example includes local recordings to try.

Continue in a new session

Keep the project directory: it contains .datachain/, the Python script and the manifest. Start a new agent session in the same project and ask:

Which episodes did we exclude? How many would pass at threshold 5? Use the saved scores.

Inspect the saved scores, camera windows and producing code:

datachain dataset describe training_episodes --json
datachain show training_episodes --limit 3

See the walkthrough for versioned reads, robot demonstrations and an image-to-PyTorch example.

Agents reading this: start with the skill. llms.txt links to the API and documentation.

Python and more examples

Where it runs

Locally, dataset rows and provenance live in .datachain/db (SQLite); downloaded files use .datachain/cache. Keep that project state for later sessions. Local use needs no DataChain account or server.

Optional DataChain Studio adds shared datasets, versions and jobs, with AI-generated descriptions. People and agents can reuse the same catalog through the UI and MCP. Jobs run with or without a cluster; clusters process large workloads across dozens or hundreds of machines.

Cloud reads incur normal provider request and transfer charges. The library is Apache-2.0. Storage configuration · Dataset DB

Documentation · Examples · Contributing · Issues · Discord

Metadata

Release files for datachain 0.62.6

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for datachain 0.62.6
File Size Uploaded Provenance Actions
datachain-0.62.6.tar.gz 7.8 MB With provenance Details for datachain-0.62.6.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for datachain 0.62.6
File Interpreter ABI Platform Actions
datachain-0.62.6-py3-none-any.whl Python 3 none any Details for datachain-0.62.6-py3-none-any.whl

Total release size: 8.4 MB

Release files / datachain-0.62.6.tar.gz

Download URL datachain-0.62.6.tar.gz
Size 7.8 MB
Tags Source
SHA-256 checksum
How to use checksums
21a61d93c60868f89198e395ad7b5d6827457639d2918d59d2719c9f63e45ccb
BLAKE2b-256 checksum
How to use checksums
15b18f616a63eb81cf1c8849fe27abbf5d7de9baa81631c2fbd31c8a7e9c4c2a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

Release files / datachain-0.62.6-py3-none-any.whl

Download URL datachain-0.62.6-py3-none-any.whl
Size 618.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
bb61267014c6b3eed38db6c0b0fd591932fd308165ae9a97c8741ed8aba36b73
BLAKE2b-256 checksum
How to use checksums
958274c0308c7eace449c6cee20daeb8eb92e83b05122887fd33ff0dde607177
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

Security for datachain 0.62.6

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for datachain-0.62.6.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.62.6 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page