Data preparation and exploration for Physical AI
DataChain is a Python library for processing large file collections in S3, GCS, Azure and local storage. Run your readers and models over images, video, audio and sensor recordings, then save the results as typed, versioned datasets.
Query saved metadata to find relevant files, episodes and clips. Build new training selections from existing labels, embeddings and features without repeating the original processing. Your coding agent can inspect the saved results and producing code, then continue the work in a new session.
- Training and evaluation: combine metadata and model outputs, select useful examples, and export datasets to training tools.
- Physical AI and robotics: select recordings by sensor topics, extract video frames, and curate data for robot learning.
- Data exploration: investigate model failures, find rare conditions and gaps in data coverage, and answer new questions from saved results.
- Large inventories: discover millions of objects using storage metadata, before reading their contents.
DataChain handles parallel workers, download overlap and checkpoints; your Python defines the processing.
File gives readers streaming access or a reusable local cache; delta updates process changed files and reuse saved results.
LeRobot episodes, MCAP time windows and WebDataset samples use the same Python API.
Quickstart · Python API · Examples
Performance
Object storage
DataChain was ~10× faster than rclone and s5cmd when listing and saving 10M S3 object versions. Below: 257K objects per store.
| Tool | S3 | GCS | Azure |
|---|---|---|---|
| DataChain | 15 s | 7 s | 26 s |
| rclone | 23 s (1.5×) | 20 s (2.9×) | 95 s (3.7×) |
| obstore | 20 s (1.3×) | 15 s (2.1×) | 66 s (2.5×) |
| AWS CLI / gcloud / azcopy | 66 s (4.4×) | 175 s (25×) | 94 s (3.6×) |
Times include saving. Full comparison, including faster single-cloud tools.
Training-data workflows
DataChain created and read these training selections 2.1–8.1× faster:
| Workload | Compared with | Speedup |
|---|---|---|
| LeRobot: 300 DROID episodes (S3) | LeRobot streaming (HF Hub) | 8.1× |
| MCAP: 207 grasp clips | mcap filter |
2.1× |
Development runs; DROID compares S3 with HF Hub. Details.
Coding agents
DataChain + skill delivered verified results up to 2.7× faster than no skill.
| Agent / LLM | Task | Speedup |
|---|---|---|
| Claude Code / Opus 5.5 | Held-out media tasks | 2.69× |
| Codex / GPT-6.1 Sol | Bucket inventory | 1.67× |
| Codex / GPT-6.1 Sol | Video updates | 2.05× |
Codex: two runs per development task; no overall gain in held-out tests. Details.
Quickstart
In an empty project directory, install DataChain:
pip install 'datachain[video]'
For a coding agent, also install the skill and start a new agent session in that directory:
datachain skill install --target claude --local
# Or: --target codex, cursor, copilot, pi
Ask your agent:
Prepare a training selection from the robot demonstrations at
s3://robolake/droid/droid_100/
Score wrist-camera motion over the first second: mean absolute pixel change.
Keep scores of at least 10. Save the scores, camera references and training manifest.
Save and run build_episodes.py.
The Python pipeline
Here is the same task in Python. Save it as build_episodes.py and run python build_episodes.py:
import datachain as dc
def motion(camera: dc.VideoFragment) -> float:
start = round(camera.start * 15)
end = round(camera.end * 15) - 1
first = camera.video.get_frame(start).get_np().astype("float32")
later = camera.video.get_frame(min(start + 15, end)).get_np()
return float(abs(later - first).mean())
episodes = dc.read_lerobot("s3://robolake/droid/droid_100/", anon=True)
scored = (
episodes.settings(parallel=4, prefetch=0)
.map(motion=motion, params=["videos.observation_images_wrist_image_left"])
.save("robot_episodes")
)
training = scored.filter(dc.C("motion") >= 10).save("training_episodes")
training.to_parquet("training_manifest.parquet")
This DROID sample records at 15 fps. Each episode keeps the camera files and time windows as references; no video repacking, model download or GPU is needed. A reference run produced:
robot_episodes@1.0.0 100 rows
training_episodes@1.0.0 63 rows
training_manifest.parquet
For MCAP sensor logs, create time windows with the same API:
windows = dc.read_mcap("recordings/*.mcap", topics="/joint_states", duration=10)
windows.save("joint_state_windows")
The standalone MCAP example includes local recordings to try.
Continue in a new session
Keep the project directory: it contains .datachain/, the Python script and the manifest.
Start a new agent session in the same project and ask:
Which episodes did we exclude? How many would pass at threshold 5? Use the saved scores.
Inspect the saved scores, camera windows and producing code:
datachain dataset describe training_episodes --json
datachain show training_episodes --limit 3
See the walkthrough for versioned reads, robot demonstrations and an image-to-PyTorch example.
Agents reading this: start with the skill. llms.txt links to the API and documentation.
Python and more examples
- Robot learning: curate LeRobot episodes: process camera clips and reuse saved selections.
- WebDataset: revise a saved selection: filter samples inside TAR shards without repacking them.
- Video datasets: extract frames and timestamps: read video with
datachain[video]. - Training: load a saved selection into PyTorch: use a pinned version in a new process.
- Public image pipeline: typed
.map(), selection and manifest. - Multi-cloud inventory: aggregate S3, GCS and Azure listings.
- Updates and resume: refresh saved results and continue interrupted work.
Where it runs
Locally, dataset rows and provenance live in .datachain/db (SQLite); downloaded files use .datachain/cache.
Keep that project state for later sessions. Local use needs no DataChain account or server.
Optional DataChain Studio adds shared datasets, versions and jobs, with AI-generated descriptions. People and agents can reuse the same catalog through the UI and MCP. Jobs run with or without a cluster; clusters process large workloads across dozens or hundreds of machines.
Cloud reads incur normal provider request and transfer charges. The library is Apache-2.0. Storage configuration · Dataset DB
Documentation · Examples · Contributing · Issues · Discord
Metadata
Release files for datachain 0.62.7
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| datachain-0.62.7.tar.gz | 7.8 MB | With provenance | Details for datachain-0.62.7.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| datachain-0.62.7-py3-none-any.whl | Python 3 | none | any | Details for datachain-0.62.7-py3-none-any.whl |
Total release size: 8.4 MB
Release files / datachain-0.62.7.tar.gz
| Download URL | datachain-0.62.7.tar.gz |
|---|---|
| Size | 7.8 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b444f050bc267c21f5d8d146203cc53b441daa92a640b89a986588eb161c864b
|
|
BLAKE2b-256 checksum How to use checksums |
083ef139e07b0629b2e60fc62c0d2624c0df7ff4ff4b393e295ab172e36d0db1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 22, 2026.
Transparency logRelease files / datachain-0.62.7-py3-none-any.whl
| Download URL | datachain-0.62.7-py3-none-any.whl |
|---|---|
| Size | 631.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
bd0dfc8212a3ecc93b1df0dcfe7999cc86c084313a6811f2099b2477a03dbea2
|
|
BLAKE2b-256 checksum How to use checksums |
11a634761ad638781451715ab8bd74bb8f3e490560fb5bd1ae993596edad3a56
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 22, 2026.
Transparency logSecurity for datachain 0.62.7
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for datachain-0.62.7.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 22, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.