Macrodata

Macrodata Refiner

Refiner is Macrodata's open-source Python library for reading, transforming, and writing robotics datasets.

It provides one pipeline model for working with robot episodes, frames, videos, metadata, and model-based processing. Use it to convert formats, transform data, run inference, and write structured outputs on your own infrastructure.

This repository also includes open-source reference versions of some of the pipelines we develop at Macrodata. They are useful starting points, but they are not the full pipelines we adapt, evaluate, and run for customers. If you want to see what those pipelines can do with your data, send us a representative sample.

Quickstart

Install:

pip install macrodata-refiner

This gives you:

the Python package as refiner
the CLI as macrodata

Launch a local pipeline:

import refiner as mdr

def add_preview(row):
    return row.update(
        preview=" ".join(row["text"].split()[:20]),
    )

(
    mdr.read_jsonl("input/*.jsonl")
    .filter(mdr.col("lang") == "en")
    .with_columns(
        text=mdr.col("text").str.strip(),
        text_len=mdr.col("text").str.len(),
    )
    .map(add_preview)
    .write_parquet("s3://my-bucket/english-cleanup/")
    .launch_local(
        name="english-cleanup",
        num_workers=2,
    )
)

Batteries included

a consistent row and episode model for robot trajectories, frames, videos, metadata, tasks, and statistics
readers and writers for LeRobot, HDF5, Zarr, MCAP, Parquet, JSONL, and other common data formats
composable transforms and model inference for converting and enriching data
open-source reference operations for motion trimming, subtask annotation, reward scoring, and hand tracking
access to storage backends supported by fsspec, including S3, GCP, and Hugging Face
in-process debugging and local multi-worker execution

Docs

Start here:

Build a dataset:

Community

join the Macrodata Discord: https://discord.gg/S8kZtmBR2x

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

macrodata_refiner-0.3.8.tar.gz (335.5 kB view details)

Uploaded Sep 7, 2026 Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

macrodata_refiner-0.3.8-py3-none-any.whl (413.6 kB view details)

Uploaded Sep 7, 2026 Python 3

File details

Details for the file macrodata_refiner-0.3.8.tar.gz.

File metadata

Download URL: macrodata_refiner-0.3.8.tar.gz
Upload date: Sep 7, 2026
Size: 335.5 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for macrodata_refiner-0.3.8.tar.gz
Algorithm	Hash digest
SHA256	`3d3c8234ba69d92be884e6b523686ca8a4f836749f6932030a602a62a5eb886c`
MD5	`eeafbbbc67d54eecd3d1fe7aacd866cc`
BLAKE2b-256	`e932abfbd963e6b8cfc2a556c16f119fdfd2cd922774e6be78ad64201f98ade5`

See more details on using hashes here.

Provenance

The following attestation bundles were made for macrodata_refiner-0.3.8.tar.gz:

Publisher: pypi-release.yml on macrodata-labs/refiner

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: macrodata_refiner-0.3.8.tar.gz
- Subject digest: 3d3c8234ba69d92be884e6b523686ca8a4f836749f6932030a602a62a5eb886c
- Sigstore transparency entry: 2751656650
- Sigstore integration time: Sep 7, 2026
Source repository:
- Permalink: macrodata-labs/refiner@532971236e969e36aab80c9cf902c10deb4bf117
- Branch / Tag: refs/heads/main
- Owner: https://github.com/macrodata-labs
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: pypi-release.yml@532971236e969e36aab80c9cf902c10deb4bf117
- Trigger Event: push