Pre-release

This release is a pre-release and may not be stable for production use.

Zephyr

Simple data processing library for Marin pipelines. Build lazy dataset pipelines that run on Iris jobs or a local backend.

Quick Start

from zephyr import Dataset, ZephyrContext, load_jsonl

# Read, transform, write
ctx = ZephyrContext(max_workers=100)
pipeline = (
    Dataset.from_files("gs://input/", "**/*.jsonl.gz")
    .flat_map(load_jsonl)
    .filter(lambda x: x["score"] > 0.5)
    .map(lambda x: transform_record(x))
    .write_jsonl("gs://output/data-{shard:05d}-of-{total:05d}.jsonl.gz")
)
ctx.execute(pipeline)

Key Patterns

Dataset Creation:

Dataset.from_files(path, pattern) - glob files
Dataset.from_list(items) - explicit list

Loading Files

.load_{file,parquet,jsonl,vortex} - load rows from a file

Transformations:

.map(fn) - transform each item
.flat_map(fn) - expand items (e.g., load_jsonl)
.filter(fn) - filter items by function or expression
.select(columna, columnb) - select out the given columns
.window(n) - group into batches
.reshard(n) - redistribute across n shards

Output:

.write_jsonl(pattern) - write JSONL (gzip if .gz)
.write_parquet(pattern, schema) - write to a Parquet file
.write_vortex(pattern) - write to a Vortex file

Execution (ZephyrContext):

ZephyrContext(max_workers=N) — auto-detects the backend (Iris inside an Iris job, local otherwise) via fray.current_client()
ZephyrContext(client=LocalClient()) — explicit local backend (testing)
ctx.execute(pipeline) — runs the pipeline; returns a ZephyrExecutionResult(results, counters)

Real Usage

Wikipedia Processing:

from zephyr import Dataset, ZephyrContext, load_jsonl

ctx = ZephyrContext(max_workers=100)
pipeline = (
    Dataset.from_list(files)
    .load_jsonl()
    .map(lambda row: process_record(row, config))
    .filter(lambda x: x is not None)
    .write_jsonl(f"{output}/data-{{shard:05d}}-of-{{total:05d}}.jsonl.gz")
)
ctx.execute(pipeline)

Dataset Sampling:

from zephyr import Dataset, ZephyrContext

ctx = ZephyrContext(max_workers=1000)
pipeline = (
    Dataset.from_files(input_path, "**/*.jsonl.gz")
    .map(lambda path: sample_file(path, weights))
    .write_jsonl(f"{output}/sampled-{{shard:05d}}.jsonl.gz")
)
ctx.execute(pipeline)

Parallel Downloads:

from zephyr import Dataset, ZephyrContext

tasks = [(config, fs, src, dst) for src, dst in file_pairs]
ctx = ZephyrContext(max_workers=32)
pipeline = Dataset.from_list(tasks).map(lambda t: download(*t))
ctx.execute(pipeline)

Installation

# From Marin monorepo
uv sync

# Standalone
cd lib/zephyr
uv pip install -e .

Running Tests

Zephyr tests run against multiple execution backends to ensure correctness across different environments.

All Tests on Both Backends (Default)

uv run pytest lib/zephyr/tests
# Runs all tests on both Local and Iris backends
# Local Iris cluster is started automatically via ClusterManager

Run Specific Backend Only

uv run pytest lib/zephyr/tests -k "local"
uv run pytest lib/zephyr/tests -k "iris"

The Iris cluster is started once per test session and reused across all tests for efficiency.

Design

Zephyr consolidates ad-hoc distributed and Hugging Face dataset processing patterns in Marin into a simple abstraction.

Key Features:

Lazy evaluation with operation fusion
Disk-based inter-stage data flow for low memory footprint
Chunk-by-chunk streaming to minimize memory pressure
Distributed execution with bounded parallelism (Iris/local backends)
Automatic chunking to prevent large object overhead
fsspec integration (GCS, S3, local)
Type-safe operation chaining

See AGENTS.md for execution internals and source layout.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

marin_zephyr-0.2.59.dev202607260759.tar.gz (87.7 kB view details)

Uploaded Jul 26, 2026 Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

marin_zephyr-0.2.59.dev202607260759-py3-none-any.whl (97.3 kB view details)

Uploaded Jul 26, 2026 Python 3

File details

Details for the file marin_zephyr-0.2.59.dev202607260759.tar.gz.

File metadata

Download URL: marin_zephyr-0.2.59.dev202607260759.tar.gz
Upload date: Jul 26, 2026
Size: 87.7 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.13.13

File hashes

Hashes for marin_zephyr-0.2.59.dev202607260759.tar.gz
Algorithm	Hash digest
SHA256	`1195373bdbb57befe22b24916505d3ea22da678854d22f791462880e3b7ce8bc`
MD5	`67f833ae07150a30db238994fedcdb29`
BLAKE2b-256	`588183dc7a09260c7c5af6972ac2ee544bae395e0fc9acb7b971becd995ca845`

See more details on using hashes here.

Provenance

The following attestation bundles were made for marin_zephyr-0.2.59.dev202607260759.tar.gz:

Publisher: marin-release-libs-wheels.yaml on marin-community/marin

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: marin_zephyr-0.2.59.dev202607260759.tar.gz
- Subject digest: 1195373bdbb57befe22b24916505d3ea22da678854d22f791462880e3b7ce8bc
- Sigstore transparency entry: 2255705139
- Sigstore integration time: Jul 26, 2026
Source repository:
- Permalink: marin-community/marin@3f8d53f85fe16af2134f135c7e3cb40a7fea204f
- Branch / Tag: refs/heads/main
- Owner: https://github.com/marin-community
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: marin-release-libs-wheels.yaml@3f8d53f85fe16af2134f135c7e3cb40a7fea204f
- Trigger Event: schedule