Skip to main content

agora-etl-rs

License Python PyPI

agora-etl-rs is the optional Rust acceleration layer for agora-etl.

It speeds up proven hot paths in the Agora runtime while preserving the same pipeline semantics, recovery behavior, and public builder model. The goal is not to create a second runtime. The goal is to make the existing runtime cheaper to execute.

agora-etl 0.4.x discovers this package automatically when it is installed. Most teams should interact with it through agora-etl, not by wiring the Rust classes directly.

What Ships in 0.2.0

agora-etl-rs 0.2.0 provides capability metadata plus native implementations for selected runtime primitives:

  • record_buffer
  • metrics_accumulator
  • linear_batch_buffer
  • checkpoint_state
  • memory_tracer
  • sync_builtin_chain_executor
  • csv_arrow_writer
  • jsonl_arrow_writer
  • jsonl_arrow_reader

These capabilities are consumed by agora-etl 0.4.x through its acceleration facade and surfaced in explain output, run summaries, and agora doctor.

When to Install It

Install agora-etl-rs when a pipeline is CPU-sensitive enough that Python-side buffering, metrics, or batch bookkeeping show up in profiles, or when a run must fail fast if acceleration is unavailable.

Do not install it expecting different runtime semantics. agora-etl-rs should make hot paths faster, not change how a pipeline behaves.

Requirements

  • Python >= 3.11
  • agora-etl >= 0.4.0,<1
  • A supported CPython runtime

Installation

Install both packages explicitly:

pip install agora-etl agora-etl-rs

Or use the convenience extra from agora-etl:

pip install "agora-etl[rs]"

Verifying the Extension

Check that the native module is importable:

from agora_rs import RUST_AVAILABLE, capabilities, is_available, version

print(RUST_AVAILABLE)   # True when the extension loaded
print(is_available())   # True when native bindings are active
print(version())        # "0.2.0"
print(capabilities())   # tuple[str, ...]

When the extension is unavailable, agora_rs still exposes compatibility stubs so package metadata can be inspected safely. In that state:

  • RUST_AVAILABLE is False
  • is_available() returns False
  • capabilities() returns ()

Using It Through agora-etl

agora-etl-rs is selected by agora-etl through the acceleration boundary in agora 0.4.x.

Declarative configuration:

[performance]
acceleration = "auto"      # auto | off | required
profile = "balanced"       # balanced | throughput | low_latency

Code-level configuration:

from agora import DeliveryConfig, Pipeline

bound = Pipeline(source).build(
    sink,
    config=DeliveryConfig(
        acceleration_mode="required",
        performance_profile="throughput",
    ),
)

Interpretation:

  • auto uses compatible Rust paths when installed
  • off forces pure-Python execution for debugging or comparison
  • required fails before source or sink open if acceleration is unavailable

Useful places to inspect the chosen fast path:

explain = pipeline.explain()
print(explain.acceleration.to_dict())

summary = await pipeline.run()
print(summary.runtime.to_dict())

From the CLI:

agora doctor --config agora.toml
agora doctor --json --config agora.toml

Public Python Surface

Most users only need:

  • agora_rs.__version__
  • agora_rs.version()
  • agora_rs.capabilities()
  • agora_rs.is_available()
  • agora_rs.RUST_AVAILABLE

Advanced users and internal integration tests may also use the native classes directly:

  • RecordBuffer
  • MetricsAccumulator
  • LinearBatchBuffer
  • CheckpointState
  • InMemoryTracer
  • SyncBuiltinChainExecutor
  • CsvArrowWriter
  • JsonlArrowWriter
  • read_jsonl_record_batches(...)

That surface is lower-level than agora-etl and should be treated as an acceleration substrate, not a replacement for the main builder API.

What It Accelerates

Area Python path Rust path
Source prefetch buffering asyncio.Queue style coordination bounded record buffer with native synchronization
Runtime metrics counting Python-side per-record updates batched native accumulation
Linear write batching Python list and bookkeeping objects native batch buffer
Checkpoint cadence checks Python object bookkeeping native checkpoint state helper
In-memory tracing helpers Python span storage native span container
Built-in sync middleware chain Python dispatch loop native chain executor
CSV / JSONL Arrow file paths Python fallback path native writer / reader helpers

Not every run will activate every primitive. agora-etl decides that per lane, per writer shape, and per performance profile.

Building From Source

This package lives inside the Agora monorepo.

git clone https://github.com/thanhtham010891/agora-etl.git
cd agora-etl/packages/agora-rs

Create a virtual environment and install build tooling:

python3.11 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install "maturin>=1.0,<2" "pytest>=8.0" "pytest-asyncio>=0.24"

Build and install the extension into the active environment:

maturin develop --release

Build distributable artifacts:

maturin build --release --out dist
maturin sdist --out dist

Testing and Quality Checks

Run the Python test suite:

pytest tests/ -v

Run Rust formatting and linting:

cargo fmt --check
cargo clippy --all-targets --all-features -- -D warnings

cargo test is not the primary test entrypoint here because the package uses pyo3 abi3 bindings and the meaningful coverage lives in the Python-driven integration tests after a maturin build.

Release Notes

The release workflow expects the GitHub release tag to match the package version exactly:

v0.2.0

Artifacts published by CI include:

  • platform wheels
  • source distribution

License

Apache-2.0. See LICENSE and NOTICE.

Metadata

Release files for agora-etl-rs 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for agora-etl-rs 0.2.0
File Size Uploaded Provenance Actions
agora_etl_rs-0.2.0.tar.gz 37.5 kB With provenance Details for agora_etl_rs-0.2.0.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for agora-etl-rs 0.2.0
File Actions
agora_etl_rs-0.2.0-cp311-abi3-win_amd64.whl CPython 3.11 abi3 Windows x86-64 Details for agora_etl_rs-0.2.0-cp311-abi3-win_amd64.whl
agora_etl_rs-0.2.0-cp311-abi3-musllinux_1_2_x86_64.whl CPython 3.11 abi3 Linux musl 1.2+ x86-64 Details for agora_etl_rs-0.2.0-cp311-abi3-musllinux_1_2_x86_64.whl
agora_etl_rs-0.2.0-cp311-abi3-manylinux_2_28_x86_64.whl CPython 3.11 abi3 Linux glibc 2.28+ x86-64 Details for agora_etl_rs-0.2.0-cp311-abi3-manylinux_2_28_x86_64.whl
agora_etl_rs-0.2.0-cp311-abi3-manylinux_2_28_aarch64.whl CPython 3.11 abi3 Linux glibc 2.28+ ARM64 Details for agora_etl_rs-0.2.0-cp311-abi3-manylinux_2_28_aarch64.whl
agora_etl_rs-0.2.0-cp311-abi3-macosx_11_0_arm64.whl CPython 3.11 abi3 macOS 11.0+ ARM64 Details for agora_etl_rs-0.2.0-cp311-abi3-macosx_11_0_arm64.whl

Total release size: 4.7 MB

Release files / agora_etl_rs-0.2.0.tar.gz

Download URL agora_etl_rs-0.2.0.tar.gz
Size 37.5 kB
Tags Source
SHA-256 checksum
How to use checksums
56500e8418a792a4b4e112ed6feee95cdd57a79ffee751a5d7e901873e4bfd8a
BLAKE2b-256 checksum
How to use checksums
c34ef3359efb2052e43b6b770d7fbdf02a5c6268cac8df06133be47e109e7181
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.12

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jun 16, 2026.

Transparency log

Release files / agora_etl_rs-0.2.0-cp311-abi3-win_amd64.whl

Download URL agora_etl_rs-0.2.0-cp311-abi3-win_amd64.whl
Size 825.5 kB
Tags CPython 3.11 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
4f4331ff5f1420f196b5733f9511a6e81fdf2cea347777a62ed6cd03d841b116
BLAKE2b-256 checksum
How to use checksums
0e340c459a26016724bd1eb2bf5a204ad0cc3bf32310f038baf769d9108363ca
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.12

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jun 16, 2026.

Transparency log

Release files / agora_etl_rs-0.2.0-cp311-abi3-musllinux_1_2_x86_64.whl

Download URL agora_etl_rs-0.2.0-cp311-abi3-musllinux_1_2_x86_64.whl
Size 1.2 MB
Tags CPython 3.11 Linux musl 1.2+ x86-64 abi3
SHA-256 checksum
How to use checksums
3a5327b662fb210d4853ef2a7cfa7405ce83d58eaaffaabb9a985bbb4d155b7a
BLAKE2b-256 checksum
How to use checksums
250eb31ff401ad42178190c8cf2ed27eff84a49af4ac7a66e1ac0dba1b90a3f0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.12

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jun 16, 2026.

Transparency log

Release files / agora_etl_rs-0.2.0-cp311-abi3-manylinux_2_28_x86_64.whl

Download URL agora_etl_rs-0.2.0-cp311-abi3-manylinux_2_28_x86_64.whl
Size 952.5 kB
Tags CPython 3.11 Linux glibc 2.28+ x86-64 abi3
SHA-256 checksum
How to use checksums
5cf570791b0753ebe8a4634d378be3ccea57558957249be2cf5f5e83264296b9
BLAKE2b-256 checksum
How to use checksums
f1bf08deb681f8e653991a091ea953ed17e1c7ed12df925cf2f8758a0e0eaf9f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.12

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jun 16, 2026.

Transparency log

Release files / agora_etl_rs-0.2.0-cp311-abi3-manylinux_2_28_aarch64.whl

Download URL agora_etl_rs-0.2.0-cp311-abi3-manylinux_2_28_aarch64.whl
Size 909.6 kB
Tags CPython 3.11 Linux glibc 2.28+ ARM64 abi3
SHA-256 checksum
How to use checksums
f68420b6643eebe4a99f821d7f6ed52ec8d37687bad7b068b20c0b5239254e78
BLAKE2b-256 checksum
How to use checksums
70fa7a7c43deb24f8fb58becba5ec46d018e0c7bf7c096def2011cdfad66cdca
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.12

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jun 16, 2026.

Transparency log

Release files / agora_etl_rs-0.2.0-cp311-abi3-macosx_11_0_arm64.whl

Download URL agora_etl_rs-0.2.0-cp311-abi3-macosx_11_0_arm64.whl
Size 854.7 kB
Tags CPython 3.11 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
b5d294711b89cc364166bb9f35ac96dd0db14cee33a0300c2a5ce2a144342447
BLAKE2b-256 checksum
How to use checksums
c157441b74de191100efb44474cc0ac23802e536b7ac5aef7c04b54d417bdbf5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.12

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jun 16, 2026.

Transparency log

Security for agora-etl-rs 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

All 6 files in this release have been uploaded with provenance metadata.

Select from 6 files

Attestation for agora_etl_rs-0.2.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jun 16, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.0 This release

6 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page