agora-etl-rs
agora-etl-rs is the optional Rust acceleration layer for
agora-etl.
It speeds up proven hot paths in the Agora runtime while preserving the same pipeline semantics, recovery behavior, and public builder model. The goal is not to create a second runtime. The goal is to make the existing runtime cheaper to execute.
agora-etl 0.4.x discovers this package automatically when it is installed.
Most teams should interact with it through agora-etl, not by wiring the Rust
classes directly.
What Ships in 0.2.0
agora-etl-rs 0.2.0 provides capability metadata plus native implementations
for selected runtime primitives:
record_buffermetrics_accumulatorlinear_batch_buffercheckpoint_statememory_tracersync_builtin_chain_executorcsv_arrow_writerjsonl_arrow_writerjsonl_arrow_reader
These capabilities are consumed by agora-etl 0.4.x through its acceleration
facade and surfaced in explain output, run summaries, and agora doctor.
When to Install It
Install agora-etl-rs when a pipeline is CPU-sensitive enough that Python-side
buffering, metrics, or batch bookkeeping show up in profiles, or when a run
must fail fast if acceleration is unavailable.
Do not install it expecting different runtime semantics. agora-etl-rs should
make hot paths faster, not change how a pipeline behaves.
Requirements
- Python
>= 3.11 agora-etl >= 0.4.0,<1- A supported CPython runtime
Installation
Install both packages explicitly:
pip install agora-etl agora-etl-rs
Or use the convenience extra from agora-etl:
pip install "agora-etl[rs]"
Verifying the Extension
Check that the native module is importable:
from agora_rs import RUST_AVAILABLE, capabilities, is_available, version
print(RUST_AVAILABLE) # True when the extension loaded
print(is_available()) # True when native bindings are active
print(version()) # "0.2.0"
print(capabilities()) # tuple[str, ...]
When the extension is unavailable, agora_rs still exposes compatibility
stubs so package metadata can be inspected safely. In that state:
RUST_AVAILABLEisFalseis_available()returnsFalsecapabilities()returns()
Using It Through agora-etl
agora-etl-rs is selected by agora-etl through the acceleration boundary in
agora 0.4.x.
Declarative configuration:
[performance]
acceleration = "auto" # auto | off | required
profile = "balanced" # balanced | throughput | low_latency
Code-level configuration:
from agora import DeliveryConfig, Pipeline
bound = Pipeline(source).build(
sink,
config=DeliveryConfig(
acceleration_mode="required",
performance_profile="throughput",
),
)
Interpretation:
autouses compatible Rust paths when installedoffforces pure-Python execution for debugging or comparisonrequiredfails before source or sink open if acceleration is unavailable
Useful places to inspect the chosen fast path:
explain = pipeline.explain()
print(explain.acceleration.to_dict())
summary = await pipeline.run()
print(summary.runtime.to_dict())
From the CLI:
agora doctor --config agora.toml
agora doctor --json --config agora.toml
Public Python Surface
Most users only need:
agora_rs.__version__agora_rs.version()agora_rs.capabilities()agora_rs.is_available()agora_rs.RUST_AVAILABLE
Advanced users and internal integration tests may also use the native classes directly:
RecordBufferMetricsAccumulatorLinearBatchBufferCheckpointStateInMemoryTracerSyncBuiltinChainExecutorCsvArrowWriterJsonlArrowWriterread_jsonl_record_batches(...)
That surface is lower-level than agora-etl and should be treated as an
acceleration substrate, not a replacement for the main builder API.
What It Accelerates
| Area | Python path | Rust path |
|---|---|---|
| Source prefetch buffering | asyncio.Queue style coordination |
bounded record buffer with native synchronization |
| Runtime metrics counting | Python-side per-record updates | batched native accumulation |
| Linear write batching | Python list and bookkeeping objects | native batch buffer |
| Checkpoint cadence checks | Python object bookkeeping | native checkpoint state helper |
| In-memory tracing helpers | Python span storage | native span container |
| Built-in sync middleware chain | Python dispatch loop | native chain executor |
| CSV / JSONL Arrow file paths | Python fallback path | native writer / reader helpers |
Not every run will activate every primitive. agora-etl decides that per lane,
per writer shape, and per performance profile.
Building From Source
This package lives inside the Agora monorepo.
git clone https://github.com/thanhtham010891/agora-etl.git
cd agora-etl/packages/agora-rs
Create a virtual environment and install build tooling:
python3.11 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install "maturin>=1.0,<2" "pytest>=8.0" "pytest-asyncio>=0.24"
Build and install the extension into the active environment:
maturin develop --release
Build distributable artifacts:
maturin build --release --out dist
maturin sdist --out dist
Testing and Quality Checks
Run the Python test suite:
pytest tests/ -v
Run Rust formatting and linting:
cargo fmt --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test is not the primary test entrypoint here because the package uses
pyo3 abi3 bindings and the meaningful coverage lives in the Python-driven
integration tests after a maturin build.
Release Notes
The release workflow expects the GitHub release tag to match the package version exactly:
v0.2.0
Artifacts published by CI include:
- platform wheels
- source distribution
License
Metadata
Release files for agora-etl-rs 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| agora_etl_rs-0.2.0.tar.gz | 37.5 kB | With provenance | Details for agora_etl_rs-0.2.0.tar.gz |
Built distributions (wheels)
| File | Actions Reset | |||
|---|---|---|---|---|
| agora_etl_rs-0.2.0-cp311-abi3-win_amd64.whl | CPython 3.11 | abi3 | Windows x86-64 | Details for agora_etl_rs-0.2.0-cp311-abi3-win_amd64.whl |
| agora_etl_rs-0.2.0-cp311-abi3-musllinux_1_2_x86_64.whl | CPython 3.11 | abi3 | Linux musl 1.2+ x86-64 | Details for agora_etl_rs-0.2.0-cp311-abi3-musllinux_1_2_x86_64.whl |
| agora_etl_rs-0.2.0-cp311-abi3-manylinux_2_28_x86_64.whl | CPython 3.11 | abi3 | Linux glibc 2.28+ x86-64 | Details for agora_etl_rs-0.2.0-cp311-abi3-manylinux_2_28_x86_64.whl |
| agora_etl_rs-0.2.0-cp311-abi3-manylinux_2_28_aarch64.whl | CPython 3.11 | abi3 | Linux glibc 2.28+ ARM64 | Details for agora_etl_rs-0.2.0-cp311-abi3-manylinux_2_28_aarch64.whl |
| agora_etl_rs-0.2.0-cp311-abi3-macosx_11_0_arm64.whl | CPython 3.11 | abi3 | macOS 11.0+ ARM64 | Details for agora_etl_rs-0.2.0-cp311-abi3-macosx_11_0_arm64.whl |
Total release size: 4.7 MB
Release files / agora_etl_rs-0.2.0.tar.gz
| Download URL | agora_etl_rs-0.2.0.tar.gz |
|---|---|
| Size | 37.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
56500e8418a792a4b4e112ed6feee95cdd57a79ffee751a5d7e901873e4bfd8a
|
|
BLAKE2b-256 checksum How to use checksums |
c34ef3359efb2052e43b6b770d7fbdf02a5c6268cac8df06133be47e109e7181
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jun 16, 2026.
Transparency logRelease files / agora_etl_rs-0.2.0-cp311-abi3-win_amd64.whl
| Download URL | agora_etl_rs-0.2.0-cp311-abi3-win_amd64.whl |
|---|---|
| Size | 825.5 kB |
| Tags | CPython 3.11 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
4f4331ff5f1420f196b5733f9511a6e81fdf2cea347777a62ed6cd03d841b116
|
|
BLAKE2b-256 checksum How to use checksums |
0e340c459a26016724bd1eb2bf5a204ad0cc3bf32310f038baf769d9108363ca
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jun 16, 2026.
Transparency logRelease files / agora_etl_rs-0.2.0-cp311-abi3-musllinux_1_2_x86_64.whl
| Download URL | agora_etl_rs-0.2.0-cp311-abi3-musllinux_1_2_x86_64.whl |
|---|---|
| Size | 1.2 MB |
| Tags | CPython 3.11 Linux musl 1.2+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
3a5327b662fb210d4853ef2a7cfa7405ce83d58eaaffaabb9a985bbb4d155b7a
|
|
BLAKE2b-256 checksum How to use checksums |
250eb31ff401ad42178190c8cf2ed27eff84a49af4ac7a66e1ac0dba1b90a3f0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jun 16, 2026.
Transparency logRelease files / agora_etl_rs-0.2.0-cp311-abi3-manylinux_2_28_x86_64.whl
| Download URL | agora_etl_rs-0.2.0-cp311-abi3-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 952.5 kB |
| Tags | CPython 3.11 Linux glibc 2.28+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
5cf570791b0753ebe8a4634d378be3ccea57558957249be2cf5f5e83264296b9
|
|
BLAKE2b-256 checksum How to use checksums |
f1bf08deb681f8e653991a091ea953ed17e1c7ed12df925cf2f8758a0e0eaf9f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jun 16, 2026.
Transparency logRelease files / agora_etl_rs-0.2.0-cp311-abi3-manylinux_2_28_aarch64.whl
| Download URL | agora_etl_rs-0.2.0-cp311-abi3-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 909.6 kB |
| Tags | CPython 3.11 Linux glibc 2.28+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
f68420b6643eebe4a99f821d7f6ed52ec8d37687bad7b068b20c0b5239254e78
|
|
BLAKE2b-256 checksum How to use checksums |
70fa7a7c43deb24f8fb58becba5ec46d018e0c7bf7c096def2011cdfad66cdca
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jun 16, 2026.
Transparency logRelease files / agora_etl_rs-0.2.0-cp311-abi3-macosx_11_0_arm64.whl
| Download URL | agora_etl_rs-0.2.0-cp311-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 854.7 kB |
| Tags | CPython 3.11 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
b5d294711b89cc364166bb9f35ac96dd0db14cee33a0300c2a5ce2a144342447
|
|
BLAKE2b-256 checksum How to use checksums |
c157441b74de191100efb44474cc0ac23802e536b7ac5aef7c04b54d417bdbf5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jun 16, 2026.
Transparency logSecurity for agora-etl-rs 0.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 6 files in this release have been uploaded with provenance metadata.
Attestation for agora_etl_rs-0.2.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jun 16, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.