rust-data-processing 0.3.6

Python bindings for rust-data-processing: schema-first CSV/JSON/Parquet/Excel ingestion into an in-memory DataSet.

rust-data-processing

Python bindings for the rust-data-processing crate: schema-first ingestion from CSV, JSON, Parquet, and Excel into an in-memory DataSet, with profiling, validation, Polars-backed pipelines, SQL, Phase 2 JSONL export, privacy transforms and summaries, median, Arrow interop, incremental ingest helpers, and Phase 3 parity with JVM JSON FFI (agent-ready structured in/out for LangGraph and tool-calling workflows).

Infographic: Phase 3 — one Rust engine; Python (PyO3 / PyPI) and Java (Panama / Maven + Gradle) bindings; Phase 1–2 capabilities; agent-ready JSON in/out; shared connectors (Postgres, S3, Kafka, Snowflake).

This page is the PyPI project description (Python-only). Clone the repository for developer setup, Rust sources, and the full monorepo README.

Install

pip install rust-data-processing

Requires Python 3.10+.

Quick start

import rust_data_processing as rdp

schema = [
    {"name": "id", "data_type": "int64"},
    {"name": "name", "data_type": "utf8"},
]
ds = rdp.ingest_from_path("path/to/data.csv", schema, {"format": "csv"})
print("rows", ds.row_count())

report = rdp.profile_dataset(ds, {"head_rows": 50, "quantiles": [0.5]})
print("profile rows sampled", report["row_count"])

validation = rdp.validate_dataset(
    ds,
    {"checks": [{"kind": "not_null", "column": "id", "severity": "error"}]},
)
print("checks", validation["summary"]["total_checks"])

Phase 2 (export, privacy, JSONL, median, Delta handoff)

Copy-paste snippets: Phase 2 Python examples (Markdown in repo). These APIs are also summarized in API.md (section Export, privacy summaries, truncation (Phase 2)).

Documentation

	Link
This package on PyPI	pypi.org/project/rust-data-processing
Python examples (HTML, pdoc)	GitHub Pages — examples
Python API (HTML, pdoc)	GitHub Pages — Python
Python API (markdown)	API.md in the repository
Combined site (landing + Rust rustdoc)	GitHub Pages — home
Rust crate API	docs.rs/rust-data-processing
JVM bindings (Java)	docs/java/README.md
Repository	github.com/scorpio-datalake/rust-data-processing

License

MIT OR Apache-2.0 - see LICENSE-MIT and LICENSE-APACHE in the repository.