rust-data-processing
Python bindings for the rust-data-processing crate: schema-first ingestion from CSV, JSON, Parquet, and Excel into an in-memory DataSet, with profiling, validation, Polars-backed pipelines, SQL, Phase 2 JSONL export, privacy transforms and summaries, median, Arrow interop, incremental ingest helpers, and Phase 3 parity with JVM JSON FFI (agent-ready structured in/out for LangGraph and tool-calling workflows).
Infographic: Phase 3 — one Rust engine; Python (PyO3 / PyPI) and Java (Panama / Maven + Gradle) bindings; Phase 1–2 capabilities; agent-ready JSON in/out; shared connectors (Postgres, S3, Kafka, Snowflake).
This page is the PyPI project description (Python-only). Clone the repository for developer setup, Rust sources, and the full monorepo README.
Install
pip install rust-data-processing
Requires Python 3.10+.
Quick start
import rust_data_processing as rdp
schema = [
{"name": "id", "data_type": "int64"},
{"name": "name", "data_type": "utf8"},
]
ds = rdp.ingest_from_path("path/to/data.csv", schema, {"format": "csv"})
print("rows", ds.row_count())
report = rdp.profile_dataset(ds, {"head_rows": 50, "quantiles": [0.5]})
print("profile rows sampled", report["row_count"])
validation = rdp.validate_dataset(
ds,
{"checks": [{"kind": "not_null", "column": "id", "severity": "error"}]},
)
print("checks", validation["summary"]["total_checks"])
Phase 2 (export, privacy, JSONL, median, Delta handoff)
Copy-paste snippets: Phase 2 Python examples (Markdown in repo). These APIs are also summarized in API.md (section Export, privacy summaries, truncation (Phase 2)).
Documentation
| Link | |
|---|---|
| This package on PyPI | pypi.org/project/rust-data-processing |
| Python examples (HTML, pdoc) | GitHub Pages — examples |
| Python API (HTML, pdoc) | GitHub Pages — Python |
| Python API (markdown) | API.md in the repository |
| Combined site (landing + Rust rustdoc) | GitHub Pages — home |
| Rust crate API | docs.rs/rust-data-processing |
| JVM bindings (Java) | docs/java/README.md |
| Repository | github.com/scorpio-datalake/rust-data-processing |
License
MIT OR Apache-2.0 - see LICENSE-MIT and LICENSE-APACHE in the repository.
1 maintainer
Release files for rust-data-processing 0.3.7
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| rust_data_processing-0.3.7.tar.gz | 5.9 MB | Details |
Built distributions (wheels)
Total release size: 568.5 MB
Release files / rust_data_processing-0.3.7.tar.gz
| Download URL | rust_data_processing-0.3.7.tar.gz |
|---|---|
| Size | 5.9 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
19588c367df8fb3c8e749bc2df19bfaf0110976c0a862e2168ac6a6b21c0f716
|
|
BLAKE2b-256 checksum How to use checksums |
04a89ba9812b6fe7890196af28abad824fff59c6b5932187ddbbfd2a893f84fc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / rust_data_processing-0.3.7-pp311-pypy311_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | rust_data_processing-0.3.7-pp311-pypy311_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 34.2 MB |
| Tags | Linux glibc 2.17+ x86-64 PyPy 3.11 PyPy 3.11 7.3 |
|
SHA-256 checksum How to use checksums |
797b11dbcf6dafd0f466932b2dc1ea1788c1896249170cdea27deae5e36e5629
|
|
BLAKE2b-256 checksum How to use checksums |
65b86686286b01ac851cd9fdcc93ceccc914981c6d4ab2f51f2fc42d21d27c7c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / rust_data_processing-0.3.7-cp315-cp315t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | rust_data_processing-0.3.7-cp315-cp315t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 34.2 MB |
| Tags | CPython 3.15 CPython 3.15 free-threading Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
458e0dc7ad88e6c58fc38f86d8e477006c62a1db96da972eb8c98b6b62678205
|
|
BLAKE2b-256 checksum How to use checksums |
94048e803342a6392593195052f5d78166f77a2e7eced583bb089080e4b5a115
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / rust_data_processing-0.3.7-cp315-cp315-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | rust_data_processing-0.3.7-cp315-cp315-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 34.2 MB |
| Tags | CPython 3.15 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
c3060b801794efddff68efa50d72addf01f77aa26cdfa0fb71cbdd73ca1fa594
|
|
BLAKE2b-256 checksum How to use checksums |
73d6821c133de6dd5ec85420a9edf03842aa2e052b69186e3519ab8143bfdd4e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / rust_data_processing-0.3.7-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | rust_data_processing-0.3.7-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 34.2 MB |
| Tags | CPython 3.14 CPython 3.14 free-threading Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
566c8e9a124328096e7a10843cc0507b7c6f9f29a430705df7465ea887e57666
|
|
BLAKE2b-256 checksum How to use checksums |
eb0aa608777688d0cc12b1bf0cddaef14ddbc195280ca9c9bb288dd7656fcd0e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / rust_data_processing-0.3.7-cp314-cp314-win_amd64.whl
| Download URL | rust_data_processing-0.3.7-cp314-cp314-win_amd64.whl |
|---|---|
| Size | 28.5 MB |
| Tags | CPython 3.14 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
627a153bd783d52e6bb279042ff7d9b0d11c4960d0395e9752df99ed9f108ddb
|
|
BLAKE2b-256 checksum How to use checksums |
5b614fef7937f80aed7cd5e3c1d4d55991ba3111cf356bd6b6a51c35351e91ee
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / rust_data_processing-0.3.7-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | rust_data_processing-0.3.7-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 34.2 MB |
| Tags | CPython 3.14 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
1052ca8e4b02378080279d5c28e055cf822c543184b78d74abd64a5fdcfb3991
|
|
BLAKE2b-256 checksum How to use checksums |
8f92349b82f12233c92f7d4c193e2358e38e9f5d8a9074a16f5ec9f428e83094
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / rust_data_processing-0.3.7-cp314-cp314-macosx_11_0_arm64.whl
| Download URL | rust_data_processing-0.3.7-cp314-cp314-macosx_11_0_arm64.whl |
|---|---|
| Size | 28.0 MB |
| Tags | CPython 3.14 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
d403eccf5e93b8c80c91da137b302af730365cea14f6afa7c374e45cef9b89e2
|
|
BLAKE2b-256 checksum How to use checksums |
d9b3f38502877fce39ca7bbe596ee0c1e01a4cc0112ce860547660248aac9589
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / rust_data_processing-0.3.7-cp313-cp313-win_amd64.whl
| Download URL | rust_data_processing-0.3.7-cp313-cp313-win_amd64.whl |
|---|---|
| Size | 28.5 MB |
| Tags | CPython 3.13 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
05768370333732644ebc31adb469a31e81f8561b8d7b893cb81fc528254f4304
|
|
BLAKE2b-256 checksum How to use checksums |
0a42edd41a05817a3df59ff30ffa4e5558ca092e79df942e3f3f5b6a2afe9a39
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / rust_data_processing-0.3.7-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | rust_data_processing-0.3.7-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 34.2 MB |
| Tags | CPython 3.13 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
15817d6d821fd3284a2154c7aebbc4981e8cbe711d87b25f8338272eb56a08bc
|
|
BLAKE2b-256 checksum How to use checksums |
a0d2ad31b0dacf752c85ad98c8db1e0a3b3205c25cf540cfa53f2ebef2e832c2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / rust_data_processing-0.3.7-cp313-cp313-macosx_11_0_arm64.whl
| Download URL | rust_data_processing-0.3.7-cp313-cp313-macosx_11_0_arm64.whl |
|---|---|
| Size | 28.0 MB |
| Tags | CPython 3.13 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
ba473495fdfa0f5084b4bd8847336f0c14a5f87555360fc129493b38457331b8
|
|
BLAKE2b-256 checksum How to use checksums |
45b21b278e146a599a50853a60064569fbed7259a9d1685ce1e85a705b65501c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / rust_data_processing-0.3.7-cp312-cp312-win_amd64.whl
| Download URL | rust_data_processing-0.3.7-cp312-cp312-win_amd64.whl |
|---|---|
| Size | 28.5 MB |
| Tags | CPython 3.12 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
3d7c6a5a7094a872ccd95d9048f93f75875a6e76de70895a7352b6a7f5bc096a
|
|
BLAKE2b-256 checksum How to use checksums |
54795490ce05ac12986d19dec7b1c284c9f8fd58b2757426b99595ecf99afd46
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / rust_data_processing-0.3.7-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | rust_data_processing-0.3.7-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 34.2 MB |
| Tags | CPython 3.12 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
efb3b275848a1ae63b945fc4dc229168dcc0247606ba022e09a54d9a49fa3f80
|
|
BLAKE2b-256 checksum How to use checksums |
a472e49bca79b4160779377ca035c383f0e14fb8381d750bcb7d8c08d9e63f2d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / rust_data_processing-0.3.7-cp312-cp312-macosx_11_0_arm64.whl
| Download URL | rust_data_processing-0.3.7-cp312-cp312-macosx_11_0_arm64.whl |
|---|---|
| Size | 28.0 MB |
| Tags | CPython 3.12 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
7a631d2075a523f78a2853c7c15b966e050a06710bec84a11b266a11cb35f2e9
|
|
BLAKE2b-256 checksum How to use checksums |
57718159458ab893413dffc476953c755f4933a8766dc7d359752937fc4a4000
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / rust_data_processing-0.3.7-cp311-cp311-win_amd64.whl
| Download URL | rust_data_processing-0.3.7-cp311-cp311-win_amd64.whl |
|---|---|
| Size | 28.5 MB |
| Tags | CPython 3.11 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
242f234a87f2c217d2c0dfbbb2e2632f2832e7f3d7700ab77229dcd992d7903e
|
|
BLAKE2b-256 checksum How to use checksums |
89b1987be092e2d5e40185ab26641a0b3fff8272927672b5f41e777268770cc5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / rust_data_processing-0.3.7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | rust_data_processing-0.3.7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 34.2 MB |
| Tags | CPython 3.11 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
c7bb8abe52e7bde1131cd974a5945301d96d01add76302a74ed04bccdf75d4ea
|
|
BLAKE2b-256 checksum How to use checksums |
5faa3f3fcdf98bbfb569cde697707173c22169e2bb35f2b62f5a65840a7739b6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / rust_data_processing-0.3.7-cp311-cp311-macosx_11_0_arm64.whl
| Download URL | rust_data_processing-0.3.7-cp311-cp311-macosx_11_0_arm64.whl |
|---|---|
| Size | 28.0 MB |
| Tags | CPython 3.11 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
fa14b94c8c4638170e80a9d1d955f501ca5ce5563346911481fdbf330b92f6d6
|
|
BLAKE2b-256 checksum How to use checksums |
948bf566cdf0314b9f901320aba742312f72187bc1412000ef83640ae787614a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / rust_data_processing-0.3.7-cp310-cp310-win_amd64.whl
| Download URL | rust_data_processing-0.3.7-cp310-cp310-win_amd64.whl |
|---|---|
| Size | 28.5 MB |
| Tags | CPython 3.10 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
31155601764986329e3ea6d7465e302541b72004b9e55f77ee552b25aef750ce
|
|
BLAKE2b-256 checksum How to use checksums |
caa5fdc144663d4aff8270a0929e97dbd38dfe3aa27189b6725a6fefda435305
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / rust_data_processing-0.3.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | rust_data_processing-0.3.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 34.2 MB |
| Tags | CPython 3.10 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
786612bd8cc38e29c4e6b3369c05f66a6d18fcfa39316a9a97ffc667675b9b22
|
|
BLAKE2b-256 checksum How to use checksums |
d023c291e0ef605c8ca1752b8927f90ba6fb6b6d1058eaa6e98253620d2cb020
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
