milkwashr
Deterministic sentence-chunk assembly for the milkwash
text-laundering pipeline. Rust core, importable from Python as milkwashr.
Sentence segmentation happens upstream in Python (sentencex, which has
explicit Arabic and Polish support). This crate does chunk assembly: group
2–8 sentences per chunk with a seeded RNG, then apply
text-splitter as a size guard so an
over-long chunk cannot overflow a small model's context window.
The RNG is seeded per source text, so every model in a sweep replays the identical chunk plan — without that, no two models see the same input and the outputs are not comparable.
Python
import milkwashr
milkwashr.ping() # "milkwashr-ok"
milkwashr.plan_chunk_spans(42, seed=1234) # [(0, 5), (5, 9), ...]
milkwashr.assemble_chunks(sentences, seed=1234) # list[str]
milkwashr.split_by_size(text, max_chars=6000) # list[str]
Rust
let spans = milkwashr::plan_chunk_spans(42, 1234, 2, 8);
License
Apache-2.0
1 maintainer
Release files for milkwashr 1.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| milkwashr-1.0.0.tar.gz | 7.6 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| milkwashr-1.0.0-cp39-abi3-macosx_11_0_arm64.whl | CPython 3.9 | abi3 | macOS 11.0+ ARM64 | Details |
Total release size: 2.8 MB
Release files / milkwashr-1.0.0.tar.gz
| Download URL | milkwashr-1.0.0.tar.gz |
|---|---|
| Size | 7.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
9075ae7b945d42ea0ad614b65bac1b4fe37ee718e77fa8143e45a192303b3e80
|
|
BLAKE2b-256 checksum How to use checksums |
6b44a358f29b7c96f08e2b6c372ca61ed810187d3c5267368c1f17370b740198
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.3 {"installer":{"name":"uv","version":"0.12.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / milkwashr-1.0.0-cp39-abi3-macosx_11_0_arm64.whl
| Download URL | milkwashr-1.0.0-cp39-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 2.8 MB |
| Tags | CPython 3.9 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
c66936dee80d23bb35c45cba4600c4607290502286187c04faf223f42ecb0691
|
|
BLAKE2b-256 checksum How to use checksums |
193d4cc4b008aa46c906b7af1a6c768b47b00aeb249cdfb21ded4094635af41a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.3 {"installer":{"name":"uv","version":"0.12.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
