milkwashr
Deterministic sentence-chunk assembly for the milkwash
text-laundering pipeline. Rust core, importable from Python as milkwashr.
Sentence segmentation happens upstream in Python (sentencex, which has
explicit Arabic and Polish support). This crate does chunk assembly: group
2–8 sentences per chunk with a seeded RNG, then apply
text-splitter as a size guard so an
over-long chunk cannot overflow a small model's context window.
The RNG is seeded per source text, so every model in a sweep replays the identical chunk plan — without that, no two models see the same input and the outputs are not comparable.
Python
import milkwashr
milkwashr.ping() # "milkwashr-ok"
milkwashr.plan_chunk_spans(42, seed=1234) # [(0, 5), (5, 9), ...]
milkwashr.assemble_chunks(sentences, seed=1234) # list[str]
milkwashr.split_by_size(text, max_chars=6000) # list[str]
Rust
let spans = milkwashr::plan_chunk_spans(42, 1234, 2, 8);
License
Apache-2.0
1 maintainer
Release files for milkwashr 1.0.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| milkwashr-1.0.1.tar.gz | 13.1 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| milkwashr-1.0.1-cp39-abi3-macosx_11_0_arm64.whl | CPython 3.9 | abi3 | macOS 11.0+ ARM64 | Details |
Total release size: 2.8 MB
Release files / milkwashr-1.0.1.tar.gz
| Download URL | milkwashr-1.0.1.tar.gz |
|---|---|
| Size | 13.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
65808d4ffc9ee11970fe056da1c22f8de262289bfae891150a6e9cc5e723d34b
|
|
BLAKE2b-256 checksum How to use checksums |
686d5a8b07fd975e0d08262028a250fe7be7190fdd5dc9589a17adb076f1a142
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.3 {"installer":{"name":"uv","version":"0.12.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / milkwashr-1.0.1-cp39-abi3-macosx_11_0_arm64.whl
| Download URL | milkwashr-1.0.1-cp39-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 2.8 MB |
| Tags | CPython 3.9 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
960c114c36f41659e86363ab6a6806816434dc3388e88753bf0241b1119a8559
|
|
BLAKE2b-256 checksum How to use checksums |
7c5fb2a70ced0156db063468870d4fcd1707b7d50e32b10948a0ada190a2a3a7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.3 {"installer":{"name":"uv","version":"0.12.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
