Premove ITN

Contextual inverse text normalization for English voice-agent transcripts.

Premove turns spoken-form ASR text into structured written text:

call me at four thirty  →  call me at 04:30
the total is twenty dollars  →  the total is $20

Deterministic Rust realizers propose valid written forms. A DeBERTa-v3-large contextual scorer uses the complete transcript to choose among them, and an exact decoder produces compatible, non-overlapping edits.

On the retained First Evaluation, Premove achieved 99.50% semantic accuracy on 400 dedicated voice-agent rows and 89.70% overall semantic accuracy on 1,500 frozen stress-suite rows. Mean warm request latency was 56.49 ms on an Apple M4 using MPS and batch size one. See the full First Evaluation report.

Installation

The package is prepared for:

pip install premove-itn

Install the exact v0.1.0 release from PyPI:

pip install premove-itn==0.1.0

The model weights are downloaded separately from the frozen premove-ai/premove-itn Hugging Face release on first use.

Python quickstart

from premove_itn import PremoveITN

itn = PremoveITN.from_pretrained()

print(itn.normalize("call me at four thirty"))
# call me at 04:30

Create one PremoveITN instance and keep it resident:

texts = [
    "call me at four thirty",
    "the total is twenty dollars",
    "the last account digits are zero eight two zero six three",
]

for text in texts:
    print(itn.normalize(text))

Model initialization is expensive. Warm normalization calls on an existing instance are much faster than loading a new instance for each request.

Command-line interface

Normalize one transcript:

premove-itn "call me at four thirty"

call me at 04:30

Process newline-delimited transcripts:

printf 'call me at four thirty\nthe total is twenty dollars\n' | premove-itn

call me at 04:30
the total is $20

Stdin mode loads the model once, then processes every input line in order. It is the correct CLI mode for files and long-lived transcript pipelines.

stdin process → one model load → line 1 → line 2 → line 3 → ...

The CLI supports --device auto, --device cpu, --device mps, --device cuda, --version, and --help. Normal stdout contains only normalized transcripts. Diagnostics and errors use stderr.

Why contextual ITN?

Obvious spoken values can often be normalized with fixed rules:

twenty dollars → $20

The harder cases have several plausible written forms. For example, a number sequence can represent a time, an identifier, a count, or part of a phone number. Premove generates valid alternatives with deterministic rules, then uses the surrounding sentence to score the intended interpretation.

ASR transcript
      ↓
structured Rust candidates
      ↓
full-sentence contextual scores
      ↓
exact interval decoding
      ↓
written transcript

How it works

Spoken ASR text
      │
      ▼
Rust candidate generation
      │  TIME / MONEY / PHONE / ID / ...
      ▼
DeBERTa-v3-large contextual scorer
      │
      ▼
Exact maximum-score decoder
      │
      ▼
Written transcript

Rust candidate generation deterministically proposes valid written representations. It does not choose the intended interpretation.

Contextual scoring uses the complete sentence to score competing candidates.

Exact decoding selects a compatible set of scored replacements without overlapping edits.

The public runtime has one implementation path. Both the Python API and CLI call PremoveITN; neither duplicates candidate generation or decoding.

Performance

Dedicated voice-agent rows

These results use only the 400 group=voice_agent rows: 50 rows in each of eight voice-agent domains.

Backend Correct entities Semantic accuracy Mean latency
Premove ITN 398/400 99.50% 57.41 ms
Thutmose 268/400 67.00% 16.04 ms
text-processing-rs 273/400 68.25% 0.15 ms

Overall frozen benchmark

Backend Semantic accuracy Strict exact Mean latency
Premove ITN 89.70% 40.53% 56.49 ms
Thutmose 59.39% 22.13% 15.98 ms
text-processing-rs 55.79% 16.53% 0.14 ms

Premove led semantic accuracy in this evaluation. It did not lead latency.

Methodology

  • The dataset is a frozen, balanced synthetic stress suite with 1,500 rows and 1,640 declared entities.
  • The voice-agent result uses only the 400 dedicated voice-agent rows. Generic multi-entity rows do not enter domain claims.
  • Semantic entity accuracy is the main structured-value metric. It ignores allowed display differences while preserving value, currency, unit, digit order, phone form, and identifier case policy.
  • Strict exact match scores the full canonical output and is reported separately.
  • Latency used sequential batch-one requests on an Apple M4 MacBook Air, MPS completion, an optimized Rust extension, and eight Rayon workers.
  • Models were initialized and warmed before request latency was measured.
  • The benchmark is not an IID sample of live production traffic.
  • The backend evaluation was blind: every backend received only transcript text and did not receive gold spans, categories, domains, difficulty, or expected output. Independent human gold adjudication is a separate task and remains pending.

Read the full report, detailed tables, and reproduction guide.

Model lifecycle and latency

Three different costs must not be combined:

Phase Meaning
First download Fetches about 1.6 GB from Hugging Face; duration depends on the network.
Cached initialization Resolves cached files, builds the model, loads weights, and transfers the model to the device; this took several seconds on the tested system.
Warm normalization Processes one transcript after loading and warm-up; the retained release-build mean was 56.49 ms on Apple M4/MPS.

The benchmark excludes model download and initialization. A one-shot CLI timing includes process startup and model initialization, so it is not comparable to warm request latency. Services and transcript streams should load one normalizer and reuse it.

Device selection

device="auto" selects CUDA when available, then Apple MPS, then CPU. Python users can pass device="cpu", device="mps", or device="cuda" to PremoveITN.from_pretrained(). The CLI exposes the same choices through --device.

Release wheels are validated on macOS 14+ arm64 and manylinux_2_28 x86_64 for Python 3.11–3.13. Real frozen-model inference is validated on Apple Silicon MPS and Linux CPU, with exact output equivalence across 1,500 inputs. See the platform support matrix and Stage 7 evidence. API availability does not imply support for an unlisted platform or device.

Limitations

  • English only.
  • A 435.6M-parameter DeBERTa-v3-large contextual scorer.
  • About a 1.6 GB first model download.
  • Multi-second model initialization.
  • A custom candidate-scoring architecture; it is not a generic AutoModel.from_pretrained() model.
  • A synthetic stress benchmark, not observed live-traffic accuracy.
  • Weaker measured categories include URL, MONEY, CARDINAL, TIME, REFERENCE_ID, and VERSION. See the report for exact per-category results.
  • Independent human gold adjudication and broader contamination checks remain incomplete.
  • CUDA, Windows, macOS Intel, Linux ARM64, and other accelerators are not validated v0.1.0 support claims.

Model and weights

The public inference artifact is premove-ai/premove-itn, release v0.1.0, at the immutable commit 80bda5e2e1fe9542aa628597090242df57c1a157. PremoveITN.from_pretrained() uses that commit by default and verifies the resolved revision, release metadata, base model, and model-file digest before inference.

The artifact is inference-only. It excludes optimizer state, scheduler state, training counters, training data, and evaluation rows. See the artifact release record and training provenance.

Training data and model selection

The production DeBERTa-v3-large scorer was trained on 418,000 examples across 15 training kinds. Training ran in three sequential stages:

Stage Examples Role
Google text normalization 378,000 General English ITN coverage
Conversational adaptation 20,000 Spoken-dialogue style and context
Structured-value adaptation 20,000 Identifiers, phones, electronic values, and hard context
Total 418,000

The frozen 1,500-row VoiceAgent benchmark was not used for training or model selection. The bulk training corpora were removed after this composition was recorded; the table below is the retained distribution of every example seen by the selected checkpoint. Counts are examples, not individual spans.

Kind Google Conversational Structured Total
CARDINAL 32,472 853 876 34,201
DATE 78,643 315 398 79,356
DECIMAL 12,033 465 701 13,199
DIGIT_SEQUENCE 3,258 398 674 4,330
ELECTRONIC 0 12 3,030 3,042
KEEP 100,000 11,429 4,150 115,579
MEASUREMENT 16,647 300 349 17,296
MONEY 23,688 258 271 24,217
MULTI 82,490 3,259 2,078 87,827
ORDINAL 21,484 258 271 22,013
PHONE 2,761 260 1,994 5,015
PUNCTUATION 624 258 272 1,154
TIME 3,894 1,879 876 6,649
WHITELIST 6 17 17 40
WORD 0 39 4,043 4,082
Total 378,000 20,000 20,000 418,000

The final 20,000-example structured stage contained candidate-bearing KEEP examples (4,000), conversational replay (2,850), Google replay (3,000), targeted electronic values (3,000), hard KEEP cases (150), identifiers (4,000), numeric IDs (750), and phone values (2,250). The source material combined the Google Text Normalization training partition with conversational examples from SLURP, Schema-Guided Dialogue, SpokenWOZ, and Taskmaster-1.

The Google stage used AdamW with learning rate 2e-5, weight decay 0.01, batch size 8, and length bucketing. Both adaptation stages used AdamW with learning rate 5e-6, weight decay 0.01, batch size 8, one epoch, and fresh optimizer state; the structured stage used microbatch size 2. The final structured-value checkpoint was selected for the best product-relevant balance of structured-value and conversational results. Full selection evidence and development results are in the production model record.

Reproducibility

The repository retains the frozen VoiceAgent ITN dataset, detailed per-record First Evaluation outputs, comparison adapters, metric summaries, and release artifact checks. The frozen benchmark was not used for training or checkpoint selection. Bulk training corpora were deleted after their composition and kind distribution were documented.

The benchmark runner is optional and is not part of normal inference. See benchmarks/README.md.

Repository

Path Purpose
src/premove_itn/ Python runtime and public API
rust/ Deterministic realization rules
benchmarks/ Optional comparison and artifact-verification tools
eval/ Frozen benchmark and retained results
examples/ Minimal Python and stdin examples
scripts/ Release checks and development utilities
tests/ Python regression tests
docs/ Architecture, provenance, and evaluation documentation

Deterministic realization coverage and lower-level APIs are documented in docs/realizer-coverage.md. Contributor workflow is documented in CONTRIBUTING.md.

Development

Requirements: Python 3.11 or newer, Rust, and uv.

uv sync --all-groups
uv run ruff format --check .
uv run ruff check .
uv run pytest
cargo test --manifest-path rust/Cargo.toml
uv build

License and attribution

Premove ITN source code and model weights are MIT licensed. The contextual scorer uses microsoft/deberta-v3-large at a pinned revision. The architecture derives from the DeBERTaV3 paper.

The Rust realization layer uses text-processing-rs, which is Apache-2.0 licensed. Required third-party licenses and notices are in THIRD_PARTY_NOTICES.md and LICENSES/.

Project links

Data verified by PyPI on Sep 9, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for aryamantodkar from gravatar.com aryamantodkar

Credits

Author: premove-ai

GitHub Statistics

Data verified by PyPI on Sep 9, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

MIT
View SPDX License List

Requires

Python >=3.11

Tags

asr inverse text normalization nlp speech processing voice agents

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

premove_itn-0.1.0.tar.gz (84.3 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

premove_itn-0.1.0-cp313-cp313-manylinux_2_28_x86_64.whl (704.8 kB view details)

Uploaded CPython 3.13manylinux: glibc 2.28+ x86-64

premove_itn-0.1.0-cp313-cp313-macosx_11_0_arm64.whl (617.6 kB view details)

Uploaded CPython 3.13macOS 11.0+ ARM64

premove_itn-0.1.0-cp312-cp312-manylinux_2_28_x86_64.whl (705.0 kB view details)

Uploaded CPython 3.12manylinux: glibc 2.28+ x86-64

premove_itn-0.1.0-cp312-cp312-macosx_11_0_arm64.whl (617.5 kB view details)

Uploaded CPython 3.12macOS 11.0+ ARM64

premove_itn-0.1.0-cp311-cp311-manylinux_2_28_x86_64.whl (708.3 kB view details)

Uploaded CPython 3.11manylinux: glibc 2.28+ x86-64

premove_itn-0.1.0-cp311-cp311-macosx_11_0_arm64.whl (622.5 kB view details)

Uploaded CPython 3.11macOS 11.0+ ARM64

File details

Details for the file premove_itn-0.1.0.tar.gz.

File metadata

  • Download URL: premove_itn-0.1.0.tar.gz
  • Upload date:
  • Size: 84.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for premove_itn-0.1.0.tar.gz
Algorithm Hash digest
SHA256 e8083e08b8bcceaf25fecd92e5af02a8cc05eea23d1be9cc3f07b049cfc8b31e
MD5 8106193ca5ede5b15b1bce8b7d2f8a1e
BLAKE2b-256 b6244109d31fb769028991b663e23944c6ab54f9e4098f14076710cc6561ec8e

See more details on using hashes here.

Provenance

The following attestation bundles were made for premove_itn-0.1.0.tar.gz:

Publisher: release.yml on premove-ai/premove-itn

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file premove_itn-0.1.0-cp313-cp313-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for premove_itn-0.1.0-cp313-cp313-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 ac4feb3aad5096a6510fc0f512cf4d9e8d77bee91533e0a4b6062375810ee9a9
MD5 74a8c548bc26f6cfd35fa7d779eeb215
BLAKE2b-256 7774a57902da5fb86d8904575dba14162d0b360391c97913c866bfc8db8a360e

See more details on using hashes here.

Provenance

The following attestation bundles were made for premove_itn-0.1.0-cp313-cp313-manylinux_2_28_x86_64.whl:

Publisher: release.yml on premove-ai/premove-itn

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file premove_itn-0.1.0-cp313-cp313-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for premove_itn-0.1.0-cp313-cp313-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 6fb2dc35740feec1ff30daf0abeead06c512692da06706b8b4f6c1c25cc61898
MD5 d0af223f061029f80ce549993a0296fc
BLAKE2b-256 5f45a67b2bea32bbd6cbf8a44cc0bf6e1d5ba7c5c8ec6ad96351739d84c396bb

See more details on using hashes here.

Provenance

The following attestation bundles were made for premove_itn-0.1.0-cp313-cp313-macosx_11_0_arm64.whl:

Publisher: release.yml on premove-ai/premove-itn

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file premove_itn-0.1.0-cp312-cp312-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for premove_itn-0.1.0-cp312-cp312-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 1abbc1d81ce40c5e0831e4e19fa711cda867822e691b63590b1a273ec7ab1b78
MD5 85b7c448d13029ce3f5229b4e292385a
BLAKE2b-256 d1e58e53063bf2c46971e7bc92bec9419332791f121665682f70a11674926214

See more details on using hashes here.

Provenance

The following attestation bundles were made for premove_itn-0.1.0-cp312-cp312-manylinux_2_28_x86_64.whl:

Publisher: release.yml on premove-ai/premove-itn

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file premove_itn-0.1.0-cp312-cp312-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for premove_itn-0.1.0-cp312-cp312-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 b91524aa2a8be2ba79c4fb03f6f24934a922c689153178ae3b741ec99ca7a67b
MD5 731ff22d641574603dabc1a106bd8069
BLAKE2b-256 071cea16fe8df0f8d67ee0b3863b07edf7096c840b6a734a478cfc06810ec822

See more details on using hashes here.

Provenance

The following attestation bundles were made for premove_itn-0.1.0-cp312-cp312-macosx_11_0_arm64.whl:

Publisher: release.yml on premove-ai/premove-itn

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file premove_itn-0.1.0-cp311-cp311-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for premove_itn-0.1.0-cp311-cp311-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 0babb3294263dad771b310d57210472200b6c95a044676a204590e0bc0b7fca1
MD5 4d91e53c0b2ab8de7398a40818ca0dc0
BLAKE2b-256 4d525e43fc1fe76e80d358104b5b1b790399723b152e604a67e8b315d615cd6c

See more details on using hashes here.

Provenance

The following attestation bundles were made for premove_itn-0.1.0-cp311-cp311-manylinux_2_28_x86_64.whl:

Publisher: release.yml on premove-ai/premove-itn

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file premove_itn-0.1.0-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for premove_itn-0.1.0-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 c32b4e16e114ae551ab42103c57249e984440fcd5f0479f769a7359c3ab2bf14
MD5 1ef0b26b5edd5d189a0af2d05258e02a
BLAKE2b-256 40299a7246f028b60b5f6a605dd7924323ada8026e8d9c53039cf4836d986c0a

See more details on using hashes here.

Provenance

The following attestation bundles were made for premove_itn-0.1.0-cp311-cp311-macosx_11_0_arm64.whl:

Publisher: release.yml on premove-ai/premove-itn

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

0.1.0 This release

7 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page