arpabo 0.1.0

Build ARPA format statistical language models with multiple smoothing methods

arpabo

Build ARPA format statistical language models with multiple smoothing methods.

Features

• Multiple smoothing methods (Good-Turing, Kneser-Ney, Katz backoff)
• Support for arbitrary n-gram orders
• Standard ARPA format output
• Binary format conversion (PocketSphinx, Kaldi)
• Corpus normalization tool
• Interactive debug mode
• Zero runtime dependencies (pure Python)

Installation

pip install arpabo

This installs two commands:

• arpabo - Build language models
• arpabo-normalize - Normalize text corpora

Quick Start

# Quick demo
arpabo --demo -o model.arpa

# Build from your corpus
arpabo corpus.txt -o model.arpa

# With binary conversion
arpabo corpus.txt -o model.arpa --to-bin

# Two-stage: normalize then build
arpabo-normalize corpus.txt -o normalized.txt -c lower -n
arpabo normalized.txt -o model.arpa

Python API

from arpabo import ArpaBoLM

# Build a language model
lm = ArpaBoLM(max_order=3, smoothing_method="good_turing")
with open("corpus.txt") as f:
    lm.read_corpus(f)
lm.compute()
lm.write_file("model.arpa")

Smoothing Methods

• good_turing (default) - Best for sparse data
• kneser_ney - Best for larger corpora
• auto - Automatically optimizes discount mass
• fixed - Fixed discount mass (use -d 0.0 for MLE)

Common Workflows

Basic Usage

arpabo corpus.txt -o model.arpa

With Options

# 4-gram with Kneser-Ney smoothing
arpabo corpus.txt -o model.arpa -m 4 -s kneser_ney

# Lowercase normalization
arpabo corpus.txt -o model.arpa -c lower -v

# Token normalization (strip punctuation)
arpabo corpus.txt -o model.arpa -n

Corpus Preprocessing

# Normalize separately
arpabo-normalize corpus.txt -o clean.txt -c lower -n

# Build model
arpabo clean.txt -o model.arpa

# Or pipeline
cat corpus.txt | arpabo-normalize -c lower -n | arpabo -o model.arpa

Binary Conversion

# Automatic PocketSphinx binary
arpabo corpus.txt -o model.arpa --to-bin

# Kaldi FST format
arpabo corpus.txt -o model.arpa --to-fst

# Manual conversion
pocketsphinx_lm_convert -i model.arpa -o model.lm.bin

Compatibility

ArpaLM produces standard ARPA format models compatible with:

• Kaldi - Convert with arpa2fst
• PocketSphinx - Convert with pocketsphinx_lm_convert
• SphinxTrain - Use ARPA directly
• NVIDIA Riva - ARPA format supported
• Julius, HTK - ARPA compatible

Development

git clone https://github.com/lenzo-ka/arpabo.git
cd arpabo
make venv
source venv/bin/activate
make test

See CONTRIBUTING.md for details.

License

MIT