khocr-gen 0.1.0

Synthetic OCR training data generator for mixed Khmer/English text

khocr-gen

Synthetic OCR training data generator for mixed Khmer/English text.

Features

• Mixed-script rendering: per-span font selection for Khmer + English
• 24 augmentation methods: unified registry covering scanner/camera degradations and training-time transforms
• Rust acceleration: 21/24 augmentation methods plus font glyph checking run through a native PyO3 extension, with automatic pure-Python fallback
• Isolated augmentation: one effect per image, weighted by configurable probabilities
• Configurable intensity ranges: per-method [min, max] with linear sampling
• Multiprocess generation: parallel workers for throughput
• LMDB packing: efficient key–value storage for training pipelines
• n-way dataset combine: merge multiple generated datasets into one
• Built-in viewer: preview and extract images from LMDB databases
• YAML config files: reproducible generation recipes
• Khmer normalization: canonical character ordering via khmernormalizer

Installation

# Clone and install with uv
git clone https://github.com/LazyGreed/khocr-gen.git
cd khocr-gen
uv sync

# Or install with pip
pip install .

uv sync also compiles and installs the optional Rust acceleration extension (khocr-gen-core, from rust/). This requires a Rust toolchain (cargo/rustc) on PATH — see RUST_ACCELERATION.md. If the extension isn't available, khocr-gen falls back to pure-Python implementations automatically.

Requirements

• Python >= 3.12
• Rust toolchain (cargo/rustc, edition 2021) — only needed to build the optional acceleration extension; the project works without it

Quick Start

1. Prepare fonts

fonts/
├── khmer/     <- .ttf / .otf Khmer fonts
└── english/   <- .ttf / .otf English fonts

Fonts placed directly in fonts/ (not in a subdirectory) are added to both pools as fallbacks.

2. Prepare corpus

A plain UTF-8 text file, one string per line:

សួស្តី
Hello World
ស្វាគមន៍ Welcome

3. Generate a dataset

khocr-gen generate --corpus corpus/corpus.txt --fonts fonts --output data --copies 3 --storage lmdb

4. Verify augmentation quality

Renders every augmentation method at min and max intensity on clean canvases:

khocr-gen verify --fonts fonts --output verify_output

5. View generated images

# Summary of an LMDB database
khocr-gen view --lmdb data/train/lmdb

# Extract images
khocr-gen view --lmdb data/train/lmdb --output-dir extracted

# Show labels only
khocr-gen view --lmdb data/train/lmdb --labels-only

6. Combine multiple datasets

khocr-gen combine data_run1 data_run2 data_run3 --output data_merged

Configuration

All generation parameters can be specified via CLI flags, a YAML config file, or both (CLI overrides YAML). See CONFIG.md for details.

khocr-gen generate --config configs/generate.yml

Config loading order (highest priority wins):

1. argparse built-in defaults
2. YAML config file values
3. Explicit CLI flags

Augmentation

24 methods in a unified registry (21 of them Rust-accelerated). Each generated image receives exactly one augmentation, chosen probabilistically by weight. See AUGMENTATION.md for the full catalog and visual examples.

CLI Reference

See CLI_REFERENCE.md for complete command documentation.

Rust Acceleration

See RUST_ACCELERATION.md for what's accelerated, how the native extension is built/installed, and how to iterate on the rust/ crate.

Project Structure

src/khocr_gen/
├── cli.py              # CLI entry point
├── config.py           # GenerationConfig + AugMethodConfig dataclasses
├── config_loader.py    # YAML config loading
├── corpus.py           # Corpus loading / filtering / counting
├── fonts.py            # FontManager with script-aware font selection
├── normalizer.py       # Khmer text normalization wrapper
├── rendering.py        # ImageRenderer: text -> clean canvas -> augmentation
├── augmentation.py     # 24 augmentation methods (unified registry)
├── _rust_accel.py      # Native/pure-Python dispatch layer for the Rust extension
├── parallel.py         # Multiprocess rendering workers
├── lmdb_pack.py        # LMDB database packing
├── combine.py          # n-way dataset merge
├── data_generator.py   # DatasetGenerator orchestrator
├── generate.py         # generate CLI command
├── verify.py           # verify CLI command
├── viewer.py           # view CLI command
├── combine_cmd.py      # combine CLI command
└── errors.py           # Exception hierarchy

rust/                    # khocr-gen-core: PyO3/maturin native acceleration extension
├── Cargo.toml
├── pyproject.toml
└── src/
    ├── lib.rs           # PyO3 bindings + module registration
    ├── augmentation.rs  # Accelerated augmentation methods
    ├── fonts.rs         # FontManager + FontFace (cmap glyph lookup)
    ├── rendering.rs     # Unwired canvas-rendering prototype (not yet bound to Python)
    └── utils.rs

Acknowledgments

khocr-gen builds on excellent open-source libraries:

• khmernormalizer (MIT) - Khmer text normalization
• OpenCV (Apache 2.0) - Image processing and augmentation
• Pillow (HPND) - Font rendering and image creation
• Albumentations (MIT) - Augmentation primitives
• NumPy (BSD-3-Clause) - Numerical operations
• LMDB (OpenLDAP 2.8) - Embedded database for training pipelines

License

MIT