Bertalign

An automatic multilingual sentence aligner. Bertalign takes two texts that are translations of each other and works out which sentences correspond — including 1-to-many, many-to-1, and many-to-many mappings — producing aligned sentence pairs for translation study, corpus construction, or training data mining.

It implements the two-pass algorithm of Liu & Zhu (2022): sentences (and their overlapping concatenations) are embedded with LaBSE, a first dynamic programming pass finds approximate 1-1 anchors via top-k similarity search, and a second pass finds the optimal m-n alignment along that path.

This is a modernized fork of bfsujason/bertalign: packaged for PyPI, fully typed, GPU-accelerated end to end, with offline language detection and no network calls at alignment time.

Install

uv add bertalign        # or: pip install bertalign
uv add "bertalign[sat]" # optional: neural sentence splitting (see below)

Python 3.12+. A CUDA GPU is used automatically when available, for both embedding and similarity search; everything also runs on CPU.

Usage

from bertalign import Bertalign

aligner = Bertalign(src_text, tgt_text)
aligner.align_sents()
for src_segment, tgt_segment in aligner.sent_pairs():
    print(src_segment, "|||", tgt_segment)

Languages are detected automatically (offline, via lingua). If your texts already hold one sentence per line, pass is_split=True to skip splitting.

Aligning many document pairs reuses one loaded model automatically; to be explicit, share an encoder:

from bertalign import Bertalign, Encoder

encoder = Encoder("LaBSE")
for src_text, tgt_text in documents:
    aligner = Bertalign(src_text, tgt_text, model=encoder)
    aligner.align_sents()

Command line

bertalign source.txt target.txt > pairs.tsv

Inputs are plain-text files; output is one tab-separated pair per line (internal whitespace is normalized during cleaning, so fields never contain tabs). Options: --model (any sentence-transformers model), --max-align (largest bead size), --is-split, --sat, --trust-remote-code, --version.

Choosing a model

Any sentence-transformers model works via model=/--model. On the text+berg German–French gold standard (RTX 5070, this repository's scripts/benchmark_model.py):

LaBSE remains the default: it was trained specifically for translation-pair retrieval, and it is both the most accurate on strict F1 and by far the fastest. Retrieval models that expect instruction prefixes (e5, Qwen3) lose accuracy here because bertalign embeds raw sentences.

Sentence splitting

The default splitter is rule-based (sentence-splitter) and supports 25 languages: Catalan, Chinese, Czech, Danish, Dutch, English, Finnish, French, German, Greek, Hungarian, Icelandic, Italian, Latvian, Lithuanian, Norwegian, Polish, Portuguese, Romanian, Russian, Slovak, Slovenian, Spanish, Swedish, and Turkish.

The sat extra adds neural splitting with wtpsplit's SaT models (state of the art on multilingual segmentation benchmarks, language-agnostic, GPU-accelerated) — useful for noisy text or languages the rule splitter does not cover:

from bertalign import Bertalign
from bertalign.sat import sat_splitter

aligner = Bertalign(src_text, tgt_text, splitter=sat_splitter())

or bertalign --sat source.txt target.txt on the command line.

Evaluation

The scorer from Vecalign ships as bertalign.eval, and the repository carries the text+berg evaluation corpus:

just eval                                            # gold-standard regression gate
uv run python scripts/benchmark_model.py BAAI/bge-m3 # benchmark another model

just eval must reproduce Strict F1 0.936 / Lax F1 0.989 exactly; it is the correctness gate for any change to the aligner, kernels, or dependencies.

Development

uv sync --extra sat
just check   # ruff (select = ALL), ty, vulture, pytest, build

Licence and citation

GPL-3.0-or-later, inherited from upstream. If you use Bertalign in research, cite:

Lei Liu & Min Zhu. 2022. Bertalign: Improved word embedding-based sentence alignment for Chinese–English parallel corpora of literary texts. Digital Scholarship in the Humanities. https://doi.org/10.1093/llc/fqac089