Skip to main content

PyPI - PyPi Downloads

Lingtrain Aligner

ML powered library for the accurate texts alignment in different languages.

Cover

Purpose

Main purpose of this alignment tool is to build parallel corpora using two or more raw texts in different languages. Texts should contain the same information (i.e., one text should be a translated analog oh the other text). E.g., it can be the Drei Kameraden by Remarque in German and the Three Comrades — it's translation into English.

Process

There are plenty of obstacles during the alignment process:

  • The translator could translate several sentences as one.
  • The translator could translate one sentence as many.
  • There are some service marks in the text
    • Page numbers
    • Chapters and other section headings
    • Author and title information
    • Notes

While service marks can be handled manually (the tool helps to detect them), the translation conflicts should be handled more carefully.

Lingtrain Aligner tool will do almost all alignment work for you. It matches the sentence pairs automatically using the multilingual machine learning models. Then it searches for the alignment conflicts and resolves them. As output you will have the parallel corpora either as two distinct plain text files or as the merged corpora in widely used TMX format.

Supported languages and models

Automated alignment process relies on the sentence embeddings models. Embeddings are multidimensional vectors of a special kind which are used to calculate a distance between the sentences. Supported languages list depend on the selected backend model.

  • distiluse-base-multilingual-cased-v2
    • more reliable and fast
    • moderate weights size — 500MB
    • supports 50+ languages
    • full list of supported languages can be found in this paper
  • LaBSE (Language-agnostic BERT Sentence Embedding)
    • can be used for rare languages
    • pretty heavy weights — 1.8GB
    • supports 100+ languages
    • full list of supported languages can be found here

Profit

  • Parallel corpora by itself can used as the resource for machine translation models or for linguistic researches.
  • My personal goal of this project is to help people building parallel translated books for the foreign language learning.

Metadata

Release files for lingtrain-aligner 0.9.6

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for lingtrain-aligner 0.9.6
File Size Uploaded Provenance Actions
lingtrain-aligner-0.9.6.tar.gz 49.7 kB Without provenance Details for lingtrain-aligner-0.9.6.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for lingtrain-aligner 0.9.6
File Interpreter ABI Platform Actions
lingtrain_aligner-0.9.6-py3-none-any.whl Python 3 none any Details for lingtrain_aligner-0.9.6-py3-none-any.whl

Total release size: 101.6 kB

Release files / lingtrain-aligner-0.9.6.tar.gz

Download URL lingtrain-aligner-0.9.6.tar.gz
Size 49.7 kB
Tags Source
SHA-256 checksum
How to use checksums
81f26d0af761f94f3710ced1de1ff5b951e8c266ccfcacbe97a08384c6a4c4d9
BLAKE2b-256 checksum
How to use checksums
39126cfec87d7b32c8914ff5cd098ecedd1c4d766773898b32c828a7bc1a6b61
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.1 CPython/3.9.5

Release files / lingtrain_aligner-0.9.6-py3-none-any.whl

Download URL lingtrain_aligner-0.9.6-py3-none-any.whl
Size 51.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f1ebea223f7ac2f286699b1877e8fb0db2cd6ff03918e66b28897e6986a503fb
BLAKE2b-256 checksum
How to use checksums
0adcb1596839b6bf2604010f87f7e8ec5f116ad68304fdffe87c14e468411071
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.1 CPython/3.9.5

Security for lingtrain-aligner 0.9.6

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.9.6 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page