Skip to main content

Alignment tool for texts in different languages

Project description

Lingtrain Aligner

ML powered library for the accurate texts alignment in different languages.

Cover

Purpose

Main purpose of this alignment tool is to build parallel corpora using two or more raw texts in different languages. Texts should contain the same information (i.e., one text should be a translated analog oh the other text). E.g., it can be the Drei Kameraden by Remarque in German and the Three Comrades — it's translation into English.

Process

There are plenty of obstacles during the alignment process:

  • The translator could translate several sentences as one.
  • The translator could translate one sentence as many.
  • There are some service marks in the text
    • Page numbers
    • Chapters and other section headings
    • Author and title information
    • Notes

While service marks can be handled manually (the tool helps to detect them), the translation conflicts should be handled more carefully.

Lingtrain Aligner tool will do almost all alignment work for you. It matches the sentence pairs automatically using the multilingual machine learning models. Then it searches for the alignment conflicts and resolves them. As output you will have the parallel corpora either as two distinct plain text files or as the merged corpora in widely used TMX format.

Supported languages and models

Automated alignment process relies on the sentence embeddings models. Embeddings are multidimensional vectors of a special kind which are used to calculate a distance between the sentences. Supported languages list depend on the selected backend model.

  • distiluse-base-multilingual-cased-v2
    • more reliable and fast
    • moderate weights size — 500MB
    • supports 50+ languages
    • full list of supported languages can be found in this paper
  • LaBSE (Language-agnostic BERT Sentence Embedding)
    • can be used for rare languages
    • pretty heavy weights — 1.8GB
    • supports 100+ languages
    • full list of supported languages can be found here

Profit

  • Parallel corpora by itself can used as the resource for machine translation models or for linguistic researches.
  • My personal goal of this project is to help people building parallel translated books for the foreign language learning.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

lingtrain-aligner-0.6.0.tar.gz (40.7 kB view details)

Uploaded Source

Built Distribution

lingtrain_aligner-0.6.0-py3-none-any.whl (42.7 kB view details)

Uploaded Python 3

File details

Details for the file lingtrain-aligner-0.6.0.tar.gz.

File metadata

  • Download URL: lingtrain-aligner-0.6.0.tar.gz
  • Upload date:
  • Size: 40.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/3.10.1 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.41.1 CPython/3.8.6

File hashes

Hashes for lingtrain-aligner-0.6.0.tar.gz
Algorithm Hash digest
SHA256 37671d106e0a73a39e9b3bdf3e43b01b09fcaca17f7e8a1e630d9fe8377bbbea
MD5 b05a5ec1701aa4b6c0790da6abd291c0
BLAKE2b-256 ab21b79ffc4da025a7598709b63c8594b9f711a4fcca833047d10da1f8e2744f

See more details on using hashes here.

File details

Details for the file lingtrain_aligner-0.6.0-py3-none-any.whl.

File metadata

  • Download URL: lingtrain_aligner-0.6.0-py3-none-any.whl
  • Upload date:
  • Size: 42.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/3.10.1 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.41.1 CPython/3.8.6

File hashes

Hashes for lingtrain_aligner-0.6.0-py3-none-any.whl
Algorithm Hash digest
SHA256 fcea3f96d99269187c08b8bb0872a8b9c6448ac05a98410a2806afb1b6ac16f6
MD5 849236d29c44dcb85199f1de7dc05121
BLAKE2b-256 d4d9961838dd89318634360880e1add69ce19140e14bc6eff06ceac2d66e8b08

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page