Skip to main content

LinguAligner is a Python library for aligning annotations in parallel corpora. It is designed to be used in the context of parallel corpora annotation alignment, where the goal is to align annotations in the source language with annotations in the target language.

Project description

LinguAligner

LinguALigner is a comprehensive corpus translation and alignment pipeline designed to facilitate the translation of corpora across different languages. It translates corpora using machine translation and aligns the translated annotations with their corresponding translated text. Initially developed for the automatic translation of ACE-2005 into Portuguese, LinguALigner has since been adapted into a versatile package for effortless translation of other corpora.

It is composed of two main components:

  • Text translation: We support DeepL Translator, Google Translator and Microsoft Translators APIs.
  • Annotations alignments: We developed an annotation alignment pipeline that uses several alignment techniques to align the translated annotations within the translated text.

Annotation Alignment Modules

Our pipeline is composed of a total of five annotation alignment components:

- Lemmatization
- Multiple word translation
- BERT-based word aligner
- Gestalt Patter Matching
- Levenstein distance

The pipeline operates sequentially, meaning that annotations aligned by earlier methods are not addressed by subsequent pipeline elements. According to our experiments, the list above corresponds to the best order sequence.

Usage

  1. Translate Corpora You can use the Translation APIs or can translate your corpus with an external tool An API key is needed to use some of the Translation APIs.
from LinguAligner import translation

# Google Translator
translator = translation.GoogleTranslator(source_lang="en", target_lang="pt")
translated_text = translator.translate("The soldiers were ordered to fire their weapons")

# DeepL Translator
translator = translation.DeepLTranslator(source_lang="en", target_lang="pt", key="DEEPL_KEY")
translated_text = translator.translate("The soldiers were ordered to fire their weapons")

# Microsoft Translator
translator = translation.MicrosoftTranslator(source_lang="en", target_lang="pt", key="MICROSOFT_TRANSLATOR_KEY")
translated_text = translator.translate("The soldiers were ordered to fire their weapons")
print(translated_text)
  1. Run the Annotation Alignment Pipeline Users can select the aligners they intend to use and specify the order in which they should be utilized. To find the best component order in the pipeline we experimented with all the permutations between the components and calculated the corresponding alignment results using a manually aligned corpus. According to our findings, the best sequence order is the ones presented in the example below, however, we encourage you to experiment with different orders for your specific use case.

    Certain alignment methods, like multiple translations (M_Trans), necessitate the prior calculation of multiple translations for each annotation (as explained at the end of this section).

from LinguAligner import AlignmentPipeline

"""
(By default, the first method used is string matching. If unsuccessful, the alignment pipeline is employed.)
Methods:
- lemma: Lemmatization
- M_Trans: Multiple Translations of a word
- word_aligner: mBERT-based word aligner
- gestalt: Gestalt pattern matching (character-based)
- levenshtein: Levenshtein distance (character-based)
"""

config= {
    "pipeline": [ "lemma", "M_Trans", "word_aligner","gestalt","leveinstein"], # can be changed according to the desired pipeline
    "spacy_model": "pt_core_news_lg", # change according to the target language
    "WAligner_model": "bert-base-multilingual-uncased", # needed for word_aligner
}

aligner = AlignmentPipeline(config)

src_sentence = "The soldiers were ordered to fire their weapons."
src_annotation = "fire"
translated_sentence = "Os soldados receberam ordens para disparar as suas armas."
translated_annotation = "incêndio"

target_annotation = aligner.align_annotation(src_sentence, src_annotation, translated_sentence, translated_annotation)
print(target_annotation)

>>> ('disparar', (34, 41))

For example, in the sentence 'The soldiers were ordered to fire their weapons,' the word 'fire' was annotated in the source corpus. However, when this sentence is translated to 'Os soldados receberam ordens para disparar as suas armas,' the word 'fire' is translated to 'incêndio' (fire as a noun) in isolation, and to 'disparar' (as a verb) in the translated sentence.

*Spacy models must be pre-installed corresponding to the target language.

Specify source annotation start index to find the closest target annotation

src_sentence = "he was a good man because he had a kind heart"
src_annotation = "he"
translated_sentence = "ele era um bom homem porque ele tinha um bom coração" # there are multiple tokens "ele" (he)
translated_annotation = "ele"

#add src_ann_start argument
target_annotation = aligner.align_annotation(src_sentence, src_annotation, translated_sentence, translated_annotation, src_ann_start=29)
print(target_annotation)

>>> ('ele', (28, 30))

Note

To use the M_trans method, multiple translations of the annotations must be computed beforehand and passed as an argument to the align_annotation function. These translations should contained in a Python dictionary, where the source annotation serves as the key, and the corresponding value is a list of alternative translations. You can generate this dictionary using the following code (need a MICROSOFT_TRANSLATOR_KEY):

from LinguAligner import translation
translator = translation.MicrosoftTranslator(source_lang="en", target_lang="pt", auth_key="MICROSOFT_TRANSLATOR_KEY")
lookupTable = {}
annotations_list = ["war","land","fire"]
for word in annotations_list:
    lookupTable[word] = translator.getMultipleTranslations(word) # change the language codes according to the desired languages

# Then, pass the lookupTable to the align_annotation method
x = aligner.align_annotation("The soldiers were ordered to fire their weapons","fire", "Os soldados receberam ordens para disparar as suas armas","incêndio",lookupTable)

The lookup table should resemble the following example:

{
    'fire': 
        [
            'fogo',
            'incêndio',
            'demitir',
            'despedir',
            'fogueira',
            'disparar',
            'chamas',
            'dispare',
            'lareira',
            'atirar',
            'atire'
        ],
    ...
}

Evaluation

To measure the effectiveness of the alignment pipeline we tested it on ACE-2005 corpus. Manual alignments were conducted on the entire ACE-2005-PT test set, which includes 1,310 annotations. These alignments were performed by a linguist expert to ensure high-quality annotations, following the same annotation guidelines of the original ACE-2005 corpus. Then we compare the manual alignments against the ones generated by our pipeline.

The evaluation results are presented in Table 1:

Results
Table 1: Evaluation Results by pipeline component

License

This project is licensed under the MIT License.

Citation

Coming Soon.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

lingualigner-0.34.tar.gz (200.5 kB view details)

Uploaded Source

Built Distribution

lingualigner-0.34-py2.py3-none-any.whl (10.9 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file lingualigner-0.34.tar.gz.

File metadata

  • Download URL: lingualigner-0.34.tar.gz
  • Upload date:
  • Size: 200.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: python-requests/2.31.0

File hashes

Hashes for lingualigner-0.34.tar.gz
Algorithm Hash digest
SHA256 cbbb142cb340de33c18bcfa74a2beea00138b2d91255f3427d92d774dcb47bc3
MD5 d8f095cdcf9760070d9183d362f9426d
BLAKE2b-256 820d5cb3c47e07701c5de79ff9deb7c47d349bfe51912896fe17c9fd60896d77

See more details on using hashes here.

File details

Details for the file lingualigner-0.34-py2.py3-none-any.whl.

File metadata

File hashes

Hashes for lingualigner-0.34-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 42a6a359529795a61ad10e373d6836a798f852a314ac2801498f0105ea8a7735
MD5 75f8c935431fade0a603f4747d84a188
BLAKE2b-256 e6b0be7c80bd6f0580aa5fb4cd06994787e3dfff7f785a996ac10b60eaeed1fd

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page