LinguAligner 0.31

LinguAligner is a Python library for aligning annotations in parallel corpora. It is designed to be used in the context of parallel corpora annotation alignment, where the goal is to align annotations in the source language with annotations in the target language.

LinguAligner

LinguALigner is a comprehensive corpus translation and alignment pipeline designed to facilitate the translation of corpora across different languages. It translates corpora using machine translation and aligns the translated annotations with their corresponding translated text. Initially developed for the automatic translation of ACE-2005 into Portuguese, LinguALigner has since been adapted into a versatile package for effortless translation of other corpora.

It is composed of two main components:

• Text translation: We support DeepL Translator, Google Translator and Microsoft Translators APIs.
• Annotations alignments: We developed an annotation alignment pipeline that uses several alignment techniques to align the translated annotations within the translated text.

Annotation Alignment Modules

Our pipeline is composed of a total of five annotation alignment components:

- Lemmatization
- Multiple word translation
- BERT-based word aligner
- Gestalt Patter Matching
- Levenstein distance

The pipeline operates sequentially, meaning that annotations aligned by earlier methods are not addressed by subsequent pipeline elements. According to our experiments, the list above corresponds to the best order sequence.

Usage

1. Translate Corpora You can use the Translation APIs or can translate your corpus with an external tool An API key is needed to use some of the Translation APIs.

from LinguAligner import translation

# Google Translator
translator = translation.GoogleTranslator(source_lang="en", target_lang="pt")
translated_text = translator.translate("The soldiers were ordered to fire their weapons")

# DeepL Translator
translator = translation.DeepLTranslator(source_lang="en", target_lang="pt", key="DEEPL_KEY")
translated_text = translator.translate("The soldiers were ordered to fire their weapons")

# Microsoft Translator
translator = translation.MicrosoftTranslator(source_lang="en", target_lang="pt", key="MICROSOFT_TRANSLATOR_KEY")
translated_text = translator.translate("The soldiers were ordered to fire their weapons")
print(translated_text)

2. Run the Annotation Alignment Pipeline Users can select the aligners they intend to use and specify the order in which they should be utilized. To find the best component order in the pipeline we experimented with all the permutations between the components and calculated the corresponding alignment results using a manually aligned corpus. According to our findings, the best sequence order is the ones presented in the example below, however, we encourage you to experiment with different orders for your specific use case.

   Certain alignment methods, like multiple translations (M_Trans), necessitate the prior calculation of multiple translations for each annotation (as explained at the end of this section).

from LinguAligner import AlignmentPipeline

"""
(By default, the first method used is string matching. If unsuccessful, the alignment pipeline is employed.)
Methods:
- lemma: Lemmatization
- M_Trans: Multiple Translations of a word
- word_aligner: mBERT-based word aligner
- gestalt: Gestalt pattern matching (character-based)
- levenshtein: Levenshtein distance (character-based)
"""

config= {
    "pipeline": [ "lemma", "M_Trans", "word_aligner","gestalt","leveinstein"], # can be changed according to the desired pipeline
    "spacy_model": "pt_core_news_lg", # change according to the target language
    "WAligner_model": "bert-base-multilingual-uncased", # needed for word_aligner
}

aligner = AlignmentPipeline(config)

src_sentence = "The soldiers were ordered to fire their weapons."
src_annotation = "fire"
translated_sentence = "Os soldados receberam ordens para disparar as suas armas."
translated_annotation = "incêndio"

target_annotation = aligner.align_annotation(src_sentence, src_annotation, translated_sentence, translated_annotation)
print(target_annotation)

>>> ('disparar', (34, 41))

For example, in the sentence 'The soldiers were ordered to fire their weapons,' the word 'fire' was annotated in the source corpus. However, when this sentence is translated to 'Os soldados receberam ordens para disparar as suas armas,' the word 'fire' is translated to 'incêndio' (fire as a noun) in isolation, and to 'disparar' (as a verb) in the translated sentence.

*Spacy models must be pre-installed corresponding to the target language.

Specify source annotation start index to find the closest target annotation

src_sentence = "he was a good man because he had a kind heart"
src_annotation = "he"
translated_sentence = "ele era um bom homem porque ele tinha um bom coração" # there are multiple tokens "ele" (he)
translated_annotation = "ele"

#add src_ann_start argument
target_annotation = aligner.align_annotation(src_sentence, src_annotation, translated_sentence, translated_annotation, src_ann_start=29)
print(target_annotation)

>>> ('ele', (28, 30))

Note

To use the M_trans method, multiple translations of the annotations must be computed beforehand and passed as an argument to the align_annotation function. These translations should contained in a Python dictionary, where the source annotation serves as the key, and the corresponding value is a list of alternative translations. You can generate this dictionary using the following code (need a MICROSOFT_TRANSLATOR_KEY):

from LinguAligner import translation
translator = translation.MicrosoftTranslator(source_lang="en", target_lang="pt", auth_key="MICROSOFT_TRANSLATOR_KEY")
lookupTable = {}
annotations_list = ["war","land","fire"]
for word in annotations_list:
    lookupTable[word] = translator.getMultipleTranslations(word) # change the language codes according to the desired languages

# Then, pass the lookupTable to the align_annotation method
x = aligner.align_annotation("The soldiers were ordered to fire their weapons","fire", "Os soldados receberam ordens para disparar as suas armas","incêndio",lookupTable)

The lookup table should resemble the following example:

{
    'fire': 
        [
            'fogo',
            'incêndio',
            'demitir',
            'despedir',
            'fogueira',
            'disparar',
            'chamas',
            'dispare',
            'lareira',
            'atirar',
            'atire'
        ],
    ...
}

Evaluation

To measure the effectiveness of the alignment pipeline we tested it on ACE-2005 corpus. Manual alignments were conducted on the entire ACE-2005-PT test set, which includes 1,310 annotations. These alignments were performed by a linguist expert to ensure high-quality annotations, following the same annotation guidelines of the original ACE-2005 corpus. Then we compare the manual alignments against the ones generated by our pipeline.

The evaluation results are presented in Table 1:

Table 1: Evaluation Results by pipeline component

License

This project is licensed under the MIT License.

Citation

Coming Soon.