spaCy + Trankit

This package wraps the Trankit library, so you can use trankit models in a spaCy pipeline.

Using this wrapper, you'll be able to use the following annotations, computed by your pretrained trankit pipeline/model:

• Statistical tokenization (reflected in the Doc and its tokens)
• Lemmatization (token.lemma and token.lemma_)
• Part-of-speech tagging (token.tag, token.tag_, token.pos, token.pos_)
• Morphological analysis (token.morph)
• Dependency parsing (token.dep, token.dep_, token.head)
• Named entity recognition (doc.ents, token.ent_type, token.ent_type_, token.ent_iob, token.ent_iob_)
• Sentence segmentation (doc.sents)
• Multiword token preservation for languages such as Arabic and Hebrew via token._.trankit_expanded

️️️⌛️ Installation

As of v0.2.1 spacy-trankit is only compatible with spaCy v3.x. On Python 3.12, spacy-trankit applies a runtime compatibility patch for the current trankit dataclass issue in adapter_transformers before creating the pipeline. To install the most recent version:

pip install git+https://github.com/imvladikon/spacy-trankit

or from pypi:

pip install spacy-trankit

📖 Usage & Examples

Load pre-trained trankit model into a spaCy pipeline:

import spacy_trankit

# Initialize the pipeline
nlp = spacy_trankit.load("en")

doc = nlp("Barack Obama was born in Hawaii. He was elected president in 2008.")
for token in doc:
    print(token.text, token.lemma_, token.pos_, token.dep_, token.ent_type_)
print(doc.ents)

By default, mwt_strategy="auto" expands multiword tokens when the expanded tokens can be aligned back to the original text without changing doc.text. Expansions that cannot be represented as substrings of the original text are kept non-destructive. For example, Arabic and Hebrew clitic expansions can differ from the surface token, so the spaCy token keeps the original surface form and stores Trankit's expansion under token._.trankit_expanded.

doc = nlp("ذهبت للبيت اليوم")
for token in doc:
    print(token.text, token._.trankit_expanded)

If you always want surface tokens, pass mwt_strategy="preserve". If you need the previous expanded-token behavior and accept that spaCy may have to replace the original text with space-separated expanded tokens for unalignable cases, pass mwt_strategy="expand":

nlp = spacy_trankit.load("ar", mwt_strategy="preserve")
nlp = spacy_trankit.load("ar", mwt_strategy="expand")

Load it from the path:

import spacy_trankit

# Initialize the pipeline
nlp = spacy_trankit.load_from_path(name="en", path="./cache") 

doc = nlp("Barack Obama was born in Hawaii. He was elected president in 2008.")
for token in doc:
    print(token.text, token.lemma_, token.pos_, token.dep_, token.ent_type_)
print(doc.ents)

📦 Model downloads

The Trankit release on PyPI fetches its pretrained models from nlp.uoregon.edu, which is currently unavailable. spacy-trankit bypasses that broken download path and pulls the same artifacts from Trankit's HuggingFace mirror (https://huggingface.co/uonlp/trankit) into the local cache before instantiating the Trankit pipeline. The behaviour is automatic; no extra setup is needed.

If you mirror the artifacts elsewhere (e.g. for offline / air-gapped use), point spacy-trankit at it via the SPACY_TRANKIT_MODEL_URL environment variable. The template understands {version}, {embedding} and {lang}:

export SPACY_TRANKIT_MODEL_URL="https://my-mirror.example.com/trankit/{version}/{embedding}/{lang}.zip"