Skip to main content

Use fast UDPipe models directly in spaCy

Project description

spaCy + UDPipe

This package wraps the fast and efficient UDPipe language-agnostic NLP pipeline (via its Python bindings), so you can use UDPipe pre-trained models as a spaCy pipeline for 50+ languages out-of-the-box. Inspired by spacy-stanza, this package offers slightly less accurate models that are in turn much faster (see benchmarks for UDPipe and Stanza).

Installation

Use the package manager pip to install spacy-udpipe.

pip install spacy-udpipe

After installation, use spacy_udpipe.download() to download the pre-trained model for the desired language.

A full list of pre-trained UDPipe models for supported languages can be found in languages.json.

Usage

The loaded UDPipeLanguage class returns a spaCy Language object, i.e., the object you can use to process text and create a Doc object.

import spacy_udpipe

spacy_udpipe.download("en") # download English model

text = "Wikipedia is a free online encyclopedia, created and edited by volunteers around the world."
nlp = spacy_udpipe.load("en")

doc = nlp(text)
for token in doc:
    print(token.text, token.lemma_, token.pos_, token.dep_)

As all attributes are computed once and set in the custom Tokenizer, the Language.pipeline is empty.

The type of text can be one of the following:

  • unprocessed: str,
  • presegmented: List[str],
  • pretokenized: List[List[str]].

Loading a custom model

The following code snippet demonstrates how to load a custom UDPipe model (for the Croatian language):

import spacy_udpipe

nlp = spacy_udpipe.load_from_path(lang="hr",
                                  path="./custom_croatian.udpipe",
                                  meta={"description": "Custom 'hr' model"})
text = "Wikipedija je enciklopedija slobodnog sadržaja."

doc = nlp(text)
for token in doc:
    print(token.text, token.lemma_, token.pos_, token.dep_)

This can be done for any of the languages supported by spaCy. For an exhaustive list, see spaCy languages.

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update the tests as appropriate. Tests are run automatically for each pull request on the master branch. To start the tests locally, first, install the package with pip install -e ., then run pytest in the root source directory.

License

  • Source code: MIT © Text Analysis and Knowledge Engineering Lab (TakeLab)
  • Available pre-trained models: CC BY-NC-SA 4.0

Project status

Maintained by Text Analysis and Knowledge Engineering Lab (TakeLab).

Notes

  • This package exposes a spacy_languages entry point in its setup.py so full suport for serialization is enabled:
    nlp = spacy_udpipe.load("en")
    nlp.to_disk("./udpipe-spacy-model")
    
    To properly load a saved model, you must pass the udpipe_model argument when loading it:
    udpipe_model = spacy_udpipe.UDPipeModel("en")
    nlp = spacy.load("./udpipe-spacy-model", udpipe_model=udpipe_model)
    
  • Known possible issues:
    • Tag map

      Token.tag_ is a CoNLL XPOS tag (language-specific part-of-speech tag), defined for each language separately by the corresponding Universal Dependencies treebank. Mappings between XPOS and Universal Dependencies POS tags should be defined in a TAG_MAP dictionary (located in language-specific tag_map.py files), along with optional morphological features. See spaCy tag map for more details.

    • Syntax iterators

      In order to extract Doc.noun_chunks, a proper syntax iterator implementation for the language of interest is required. For more details, please see spaCy syntax iterators.

    • Other language-specific issues

      A quick way to check language-specific defaults in spaCy is to visit spaCy language support. Also, please see spaCy language data for details regarding other language-specific data.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for spacy-udpipe, version 0.3.2
Filename, size File type Python version Upload date Hashes
Filename, size spacy_udpipe-0.3.2-py3-none-any.whl (13.3 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size spacy_udpipe-0.3.2.tar.gz (12.9 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page