This release is a pre-release and may not be stable for production use.
HuSpaCy is a spaCy library providing industrial-strength Hungarian language processing facilities through spaCy models. The released pipelines consist of a tokenizer, sentence splitter, lemmatizer, tagger (predicting morphological features as well), dependency parser and a named entity recognition module. Word and phrase embeddings are also available through spaCy's API. All models have high throughput, decent memory usage and close to state-of-the-art accuracy. A live demo is available here, model releases are published to Hugging Face Hub.
This repository contains material to build HuSpaCy and all of its models in a reproducible way.
Installation
To get started using the tool, first, we need to download one of the models. The easiest way to achieve this is to install huspacy (from PyPI) and then fetch a model through its API.
pip install huspacy
import huspacy
# Download the latest CPU optimized model
huspacy.download()
Install the models directly
You can install the latest models directly from 🤗 Hugging Face Hub:
- CPU optimized large model:
pip install https://huggingface.co/huspacy/hu_core_news_lg/resolve/main/hu_core_news_lg-any-py3-none-any.whl - GPU optimized transformers model:
pip install https://huggingface.co/huspacy/hu_core_news_trf/resolve/main/hu_core_news_trf-any-py3-none-any.whl
To speed up inference on GPUs, CUDA should be installed as described in https://spacy.io/usage.
Read more on the models here
Quickstart
HuSpaCy is fully compatible with spaCy's API, newcomers can easily get started with spaCy 101 guide.
Although HuSpacy models can be loaded with spacy.load(...), the tool provides convenience methods to easily access downloaded models.
# Load the model using spacy.load(...)
import spacy
nlp = spacy.load("hu_core_news_lg")
# Load the default large model (if downloaded)
import huspacy
nlp = huspacy.load()
# Load the model directly as a module
import hu_core_news_lg
nlp = hu_core_news_lg.load()
To process texts, you can simply call the loaded model (i.e. the nlp callable object)
doc = nlp("Csiribiri csiribiri zabszalma - négy csillag közt alszom ma.")
As HuSpaCy is built on spaCy, the returned doc document contains all the annotations given by the pipeline components.
API Documentation is available in our website.
Models overview
We provide several pretrained models:
hu_core_news_lgis a CNN-based large model which achieves a good balance between accuracy and processing speed. This default model provides tokenization, sentence splitting, part-of-speech tagging (UD labels w/ detailed morphosyntactic features), lemmatization, dependency parsing and named entity recognition and ships with pretrained word vectors.hu_core_news_trfis built on huBERT and provides the same functionality as the large model except the word vectors. It comes with much higher accuracy in the price of increased computational resource usage. We suggest using it with GPU support.hu_core_news_mdgreatly improves onhu_core_news_lg's throughput by loosing some accuracy. This model could be a good choice when processing speed is crucial.hu_core_news_trf_xlis an experimental model built on xlm-roberta-large. It provides as thehu_core_news_trfmodel, however it comes with slightly higher accuracy in the price of significantly increased computational resource usage. We suggest using it with GPU support.
HuSpaCy's model versions follows spaCy's versioning scheme.
A demo of the models is available at Hugging Face Spaces.
To read more about the model's architecture we suggest reading the relevant sections from spaCy's documentation.
Comparison
| Models | md |
lg |
trf |
trf_xl |
|---|---|---|---|---|
| Embeddings | 100d floret | 300d floret | transformer:huBERT |
transformer:XLM-RoBERTa-large |
| Target hardware | CPU | CPU | GPU | GPU |
| Accuracy | ⭑⭑⭑⭒ | ⭑⭑⭑⭑ | ⭑⭑⭑⭑⭒ | ⭑⭑⭑⭑⭑ |
| Resource usage | ⭑⭑⭑⭑⭑ | ⭑⭑⭑⭑ | ⭑⭑ | ⭒ |
Citing
If you use any of the models or HuSpaCy as library in your research, please cite this paper:
@misc{HuSpaCy:2021,
title = {{HuSpaCy: an industrial-strength Hungarian natural language processing toolkit}},
booktitle = {{XVIII. Magyar Sz{\'a}m{\'\i}t{\'o}g{\'e}pes Nyelv{\'e}szeti Konferencia}},
author = {Orosz, Gy{\"o}rgy and Sz{\' a}nt{\' o}, Zsolt and Berkecz, P{\' e}ter and Szab{\' o}, Gerg{\H o} and Farkas, Rich{\' a}rd},
location = {{Szeged}},
year = {2022},
}
Additionally, please indicate the version of the model you used so that your research can be reproduced.
Contact
For feature requests, issues and bugs please use the GitHub Issue Tracker. Otherwise, reach out to us in the Discussion Forum.
Authors
HuSpaCy is implemented in the SzegedAI team, coordinated by Orosz György in the Hungarian AI National Laboratory, MILAB program.
License
This library is released under the Apache 2.0 License
Trained models have their own license (CC BY-SA 4.0) as described on the models page.
1 maintainer
Release files for huspacy-nightly 0.9.0.dev224
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| huspacy_nightly-0.9.0.dev224-py3-none-any.whl | Python 3 | none | any | Details |
Release files / huspacy_nightly-0.9.0.dev224-py3-none-any.whl
| Download URL | huspacy_nightly-0.9.0.dev224-py3-none-any.whl |
|---|---|
| Size | 92.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
59c22ff42af6bea80dd110cb821819b0262f5732b5ef89429e3ca294a5ccf16f
|
|
BLAKE2b-256 checksum How to use checksums |
ec286c181f86112bde622a23031ef997014ee4c88e3b33be2886e508a2cc7f3b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.10.6
|
