Skip to main content

Parallel and distributed training with spaCy and Ray

Project description

spacy-ray: Parallel and distributed training with spaCy and Ray

⚠️ This repo is still a work in progress and requires the new spaCy v3.0.

Ray is a fast and simple framework for building and running distributed applications. This very lightweight extension package lets you use Ray for parallel and distributed training with spaCy. If spacy-ray is installed in the same environment as spaCy, it will automatically add spacy ray commands to your spaCy CLI.

The main command is spacy ray train for parallel and distributed training, but we expect to add spacy ray pretrain and spacy ray parse as well.

Azure Pipelines Current Release Version PyPi Version

🚀 Quickstart

You can install spacy-ray from pip:

pip install spacy-ray

To check if the command has been registered successfully:

python -m spacy ray --help

Train a model using the same API as spacy train:

python -m spacy ray train config.cfg --n-workers 2

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

spacy_ray-0.1.4.tar.gz (12.3 kB view hashes)

Uploaded source

Built Distribution

spacy_ray-0.1.4-py2.py3-none-any.whl (13.3 kB view hashes)

Uploaded py2 py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page