Skip to main content

spaCy pipeline component for guessing the language of Doc and Span objects.

Project description

# spaCy-CLD: Bringing simple language detection to spaCy

## Installation

`pip install spacy_cld`

## Usage

Adding the spaCy-CLD component to the processing pipeline is relatively simple:

```
import spacy
from spacy_cld import LanguageDetector

nlp = spacy.load('en')
language_detector = LanguageDetector()
nlp.add_pipe(language_detector)
doc = nlp('This is some English text.')

doc._.languages # ['en']
doc._.language_scores['en'] # 0.96
```

spaCy-CLD operates on `Doc` and `Span` spaCy objects. When called on a `Doc` or `Span`, the object is given two attributes: `languages` (a list of up to 3 language codes) and `language_scores` (a dictionary mapping language codes to confidence scores between 0 and 1).

## Under the hood

spacy-cld is a little extension that wraps the [PYCLD2](https://github.com/aboSamoor/pycld2) Python library, which in turn wraps the [Compact Language Detector 2](https://github.com/CLD2Owners/cld2) C library originally built at Google for the Chromium project. CLD2 uses character n-grams as features and a Naive Bayes classifier to identify 80+ languages from Unicode text strings (or XML/HTML). It can detect up to 3 different languages in a given document, and reports a confidence score (reported in with each language.

For additional details, see the linked project pages for PYCLD2 and CLD2.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

spacy_cld-0.1.0.tar.gz (3.3 kB view details)

Uploaded Source

File details

Details for the file spacy_cld-0.1.0.tar.gz.

File metadata

  • Download URL: spacy_cld-0.1.0.tar.gz
  • Upload date:
  • Size: 3.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for spacy_cld-0.1.0.tar.gz
Algorithm Hash digest
SHA256 f40178116c90bcb77d343976f9502c865b56643cb9dd8c7e3ca93c91303872cd
MD5 0572f0ff474332ec85c0b348ad248619
BLAKE2b-256 e33bf5344007259b5beb0a8e0d7b9e6b0d2c5c4dcfe674bc94b7497bcc201ee0

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page