About The Project
This project is an open-source Python package for existing NCHLT core technologies for ten South African languages (Afrikaans, isiNdebele, isiXhosa, isiZulu, Sesotho sa Leboa, Sesotho, Setswana, Siswati, Tshivenḓa, Xitsonga). The technologies include the following: Tokenisers, Sentence Separators, Part of Speech Taggers, Named Entity Recognisers, Phrase Chunkers, Optical Character Recognisers, Universal Part of Speech Taggers, Lemmatisers, and a Language Identifier. The package also includes a Morphological Analyser for isiNdebele, isiZulu, isiXhosa, and Siswati, totalling 85 technologies.
Getting Started
To get a local copy installed and running, follow these steps.
Prerequisites
- Python 3.8+ (https://www.python.org/downloads/)
- Java OpenJDK 17+ (https://openjdk.org)
- Requests 2.32.3 (https://pypi.org/project/requests/2.32.3/)
Installation
pip
pip install ctextcore
GitHub
# Download the source code from GitHub
git clone https://github.com/ctextdev/ctextcore.git
# Install from source
cd ctextcore
py -m pip install .
# Install from source in Development Mode
cd ctextcore
py -m pip install -e .
Usage
Importing the CTexT Core library
from ctextcore.core import CCore as core
server = core()
The core method accepts the following configuration arguments:
port: 8079 # Set the port the server should use
timeout: 60000 # Set the timeout of HTTP requests
threads: 5 # Set the total number of threads to use
memory: "4G" # Set the maximum memory allowed to be used by the server
be_quiet: False # Set the logging output from the server
max_char_length: 10000 # Set the maximum character length
server = core(port=8081,memory="16G",...)
Language codes
- Afrikaans -> af
- isiNdebele -> nr
- isiXhosa -> xh
- isiZulu -> zu
- Sesotho sa Leboa -> nso
- Sesotho -> st
- Setswana -> tn
- Siswati -> ss
- Tshivenḓa -> ve
- Xitsonga -> ts
Downloading models
Download all language models for a specific technology
# This call will download all the language models for POS.
server.download_model(tech='pos', language='all')
Download all technologies for a specific language
# This call will download all the technology models for isiZulu.
server.download_model(tech='all', language='zu')
Download a specific language model for a specific technology
# This call will download the POS technology model for Sesotho sa Leboa.
server.download_model(tech='pos', language='nso')
Using a model
# This call will run the isiZulu POS tagger on the input text 'E uma lungekho usuku olufakiwe, usuku lwakho lokubhalisa luyofakwa nge-othomathikhi kube usuku lokuqala lwenyanga elandelayo ukuze kungadaleki izikweletu.'.
output_process = server.process_text(text_input='E uma lungekho usuku olufakiwe, usuku lwakho lokubhalisa luyofakwa nge-othomathikhi kube usuku lokuqala lwenyanga elandelayo ukuze kungadaleki izikweletu.', language='zu', tech='pos')
print(output_process)
from pathlib import Path # Path needs to be imported to be able to use OCR
# This call will run the Sesotho sa Leboa OCR on the image or pdf path provided in the text_input argument.
output_process = server.process_text(text_input=Path('<path-to-image-or-pdf>'), language='nso', tech='ocr')
print(output_process)
# This call will run LID on the input text 'Sizoqhubeka ukwenza ngcono ukusebenza kukagesi wethu kanye nokuthembela kugesi ophinde uvuseleleke.' and the confidence level should be above 50%.
output_process = server.process_text(text_input='Sizoqhubeka ukwenza ngcono ukusebenza kukagesi wethu kanye nokuthembela kugesi ophinde uvuseleleke.', tech='lid', confidence=0.5)
print(output_process)
Output formats
The ctextcore package offers three different output formats (JSON, Delimited, List), the default output format is JSON and can be changed by providing the output_format argument in the process_text method. An extra argument, delimiter, can be used together with the delimited output format to change the delimiter used in the output. The default delimiter is _.
# This call will run the Afrikaans POS tagger on the input text 'Hierdie is ''n voorbeeldsin om die funksionaliteit te toets.' and will return a delimited output.
output_process = server.process_text(text_input='Hierdie is \'n voorbeeldsin om die funksionaliteit te toets.', language='af', tech='pos', output_format="delimited", delimiter="|")
print(output_process)
Output examples:
# JSON
[{'doc': {'p': {'lid': 'NONE', 'tokenised': True, 'sent': {'tokens': [{'start_char': 0, 'pos': 'PA', 'end_char': 7, 'id': 1, 'text': 'Hierdie'}, {'start_char': 8, 'pos': 'VTHOK', 'end_char': 10, 'id': 2, 'text': 'is'}, {'start_char': 11, 'pos': 'LO', 'end_char': 13, 'id': 3, 'text': "'n"}, {'start_char': 14, 'pos': 'NSE', 'end_char': 26, 'id': 4, 'text': 'voorbeeldsin'}, {'start_char': 27, 'pos': 'SVS', 'end_char': 29, 'id': 5, 'text': 'om'}, {'start_char': 30, 'pos': 'LB', 'end_char': 33, 'id': 6, 'text': 'die'}, {'start_char': 34, 'pos': 'NSE', 'end_char': 49, 'id': 7, 'text': 'funksionaliteit'}, {'start_char': 50, 'pos': 'UPI', 'end_char': 52, 'id': 8, 'text': 'te'}, {'start_char': 53, 'pos': 'VTHOG', 'end_char': 58, 'id': 9, 'text': 'toets'}, {'start_char': 58, 'pos': 'ZE', 'end_char': 59, 'id': 10, 'text': '.'}]}}}}]
# List
[('Hierdie', 'PA'), ('is', 'VTHOK'), ("'n", 'LO'), ('voorbeeldsin', 'NSE'), ('om', 'SVS'), ('die', 'LB'), ('funksionaliteit', 'NSE'), ('te', 'UPI'), ('toets', 'VTHOG'), ('.', 'ZE')]
# Delimited
['Hierdie|PA', 'is|VTHOK', "'n|LO", 'voorbeeldsin|NSE', 'om|SVS', 'die|LB', 'funksionaliteit|NSE', 'te|UPI', 'toets|VTHOG', '.|ZE']
Testing
The ctextcore package uses pytest version 8.0.0 or above as a testing framework and is a required prerequisite to be able to run the unit tests of the package.
Running all the unit tests of the ctextcore package
py -m pytest --pyargs ctextcore.tests
Running individual unit tests of the ctextcore package
The ctextcore package contains the following unit tests:
- zalid
- ner
- ocr
- pc
- pos
- upos
- sent
- tok
- morph
- lemma
Running an individual unit test
py -m pytest --pyargs ctextcore.tests.test_name
Example
py -m pytest --pyargs ctextcore.tests.test_zalid
License
Licensed under the Apache License, Version 2.0. See LICENSE.txt for more information.
Contact
Centre for Text Technology (CTexT) - ctext@nwu.ac.za - https://humanities.nwu.ac.za/ctext
Metadata
Release files for ctextcore 1.0.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| ctextcore-1.0.1.tar.gz | 93.3 MB | Without provenance | Details for ctextcore-1.0.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| ctextcore-1.0.1-py3-none-any.whl | Python 3 | none | any | Details for ctextcore-1.0.1-py3-none-any.whl |
Total release size: 187.0 MB
Release files / ctextcore-1.0.1.tar.gz
| Download URL | ctextcore-1.0.1.tar.gz |
|---|---|
| Size | 93.3 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
dc498c39ec8c8598ad1a78cafbc0f2a02443cb7943cbd3da233d2d0cfa6407a1
|
|
BLAKE2b-256 checksum How to use checksums |
1076d95cf910a87188c9fff6062ce9dffccbe78b04fa29c324fb29f98f8dc248
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.4
|
Release files / ctextcore-1.0.1-py3-none-any.whl
| Download URL | ctextcore-1.0.1-py3-none-any.whl |
|---|---|
| Size | 93.6 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9de56f13dd9e280c31bf5e927d0f1c6223d20bc28ff66985cc7b92d54b08c48b
|
|
BLAKE2b-256 checksum How to use checksums |
fb5c0db7c0ca7e9bcfa01420a641c156a342e058c6cf442f39b0989f20e0366f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.4
|
Security for ctextcore 1.0.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.