nlcodec is a collection of encoding schemes for natural language sequences. nlcodec.db is a efficient storage and retrieval layer for integer sequences of varying lengths.
Project description
NLCodec
📕 Docs: https://isi-nlp.github.io/nlcodec
A set of (low level) Natural Language Encoder-Decoders (codecs), that are useful in preprocessing stage of NLP pipeline. These codecs include encoding of sequences into one of the following:
- Character
- Word
- BPE based subword
- Class
It provides python (so embed into your app) and CLI APIs (use it as stand alone tool).
There are many BPE implementations available already, but this one provides differs:
- Pure python implementation that is easy to modify anything to try new ideas. (other implementations require c++/rust expertise to modify the core)
- An easily shareable and inspectable model file. It is a simple text that can be inspected with
less
orcut
. It includes info on which pieces were put together and what frequencies etc. - Reasonably faster than the other pure python implementations. Under the hood tries, doubly linked lists, max-heaps, hash maps etc data-structures to boost performance.
- PySpark backend for extracting term frequencies from large datasets.
Installation
Please run only one of these
# Install from pypi (preferred)
$ pip install nlcodec --ignore-installed
# Clone repo for development mode
git clone https://github.com/isi-nlp/nlcodec
cd nlcodec
pip install --editable .
pip installer registers these CLI tools in your PATH:
nlcodec
-- CLI for learn, encode, decode. Same aspython -m nlcodec
nlcodec-learn
-- CLI for learn BPE, with PySpark backend. Same aspython -m nlcodec.learn
nlcodec-db
-- CLI for bitextdb.python -m nlcodec.bitextdb
nlcodec-freq
-- CLI for extracting word and char frequencies using spark backend.
Docs are available at
- HTML format: https://isi-nlp.github.io/nlcodec (recommended)
- Locally at docs/intro.adoc
Citation
Refer to https://arxiv.org/abs/2104.00290 To-appear: ACL 2021 Demos
@article{DBLP:journals/corr/abs-2104-00290,
author = {Thamme Gowda and
Zhao Zhang and
Chris A. Mattmann and
Jonathan May},
title = {Many-to-English Machine Translation Tools, Data, and Pretrained Models},
journal = {CoRR},
volume = {abs/2104.00290},
year = {2021},
url = {https://arxiv.org/abs/2104.00290},
archivePrefix = {arXiv},
eprint = {2104.00290},
timestamp = {Mon, 12 Apr 2021 16:14:56 +0200},
biburl = {https://dblp.org/rec/journals/corr/abs-2104-00290.bib},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
Authors
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file nlcodec-0.5.tar.gz
.
File metadata
- Download URL: nlcodec-0.5.tar.gz
- Upload date:
- Size: 41.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.2 importlib_metadata/4.6.4 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.1 CPython/3.9.6
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | f25f1f6486032bedee76ce96a8178a61c5202e366c4d65b968c4311408f291b8 |
|
MD5 | 474f5413b982fdb31e3dbc056cfcca14 |
|
BLAKE2b-256 | 342e9d4b9f433577d503e62c82d96c9c8a17593f440f8b3c268305d75783070e |
File details
Details for the file nlcodec-0.5-py3-none-any.whl
.
File metadata
- Download URL: nlcodec-0.5-py3-none-any.whl
- Upload date:
- Size: 53.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.2 importlib_metadata/4.6.4 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.1 CPython/3.9.6
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6c906d89f0d7d7ffcc3a444d3a85dfc5838c849420da901425b653aad3e435fa |
|
MD5 | 83a44081ed13a51d9c9d8fb4cc6d3860 |
|
BLAKE2b-256 | 09a990c8f874ae72c93b9320751811d1233bf0970f0de13fbae1a59cc37c7646 |