Skip to main content

nlcodec is a collection of encoding schemes for natural language sequences. nlcodec.db is a efficient storage and retrieval layer for integer sequences of varying lengths.

Project description

NLCodec

image Travis (.com)

📕 Docs: https://isi-nlp.github.io/nlcodec

A set of (low level) Natural Language Encoder-Decoders (codecs), that are useful in preprocessing stage of NLP pipeline. These codecs include encoding of sequences into one of the following:

  1. Character
  2. Word
  3. BPE based subword
  4. Class

It provides python (so embed into your app) and CLI APIs (use it as stand alone tool).

There are many BPE implementations available already, but this one provides differs:

  1. Pure python implementation that is easy to modify anything to try new ideas. (other implementations require c++/rust expertise to modify the core)
  2. An easily shareable and inspectable model file. It is a simple text that can be inspected with less or cut. It includes info on which pieces were put together and what frequencies etc.
  3. Reasonably faster than the other pure python implementations. Under the hood tries, doubly linked lists, max-heaps, hash maps etc data-structures to boost performance.
  4. PySpark backend for extracting term frequencies from large datasets.

Installation

Please run only one of these

# Install from pypi (preferred)
$ pip install nlcodec --ignore-installed 

# Clone repo for development mode 
git clone https://github.com/isi-nlp/nlcodec
cd nlcodec
pip install --editable . 

pip installer registers these CLI tools in your PATH:

  • nlcodec -- CLI for learn, encode, decode. Same as python -m nlcodec
  • nlcodec-learn -- CLI for learn BPE, with PySpark backend. Same as python -m nlcodec.learn
  • nlcodec-db -- CLI for bitextdb. python -m nlcodec.bitextdb
  • nlcodec-freq -- CLI for extracting word and char frequencies using spark backend.

Docs are available at

Citation

Refer to https://arxiv.org/abs/2104.00290 To-appear: ACL 2021 Demos

@article{DBLP:journals/corr/abs-2104-00290,
  author    = {Thamme Gowda and
               Zhao Zhang and
               Chris A. Mattmann and
               Jonathan May},
  title     = {Many-to-English Machine Translation Tools, Data, and Pretrained Models},
  journal   = {CoRR},
  volume    = {abs/2104.00290},
  year      = {2021},
  url       = {https://arxiv.org/abs/2104.00290},
  archivePrefix = {arXiv},
  eprint    = {2104.00290},
  timestamp = {Mon, 12 Apr 2021 16:14:56 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/abs-2104-00290.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}

Authors

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

nlcodec-0.5.tar.gz (41.7 kB view details)

Uploaded Source

Built Distribution

nlcodec-0.5-py3-none-any.whl (53.6 kB view details)

Uploaded Python 3

File details

Details for the file nlcodec-0.5.tar.gz.

File metadata

  • Download URL: nlcodec-0.5.tar.gz
  • Upload date:
  • Size: 41.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.4 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.1 CPython/3.9.6

File hashes

Hashes for nlcodec-0.5.tar.gz
Algorithm Hash digest
SHA256 f25f1f6486032bedee76ce96a8178a61c5202e366c4d65b968c4311408f291b8
MD5 474f5413b982fdb31e3dbc056cfcca14
BLAKE2b-256 342e9d4b9f433577d503e62c82d96c9c8a17593f440f8b3c268305d75783070e

See more details on using hashes here.

File details

Details for the file nlcodec-0.5-py3-none-any.whl.

File metadata

  • Download URL: nlcodec-0.5-py3-none-any.whl
  • Upload date:
  • Size: 53.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.4 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.1 CPython/3.9.6

File hashes

Hashes for nlcodec-0.5-py3-none-any.whl
Algorithm Hash digest
SHA256 6c906d89f0d7d7ffcc3a444d3a85dfc5838c849420da901425b653aad3e435fa
MD5 83a44081ed13a51d9c9d8fb4cc6d3860
BLAKE2b-256 09a990c8f874ae72c93b9320751811d1233bf0970f0de13fbae1a59cc37c7646

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page