Skip to main content

nlcodec is a collection of encoding schemes for natural language sequences. nlcodec.db is a efficient storage and retrieval layer for integer sequences of varying lengths.

Project description


NOTE: The docs are available at

A set of (low level) Natural Language Encoder-Decoders (codecs), that are useful in preprocessing stage of NLP pipeline. These codecs include encoding of sequences into one of the following:

  1. Character
  2. Word
  3. BPE based subword

It provides python (so embed into your app) and CLI APIs (use it as stand alone tool).

There are many BPE implementations available already, but this one provides differs:

  1. Pure python implementation that is easy to modify anything to try new ideas. (other implementations require c++/rust expertise to modify the core)
  2. An easily shareable and inspectable model file. It is a simple text that can be inspected with less or cut. It includes info on which pieces were put together and what frequencies etc.
  3. Reasonably faster than the other pure python implementations. Under the hood tries, doubly linked lists, max-heaps, hash maps etc data-structures to boost performance.
  4. PySpark backend for extracting term frequencies from large datasets.


Please run only one of these

# Install from pypi (preferred)
$ pip install nlcodec --ignore-installed 

# Clone repo for development mode 
git clone
cd nlcodec
pip install --editable . 

pip installer registers these CLI tools in your PATH:

  • nlcodec -- CLI for learn, encode, decode. Same as python -m nlcodec
  • nlcodec-learn -- CLI for learn BPE, with PySpark backend. Same as python -m nlcodec.learn
  • nlcodec-db -- CLI for bitextdb. python -m nlcodec.bitextdb
  • nlcodec-freq -- CLI for extracting word and char frequencies using spark backend.

Docs are available at


Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for nlcodec, version 0.3.1
Filename, size File type Python version Upload date Hashes
Filename, size nlcodec-0.3.1-py3-none-any.whl (51.3 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size nlcodec-0.3.1.tar.gz (37.8 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page