Skip to main content

Language identification Toolkit

Project description

DOI PyPI version Python Support Build Status Code style: black GitHub last commit GitHub commits since latest release (by SemVer) CodeFactor


lidtk - the language identification toolkit - was written in order to investigate the current state of language performance.


The recommended way to install clana is:

$ pip install lidtk --user

If you want the latest version:

$ git clone; cd lidtk
$ pip install -e . --user

I recommend getting the WiLI-2018 dataset.


$ lidtk --help

Usage: lidtk [OPTIONS] COMMAND [ARGS]...

  --version  Show the version and exit.
  --help     Show this message and exit.

  analyze-data           Utility function for the languages...
  analyze-unicode-block  Analyze how important a Unicode block is for...
  char-distrib           Use the character distribution language...
  cld2                   Use the CLD-2 language classifier.
  create-dataset         Create sharable dataset from downloaded...
  download               Download 1000 documents of each language.
  google-cloud           Use the CLD-2 language classifier.
  langdetect             Use the langdetect language classifier.
  langid                 Use the langid language classifier.
  map                    Map predictions to something known by WiLI
  nn                     Use a neural network classifier.
  textcat                Use the CLD-2 language classifier.
  tfidf_nn               Use the TfidfNNClassifier classifier.

For example:

$ lidtk cld2 predict --text 'This is a test.'

The usual order is:

  1. lidtk download: Please use WiLI-2018 instead of downloading the dataset on your own.
  2. lidtk create-dataset: This step can be skipped if you use WiLI-2018
  3. lidtk analyze-unicode-block --start 0 --end 128
  4. lidtk tfidf_nn train vectorizer --config lidtk/classifiers/config/tfidf_nn.yaml
  5. lidtk tfidf_nn train vectorizer --config lidtk/classifiers/config/tfidf_nn.yaml
  6. lidtk tfidf_nn wili --config lidtk/classifiers/config/tfidf_nn.yaml

Or to use one directly:

$ lidtk cld2 predict --text 'This text is written in some language.'



Check tests with tox.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

lidtk-0.3.0.tar.gz (38.7 kB view hashes)

Uploaded source

Built Distribution

lidtk-0.3.0-py3-none-any.whl (54.7 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page