Skip to main content

Extraction de LExique par Variation d'Entropie - Lexicon extraction based on the variation of entropy

Project description

What is ELeVE ?

ELeVE is a library intended for computing an “autonomy estimate” score for substrings (all n-grams) in a corpus of text.

The autonomy score is based on normalised variation of branching entropies (nVBE) of strings, See [MagistrySagot2012] for a definiton of these terms

It was developed mainly for unsupervised segmentation of mandarin Chinese, but is language independant and was successfully used in research on other tasks like keyphrase extraction.

Full documentation is available on

In a nutshell

Here is a simple “getting started”. First you have to train a model:

>>> from eleve import MemoryStorage
>>> storage = MemoryStorage()
>>> # Then the training itself:
>>> storage.add_sentence(["I", "like", "New", "York", "city"])
>>> storage.add_sentence(["I", "like", "potatoes"])
>>> storage.add_sentence(["potatoes", "are", "fine"])
>>> storage.add_sentence(["New", "York", "is", "a", "fine", "city"])

And then you cat query it:

>>> storage.query_autonomy(["New", "York"])
>>> storage.query_autonomy(["like", "potatoes"])

Eleve also store n-gram’s occurence count:

>>> storage.query_count(["New", "York"])
>>> storage.query_count(["New", "potatoes"])
>>> storage.query_count(["I", "like", "potatoes"])
>>> storage.query_count(["potatoes"])

Then, you can use it for segmentation, using an algorithm that look for the solution which maximize nVBE of resulting words:

>>> from eleve import Segmenter
>>> s = Segmenter(storage)
>>> # segment up to 4-grams, if we used the same storage as before.
>>> s.segment(["What", "do", "you", "know", "about", "New", "York"])
[['What'], ['do'], ['you'], ['know'], ['about'], ['New', 'York']]


You will need some dependencies. On Ubuntu:

$ sudo apt-get install python3-dev libboost-python-dev libboost-filesystem-dev libleveldb-dev

Then to install eleve:

$ pip install eleve

or if you have a local clone of source folder:

$ python install

Get the source

Source are stored on github:

$ git clone


Install the development environment:

$ git clone
$ cd eleve
$ virtualenv ENV -p /usr/bin/python3
$ source ENV/bin/activate
$ pip install -r requirements.txt
$ pip install -r

Pull requests are welcome!

To run tests:

$ make testall

To build the doc:

$ make doc

then open: docs/_build/html/index.html

Warning: You need to have eleve accesible in the python path to run tests (and to build doc). For that you can install eleve as a link in local virtualenv:

$ pip install -e .

(Note: this is indicated in pytest good practice )


If you use eleve for an academic publication, please cite this paper:

[MagistrySagot2012]Magistry, P., & Sagot, B. (2012, July). Unsupervized word segmentation: the case for mandarin chinese. In Proceedings of the 50th Annual Meeting of the ACL: Short Papers-Volume 2 (pp. 383-387).

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for eleve, version 20.10.8
Filename, size File type Python version Upload date Hashes
Filename, size eleve-20.10.8.tar.gz (70.0 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page