eleve 20.10.5

Extraction de LExique par Variation d'Entropie - Lexicon extraction based on the variation of entropy

What is ELeVE ?

ELeVE is a library intended for computing an “autonomy estimate” score for substrings (all n-grams) in a corpus of text.

The autonomy score is based on normalised variation of branching entropies (nVBE) of strings, See [MagistrySagot2012] for a definiton of these terms

It was developed mainly for unsupervised segmentation of mandarin Chinese, but is language independant and was successfully used in research on other tasks like keyphrase extraction.

Full documentation is available on http://pythonhosted.org/eleve/.

In a nutshell

Here is a simple “getting started”. First you have to train a model:

>>> from eleve import MemoryStorage
>>>
>>> storage = MemoryStorage()
>>>
>>> # Then the training itself:
>>> storage.add_sentence(["I", "like", "New", "York", "city"])
>>> storage.add_sentence(["I", "like", "potatoes"])
>>> storage.add_sentence(["potatoes", "are", "fine"])
>>> storage.add_sentence(["New", "York", "is", "a", "fine", "city"])

And then you cat query it:

>>> storage.query_autonomy(["New", "York"])
2.0369977951049805
>>> storage.query_autonomy(["like", "potatoes"])
-0.3227022886276245

Eleve also store n-gram’s occurence count:

>>> storage.query_count(["New", "York"])
2
>>> storage.query_count(["New", "potatoes"])
0
>>> storage.query_count(["I", "like", "potatoes"])
1
>>> storage.query_count(["potatoes"])
2

Then, you can use it for segmentation, using an algorithm that look for the solution which maximize nVBE of resulting words:

>>> from eleve import Segmenter
>>> s = Segmenter(storage)
>>> # segment up to 4-grams, if we used the same storage as before.
>>>
>>> s.segment(["What", "do", "you", "know", "about", "New", "York"])
[['What'], ['do'], ['you'], ['know'], ['about'], ['New', 'York']]

Installation

You will need some dependencies. On Ubuntu:

$ sudo apt-get install python3-dev libboost-python-dev libboost-filesystem-dev libleveldb-dev

Then to install eleve:

$ pip install eleve

or if you have a local clone of source folder:

$ python setup.py install

Get the source

Source are stored on github:

$ git clone https://github.com/kodexlab/eleve

Contribute

Install the development environment:

$ git clone https://github.com/kodexlab/eleve
$ cd eleve
$ virtualenv ENV -p /usr/bin/python3
$ source ENV/bin/activate
$ pip install -r requirements.txt
$ pip install -r requirements.dev.txt

Pull requests are welcome!

To run tests:

$ make testall

To build the doc:

$ make doc

then open: docs/_build/html/index.html

Warning: You need to have eleve accesible in the python path to run tests (and to build doc). For that you can install eleve as a link in local virtualenv:

$ pip install -e .

(Note: this is indicated in pytest good practice )

References

If you use eleve for an academic publication, please cite this paper:

[MagistrySagot2012]

Magistry, P., & Sagot, B. (2012, July). Unsupervized word segmentation: the case for mandarin chinese. In Proceedings of the 50th Annual Meeting of the ACL: Short Papers-Volume 2 (pp. 383-387). http://www.aclweb.org/anthology/P12-2075

Copyright, license and authors

Copyright (C) 2014-2015 Kodex⋅Lab., 2020 Pierre Magistry

eleve is available under the LGPL Version 3 license.

eleve was originaly designed and prototyped by Pierre Magistry during its PhD. It then has been completly rewriten by Korantin Auguste and Emmanuel Navarro (with the help of Pierre).

For the sake of easier installation and evolution, it was later refactored in Cython by Pierre Magistry.