generic-iterative-stemmer 0.3.1

A generic language stemming utility, dedicated for gensim word-embedding.

Word embedding: generic iterative stemmer

A generic helper for training gensim and fasttext word embedding models.
Specifically, this repository was created in order to implement stemming on a Wikipedia-based corpus in Hebrew, but it will probably also work for other corpus sources and languages as well.

Important to note that while there are sophisticated and efficient approaches to the stemming task, this repository implements a naive approach with no strict time or memory considerations (more about that in the explanation section).

Based on https://github.com/liorshk/wordembedding-hebrew.

Setup

1. Create a python3 virtual environment.
2. Install dependencies using make install (this will run tests too).

Usage

This section shows the basic flow this repository was designed to perform. It supports more complicated flows as well.

The output of the training process is a StemmedKeyedVectors object (in the form of a .kv file), which inherits the standard gensim.models.KeyedVectors.

1. Under ./data folder, create a directory for your corpus (for example, wiki-he).

2. Download Hebrew (or any other language) dataset from Wikipedia:

   1. Go to wikimedia dumps.
   2. Download hewiki-latest-pages-articles.xml.bz2, and save it under ./data/wiki-he.

3. Create your initial text corpus:

   TODO: create a notebook for that.

4. Train the model:

   TODO: create a notebook for that.

5. Play with your trained model using playground.ipynb.

Generic iterative stemming

TODO: Explain the algorithm.