Python package for measuring distance between the lects represented by small raw corpora

What is it?

corpus_distance is a Python package that allows to measure distance between the lects that are presented only by small (down to extremely small, <1000 tokens) raw (without any kind of morhological tagging, lemmatisation, or dependency parsing) corpora, and classify them. It joins frequency-based metrics and string similarity measurements into a hybrid distance scorer.

corpus_distance operates with 3-shingles, a sequences of 3 symbols, by which words are split (Zelenkov and Segalovich, 2007). This helps to spot more intricate patterns and correspondences within raw data, as well as to enhance the dataset size.

NB!

The classification is going to be only (and extremely) preliminary, as it is by default language-agnostic and does not use preliminary expert judgements or linguistic information. Basically, the more effort is put into the actual data, the more reliable are final results.

In addition, the results may not be used as a proof of language relationship (external classification), only as a supporting evidence for a tree topology (internal classification), as it is with any kind of phylogenetic methods in historical comparative studies.

One more important notion is that one should be very careful with using this package for a distantly related lects. As it is with any kind of language-agnostic methods, it loses precision with the increase of distance between analysed groups (Holman et al., 2008).

How to install

Version warning

By now, corpus_distance is not available for Python>=3.14. Use 3.12 for the smoothest possible experience.

From TestPyPI (development version; requires manual installation of dependencies; may contain bugs)

pip install biopython
pip install Levenshtein
pip install gensim
pip install pyjarowinkler

python3 -m pip install --index-url https://test.pypi.org/simple/ --no-deps --force-reinstall corpus_distance

From PyPI (release version)

pip install corpus_distance

From Docker

1. Clone repository.

git clone https://github.com/The-One-Who-Speaks-and-Depicts/corpus_distance.git

2. Cd to the repository root directory.
3. (optionally) Add your data and configuration file (see How to use/Preparation below), put both into the repository directory.
4. Create Docker image.

sudo docker build --tag IMAGE_NAME .

IMAGE_NAME is a name of the docker image of your choice 5. Run Docker image.

sudo docker run -i -t IMAGE_NAME /bin/bash

How to use

Preparation

1. Create a virtual environment with python3 -m venv ENV, where ENV is a name of your virtual environment.
2. Install package, following the instructions above.
3. Create a folder for your data.
4. Put files with your texts into this folder. Texts should be tokenised manually, and joined into a single string afterwards. The name of the files with texts should be of the following format: TEXT.LECT.EXTENSION, where:
   • EXTENSION is preferrably .txt, as package works with files with raw text data;
   • LECT is a name of the lect (idiolect, dialect, sociolect, regiolect, standard, etc.; any given variety of the language, such as English, or Polish, or Khislavichi, or Napoleon's French) that is the object of the classification
   • TEXT is a unique identifier of the text within a given lect (for instance, NapoleonSpeech1, or John_Gospel)
5. Set up a configuration .json file (the example is in the repository). The parameters are:
   • metrics_name: a user-defined name of the metrics
   • store_path: a path to the folder for results storage
   • content_path: a path to the data folder
   • split: a share of tokens from your files that would be taken into consideration (useful for exploring size effects)
   • topic_modelling: model may delete topic words, if this flag has value substitute, or not, if value is not_substitute. This heuristic helps to exclude the words that define the text, on the contrary to the ones that define the language. Alternatively, if the value is topic_words_only, the model will delete all non-topic words.
   • lda_params: a set of parameters for a Latent Dirichlet Association model from gensim package and two custom parameters, required_topics_start and required_topics_num, used to regulate, the tokens from which topics the model will utilise further. required_topics_start is the first topic (defaults to 0), required_topics_num is the number of topics after required_topics_start that model will collect (defaults to 10).
   • fasttext_params: a set of parameters for a FastText model that provides the classifier with the symbol embeddings
   • soerensen: normalisation of frequency-based metrics by the Soerensen-Dice coefficient
   • hybridisation: flag for use (or not use) of string similarity measure for non-coinciding 3-shingles
   • hybridisation_as_array: regulates the way of hybridisation: either frequency-based metrics and string similarity measures values are taken as a single array, for which the mean score is counted, or they are taken separately, and their means are multiplied by each other. soerensen normalisation applies only when this parameter has false value.
   • metrics: a particular string similarity measure. May be user defined, defaults are corpus_distance.distance_measurement.string_similarity.levenshtein_wrapper (simple edit distance), corpus_distance.distance_measurement.string_similarity.weighted_jaro_winkler_wrapper(edit distance, weighted by Jaro-Winkler distance), corpus_distance.distance_measurement.string_similarity.vector_measure_wrapper (counting differences by euclidean distance between vectors of symbols), and corpus_distance.distance_measurement.string_similarity.jaro_vector_wrapper (counting differences by euclidean distance between vectors of symbols, weighted by Jaro distance, in order to count for order)
   • alphabet_normalisation: a normalisation of vector-based metrics by difference of alphabet entropy between given lects
   • data_name: name of the dataset for visualisation (for example. South Slavic)
   • outgroup: name of the lect that is the farthest from the others
   • metrics: name of the metrics combination, by default containing all the given parameters
   • classification_method: classification method for building tree, upgma or nj: either Unweighted Pair Group Method with Arithmetic Mean, or Neighbourhood-Joining
   • store_path: the same as store path on the top.
6. The example of the config.json:

    {
        "metrics_name": "default_metrics_name",
        "data": {
            "dataset_params": {
                "store_path": "default",
                "split": 1,
                "content_path": "default",
                "topic_modelling": "not_substitute"
            },
            "lda_params": {
                "num_topics": 10,
                "alpha": "auto",
                "epochs": 300,
                "passes": 500,
                "random_state": 1,
                "required_topics_start": 0,
                "required_topics_num": 10
            },
            "fasttext_params": {
                "vector_size": 128,
                "window": 15,
                "min_count": 3,
                "workers": 1,
                "epochs": 300,
                "seed": 42,
                "sg": 1
            }
        },
        "hybridisation_parameters": {
            "soerensen": true,
            "hybridisation": true,
            "hybridisation_as_array": true,
            "metrics": "corpus_distance.distance_measurement.string_similarity.jaro_vector_wrapper",
            "alphabet_normalisation": true
        },
        "clusterisation_parameters": {
            "data_name": "Modern Standard Slavic",
            "outgroup": "Slovak",
            "metrics": "default_metrics_name",
            "classification_method": "upgma"
        }
    }

7. In case you are using Docker, do not forget to put your data and configuration file into the repository directory before creating Docker image.

Running the code

There are two ways of running the code: with prepared Jupyter Notebook, or independently.

Ready-made Jupyter Notebook

In the folder example, there is a tutorial notebook that outlines the inner workings of the package.

Using your own file (or from Docker console)

After data and configuration are ready, open Python interpreter:

python

Run the following commands:

from corpus_distance.pipeline import perform_clusterisation
perform_clusterisation(PATH_TO_CONFIG)

PATH_TO_CONFIG here is a path to config.json. This parameter may be empty, then the model performs clusterisation with default dataset (Modern Standard Slavic Gospels of John: Croatian, Slovak, Slovenian) and default parameters.