Distributional Semantics Toolkit

This library implements many of the algorithms described in Distributional Semantics by Alessandro Lenci and Magnus Sahlgren, providing reusable building blocks for count-based and predictive distributional semantic models.

Table of Contents

1. Documentation
2. Installation
3. Usage
4. Status
5. Contributing
6. License

Documentation

You can find the (temporal) basic documentation here. You can also find a demonstration of how to use it on Google Colab (in spanish) here.

Installation

To install it just run the command:

pip install dstklib

DSTK requires python <3.14 to work.

Usage

Individual methods

You can use the included methods individually to do linguistic analysis. Just import the method you want to use from its respective module:

from dstk.corpus.annotation import annotate_corpus
from dstk.parameters.context.selection.unit import get_words
from dstk.parameters.context.selection.lexical import remove_stop_words, to_base_form, to_lower
from dstk.parameters.context.extraction.linguistic.word.window import extract_ngrams
from dstk.parameters.co_matrix.creation.linguistic.word.window import create_word_by_word_matrix
from dstk.parameters.co_matrix.weighting.associative_measures import pmi
from dstk.parameters.dimensionality_reduction import svd
from dstk.parameters.vector_similarity.geometric_measures.similarity import cos_similarity

corpus = ["El rápido zorro marrón salta sobre el perro perezoso. ¿Por qué el zorro no corre más rápido?"]

documents = annotate_corpus(corpus=corpus, language_model="es")

words = get_words(document=documents["document_0"])
filtered_words = remove_stop_words(words=words, language="es")
lemmas = to_base_form(words=filtered_words)
lowered = to_lower(words=lemmas)

contexts = extract_ngrams(words=lowered, window_size=3)

co_matrix = create_word_by_word_matrix(contexts=contexts)
weighted_matrix = pmi(word_by_word_matrix=co_matrix, positive=True)
embeddings = svd(matrix=weighted_matrix, n_dimensions=2)

similarity = cos_similarity(embeddings=embeddings, first_word="zorro", second_word="marrón")

print(similarity)

# Output: 0.22279958362756935

Models

DSTK has some models included that already cover most of the frequent tasks in distributional semantics:

• StandardModel: This model generates word embeddings using the Standard Model as defined by (Lenci & Sahlgren 97-99). It extracts word co-occurrences within a context window, weights the matrix using positive PMI (PPMI), reduces its dimensionality with SVD, and provides cosine-based similarity measures.

• LatentSemanticAnalysis: This model generates word embeddings using Latent Semantic Analysis (LSA) as defined by (Lenci & Sahlgren 100-103). The model builds a word-document matrix, applies TF-IDF weighting, reduces dimensionality with SVD, and provides cosine-based similarity measures.

• SGNS: This model generates word embeddings using Skip-Gram with Negative Sampling (SGNS) as defined by (Lenci & Sahlgren 162-163). The document is optionally normalized (lowercasing, lemmatization/stemming, POS filtering, stop-word removal) before training a Word2Vec model. The resulting embeddings can be explored through cosine similarity and nearest-neighbor methods.

• Fasttext: This model generates word embeddings using using FastText as defined by (Lenci & Sahlgren 164-165). The document is optionally normalized (lowercasing, lemmatization/stemming, POS filtering, stop-word removal) before training a FastText model. Subword information is used to improve representations of rare and out-of-vocabulary words.

In order to use them just import the respective model and pass as an input an annotated document(s):

from dstk.corpus.annotation import annotate_corpus
from dstk.models.count.matrix.classical import StandardModel
from dstk.parameters.vector_similarity.geometric_measures import similarity

corpus = ["El rápido zorro marrón salta sobre el perro perezoso. ¿Por qué el zorro no corre más rápido?"]

documents = annotate_corpus(corpus=corpus, language_model="es")

distance = StandardModel(
    document=documents["document_0"], 
    language="es", 
    frequency_threshold=1, 
    n_dimensions=2
)
similarity = distance.cos_similarity(first_word="zorro", second_word="marrón")

print(similarity)

# Output: 0.22279958362756935

Building your own models

You can build your own models using the class ModelBuilder and passing it a custom workflow. Just input the name of the methods (in the correct order) you want to use and their corresponding arguments as a list of dictionaries, along with the name of the module from dstk.parameters you are importing them from:

from dstk.corpus.annotation import annotate_corpus
from dstk.models.tools import ModelBuilder

corpus = ["El rápido zorro marrón salta sobre el perro perezoso. ¿Por qué el zorro no corre más rápido?"]

documents = annotate_corpus(corpus=corpus, language_model="es")

MyModel = ModelBuilder(
    workflow={
            "context.selection.unit": [
                {"get_words": {}}, # IMPORTANT: The first input is passed automatically; only specify the remaining arguments.
            ],
            "context.selection.lexical": [
                {"to_lower": {}},
                {"to_base_form": {"base_form": "lemma"}},
                {"remove_stop_words": {"language": "es"}},
            ],
            "context.extraction.linguistic.word.window": [
                {"extract_ngrams": {"window_size": 3}}
            ],
            "co_matrix.creation.linguistic.word.window": [
                {"create_word_by_word_matrix": {}}
            ],
            "co_matrix.weighting.associative_measures": [{"pmi": {"positive": True}}],
            "dimensionality_reduction": [{"svd": {"n_dimensions": 2}}],
            "vector_similarity.geometric_measures.similarity": [
                {"cos_similarity": {"first_word": "zorro", "second_word": "marrón"}},
            ],
        }
)

similarity = MyModel(input_data=documents["document_0"])

print(similarity)

# Output: 0.22279958362756935

You can also get specific results in the workflow or even all of them by using return_parameters and return all:

contexts, similarity = MyModel(
    input_data=documents["document_0"], 
    return_parameters=[
            "context.selection.lexical", 
            "vector_similarity.geometric_measures.similarity"
        ]
    )

print([word.text for word in contexts])
print(similarity)

# Output: 
# ['rápido', 'zorro', 'marrón', 'saltar', 'perro', 'perezoso', 'zorro', 'correr', 'rápido']
# 0.22279958362756935

If you choose to return all of the results, the workflow will return a generator with a tuple containing the name of the method and its result:

results = MyModel(input_data=documents["document_0"], return_all=True)

first_param = next(results)

print(first_param.name)
print([word.text for word in first_param.result])

# Output: 
# context.selection.unit
# ['El', 'rápido', 'zorro', 'marrón', 'salta', 'sobre', 'el', 'perro', 'perezoso', 'Por', 'qué', 'el', 'zorro', 'no', 'corre', 'más', 'rápido']

You can also make a model return a Wrapper class containing methods you might want to use multiple times. For example, if you wish to calculate the cos_similarity of different words in the same embeddings, you can do:

MyModel = ModelBuilder(
    workflow={
            "context.selection.unit": [
                {"get_words": {}},
            ],
            "context.selection.lexical": [
                {"to_lower": {}},
                {"to_base_form": {"base_form": "lemma"}},
                {"remove_stop_words": {"language": "es"}},
            ],
            "context.extraction.linguistic.word.window": [
                {"extract_ngrams": {"window_size": 3}}
            ],
            "co_matrix.creation.linguistic.word.window": [
                {"create_word_by_word_matrix": {}}
            ],
            "co_matrix.weighting.associative_measures": [{"pmi": {"positive": True}}],
            "dimensionality_reduction": [{"svd": {"n_dimensions": 2}}],
            "vector_similarity.geometric_measures.similarity": [
                {"cos_similarity": {}} # The methods should NOT have args
            ]
        },
    wrapper=True
)

distance = MyModel(input_data=documents["document_0"])

similarity = distance.cos_similarity(first_word="zorro", second_word="marrón")

# Note: The original functions normally require the input data as their first argument.
# This wrapper class stores that input internally,
# so when calling methods on the wrapper instance, you only need to provide the additional parameters.
# This pattern works with any method from any module and any type of input data,
# allowing convenient repeated use without passing the main input every time.
print(distance.cos_similarity(first_word="zorro", second_word="marrón"))
print(distance.cos_similarity(first_word="perro", second_word="perezoso"))

# Output: 
# 0.22279958362756935
# 0.9688325516828031

Hooks

You can add hooks (functions with custom logic) to a model. You must only follow two rules:

1. The function must accept only one positional argument return a single result
2. The type of its input must be the same as the one returned from the previous workflow. Also, the type it returns must match the input of the next workflow.

Following these rules you can insert your custom hooks this way:

from dstk.corpus.annotation import annotate_corpus
from dstk.models.tools import ModelBuilder
from dstk.hooks.tools import Hook

corpus = ["El rápido zorro marrón salta sobre el perro perezoso. ¿Por qué el zorro no corre más rápido?"]

documents = annotate_corpus(corpus=corpus, language_model="es")

def cool_method(words):
    for word in words:
        word.text = "cool"
    
    return words
    
CoolHook = Hook(method=cool_method)

MyModel = ModelBuilder(
    workflow={
            "context.selection.unit": [
                {"get_words": {}},
            ],
            "context.selection.lexical": [
                {"to_lower": {}},
                {"to_base_form": {"base_form": "lemma"}},
                {"remove_stop_words": {"language": "es"}},
            ],
            "cool_hook": CoolHook
        }
)

words = MyModel(input_data=documents["document_0"])

print([word.text for word in words])

# Output: ['cool', 'cool', 'cool', 'cool', 'cool', 'cool', 'cool', 'cool', 'cool']

If your custom hook has multiple keyword arguments, you can pass them this way:

from dstk.corpus.annotation import annotate_corpus
from dstk.models.tools import ModelBuilder
from dstk.hooks.tools import Hook

corpus = ["El rápido zorro marrón salta sobre el perro perezoso. ¿Por qué el zorro no corre más rápido?"]

documents = annotate_corpus(corpus=corpus, language_model="es")

def cool_method(words, text="cool"):
    for word in words:
        word.text = text
    
    return words
    
CoolHook = Hook(method=cool_method)
CoolHook.set_default_args({"text": "custom_text"})

MyModel = ModelBuilder(
    workflow={
            "context.selection.unit": [
                {"get_words": {}},
            ],
            "context.selection.lexical": [
                {"to_lower": {}},
                {"to_base_form": {"base_form": "lemma"}},
                {"remove_stop_words": {"language": "es"}},
            ],
            "cool_hook": CoolHook
        }
)

words = MyModel(input_data=documents["document_0"])

print([word.text for word in words])

# Output: ['custom_text', 'custom_text', 'custom_text', 'custom_text', 'custom_text', 'custom_text', 'custom_text', 'custom_text', 'custom_text']

Status

This release (v3.0.2) is mostly stable, but still considered beta.

• Most algorithms are stable and rely on well-tested libraries. However, some custom implementations might need verification.
• Performance may be limited for large datasets.
• All models should deliver correct results. However, external verification is strongly encouraged.

This library is open-source, and your verification matters! If you use this library:

• Try the models on your data.
• Compare results with expected behavior.
• Report any inconsistencies, crashes, or performance bottlenecks.

Even small confirmations help a lot. Your feedback will help make the next stable release solid and reliable.

Contributing

I welcome contributions to improve this toolkit. If you have ideas or fixes, feel free to fork the repository and submit a pull request. Here are some ways you can help:

• Report bugs or issues.

• Suggest new features or algorithms to add.

License

This project is licensed under the GPL-3 License - see the LICENSE file for details.