Skip to main content

hyperhyper CI PyPI PyPI - Python Version PyPI - Downloads

hyperhyper is a Python package to construct word embeddings for small data.

Why?

Nowadays, word embeddings are mostly associated with Word2vec or fastText. These approaches focus on scenarios, where an abundance of data is available. And big players such as Facebook provide ready-to-use pre-trained word embeddings. So often you don't have to train new word embeddings from scratch. But sometimes you do.

Word2vec or fastText require a lot of data – but texts, especially domain-specific texts, may be scarce. There exist alternative methods based on counting co-locations (word pairs) that require fewer data to work. This package implements these approaches (somewhat) efficiently.

Installation

Requires Python >=3.10,<3.14. Python 3.14 is deliberately excluded: no gensim wheel exists for it at any version, and its sdist does not compile against CPython 3.14 (gensim's vendored Cython dereferences a struct field CPython 3.14 removed).

pip install hyperhyper
# or: uv add hyperhyper

To enable all features (pre-processing with spaCy, impl="scikit" for the SVD):

pip install hyperhyper[full]

The spaCy-based preprocessing also needs a language model. hyperhyper downloads it on first use, but in CI or offline environments install it up front:

python -m spacy download en_core_web_sm

Usage

import hyperhyper as hy

# download and uncompress the data
# wget http://www.statmt.org/wmt14/training-monolingual-news-crawl/news.2010.en.shuffled.gz && gzip -d news.2010.en.shuffled.gz
corpus = hy.Corpus.from_file("news.2010.en.shuffled")
bunch = hy.Bunch("news_bunch", corpus)

# `hyperhyper` is built on `gensim`. So you can get word embeddings in a keyed vectors format.
# https://radimrehurek.com/gensim/models/keyedvectors.html
vectors, results = bunch.svd(keyed_vectors=True)

results["results"][1]
>>> {"name": "en_ws353",  # the evaluation dataset
 "score": ...,           # Spearman correlation with the human judgements
 "oov": ...,             # fraction of pairs skipped as out-of-vocabulary
 "fullscore": ...}       # score, penalized by the out-of-vocabulary fraction

vectors.most_similar("berlin")
>>> [("vienna", ...), ("frankfurt", ...), ("munich", ...),
 ("amsterdam", ...), ("stockholm", ...)]

See the usage guide for the full public API — Corpus constructors, bunch.pmi(...) / bunch.svd(...), the evaluation result shape and bunch.results(...) — and check out the runnable examples.

The general concepts:

  • preprocess data once and save them in a bunch
  • cache all results and also record their performance on test data
  • make it easy to fine-tune parameters for your data

For anything the usage guide does not cover, read the source code.

Performance Optimization

BLAS backend

pip install hyperhyper pulls numpy wheels linked against OpenBLAS (Linux/Windows) or Accelerate (recent macOS). Both are fine for typical use.

If you have an Intel CPU and want MKL, install it from conda-forge:

conda install -c conda-forge "libblas=*=*mkl"

Check which backend numpy actually uses — look at the Build Dependencies → blas → name field (mkl, openblas or accelerate):

>>> import numpy
>>> numpy.show_config()

Disable Numerical Multithreading

Further, disable the internal multithreading ability of MKL or OpenBLAS (numerical libraries). This speeds up computation because you should do multiprocessing on an outer loop anyhow. But you can also leave the default to take advantage of all cores for your numerical computations. Some Tweets why multithreading with OpenBLAS can cause problems.

export OPENBLAS_NUM_THREADS=1
export MKL_NUM_THREADS=1

Background

hyperhyper is based on research by Omer Levy et al. from 2015 (the paper). The authors published the code they used in their experiments as Hyperwords. Initially, I tried to port their original software to Python 3 but I ended up re-writing large parts of it. So this package was born.

How pairs are counted

The basic idea: Construct pairs of words that appear together in sentences (within a given window size). Then do some math magic around matrix operations (PPMI, SVD) to get low-dimensional embeddings.

The count-based word-embeddings by hyperhyper are deterministic. So multiple runs of experiments with identical parameters will yield the same results. (The randomized options — dynamic_window="prob", subsample="prob" — are reproducible through the seed parameter.) Word2vec and others are unstable. Due to randomness, their results will vary.

hyperhyper is built upon the seminal Python NLP package gensim.

Limitations: With hyperhyper you will run into (memory) problems if you need large vocabularies (set of possible words). It's fine if you have a vocabulary up until ~ 50k. Word2vec and fastText especially solve this curse of dimensionality. If you're interested in details you should read the aforementioned excellent paper by Omer Levy et al..

Scientific Literature

This software is based on ideas stemming from the following papers:

  • Improving Distributional Similarity with Lessons Learned from Word Embeddings, Omer Levy, Yoav Goldberg, Ido Dagan, TACL 2015. Paper Code

    Recent trends suggest that neural-network-inspired word embedding models outperform traditional count-based distributional models on word similarity and analogy detection tasks. We reveal that much of the performance gains of word embeddings are due to certain system design choices and hyperparameter optimizations, rather than the embedding algorithms themselves. Furthermore, we show that these modifications can be transferred to traditional distributional models, yielding similar gains. In contrast to prior reports, we observe mostly local or insignificant performance differences between the methods, with no global advantage to any single approach over the others.

  • The Influence of Down-Sampling Strategies on SVD Word Embedding Stability, Johannes Hellrich, Bernd Kampe, Udo Hahn, NAACL 2019. Paper Code Code

    The stability of word embedding algorithms, i.e., the consistency of the word representations they reveal when trained repeatedly on the same data set, has recently raised concerns. We here compare word embedding algorithms on three corpora of different sizes, and evaluate both their stability and accuracy. We find strong evidence that down-sampling strategies (used as part of their training procedures) are particularly influential for the stability of SVD-PPMI-type embeddings. This finding seems to explain diverging reports on their stability and lead us to a simple modification which provides superior stability as well as accuracy on par with skip-gram embedding

Development

Install and use uv.

# `--extra full` pulls in spaCy + scikit-learn; `--group dev` adds pytest and ruff
uv sync --extra full --group dev

Run the fast test suite (no spaCy model required):

uv run pytest -m "not slow"

The slow tests need the spaCy language model, so download it first:

uv run python -m spacy download en_core_web_sm
uv run pytest -m slow          # or `uv run pytest` to run everything

Lint and format:

uv run ruff check .
uv run ruff format --check .   # drop `--check` to apply formatting

See CONTRIBUTING.md for the equivalence gate that guards changes to the pair-counting code.

Contributing

If you have a question, found a bug or want to propose a new feature, have a look at the issues page.

Pull requests are especially welcomed when they fix bugs or improve the code quality. See CONTRIBUTING.md for how to set up the project, run the tests and respect the correctness gate around the counting code.

Future Work / TODO

  • SVD is the largest stage. On 1.7M synthetic tokens (window=2, dim=300), building the corpus takes 1.4s, counting and PMI 1.4s, and the SVD 4.5s. bench/bench_svd.py measures the backends and their knobs.
  • corpus.pkl is the last pickle in a bunch directory. It carries preproc_func by reference, so replacing it needs another way for a bunch to carry its tokenizer (ADR 0002).
  • Python 3.14 waits on gensim publishing cp314 wheels (see the comment on requires-python in pyproject.toml).

Pair counting, tokenization and vocabulary building have been optimized bit-identically; the CHANGELOG has the measurements, and bench/ the harnesses that produced them.

hyperhyper?

Scooter – Hyper Hyper (Song)

Acknowledgments

Building upon the work by Omer Levy et al. for Hyperwords.

License

BSD-2-Clause

Sponsoring

This work was created as part of a project that was funded by the German Federal Ministry of Education and Research.

Metadata

Release files for hyperhyper 0.3.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for hyperhyper 0.3.1
File Size Uploaded Provenance Actions
hyperhyper-0.3.1.tar.gz 453.0 kB With provenance Details for hyperhyper-0.3.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for hyperhyper 0.3.1
File Interpreter ABI Platform Actions
hyperhyper-0.3.1-py3-none-any.whl Python 3 none any Details for hyperhyper-0.3.1-py3-none-any.whl

Total release size: 834.6 kB

Release files / hyperhyper-0.3.1.tar.gz

Download URL hyperhyper-0.3.1.tar.gz
Size 453.0 kB
Tags Source
SHA-256 checksum
How to use checksums
ec0ad184d8ca161e900c43fe9606d6c7afa8ad4ff20739c6b920315c6f763e36
BLAKE2b-256 checksum
How to use checksums
59e96d96819e61184b6eb96bc89bbedeb522316256ba30011aea3842b4bb0ba3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.

Transparency log

Release files / hyperhyper-0.3.1-py3-none-any.whl

Download URL hyperhyper-0.3.1-py3-none-any.whl
Size 381.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
a9615390497bae843552200a38b4cd32e17fa57bc343ed82f9c16620548298b9
BLAKE2b-256 checksum
How to use checksums
463ed90d2194ed6414b908a32579c218806e53e4a6b7de0bcff204219714c9de
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.

Transparency log

Security for hyperhyper 0.3.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for hyperhyper-0.3.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page