embedding-condensation 0.1.0

Measure layer-wise token embedding cosine similarity (embedding condensation diagnostic).

embedding-condensation

Minimal library for the embedding condensation diagnostic from LM-Dispersion: layer-wise token cosine-similarity matrices and optional heatmaps.

Install

cd pypi
pip install -e ".[test]"

Usage

from transformers import AutoModel, AutoTokenizer
from embedding_condensation import measure_embedding_condensation

model = AutoModel.from_pretrained("gpt2").eval()
tokenizer = AutoTokenizer.from_pretrained("gpt2")

result = measure_embedding_condensation(
    model,
    tokenizer,
    texts=["Your long input text here. " * 200],
    repetitions=1,
    plot=False,
)
print(result.mean_cossim_by_layer)

PyPI upload

cd pypi
pip install build twine
python -m build
twine upload dist/*

Test

cd pypi
pytest