A package of useful functions to analyze transformer based language models.
Project description
minicons
Helper functions for analyzing Transformer based representations of language
This repo is a wrapper around the transformers
library from hugging face :hugs:
Installation
Install from Pypi using:
pip install minicons
Supported Functionality
- Extract word representations from Contextualized Word Embeddings
- Score sequences using language model scoring techniques, including masked language models following Salazar et al. (2020).
Examples
- Extract word representations from contextualized word embeddings:
from minicons import cwe
model = cwe.CWE('bert-base-uncased')
context_words = [("I went to the bank to withdraw money.", "bank"),
("i was at the bank of the river ganga!", "bank")]
print(model.extract_representation(context_words, layer = 12))
'''
tensor([[ 0.5399, -0.2461, -0.0968, ..., -0.4670, -0.5312, -0.0549],
[-0.8258, -0.4308, 0.2744, ..., -0.5987, -0.6984, 0.2087]],
grad_fn=<MeanBackward1>)
'''
- Compute sentence acceptability measures (surprisals) using Incremental Language Models:
from minicons import scorer
# Masked LM scoring is temporarily broken. I am working on fixing it asap (date: Oct 21, 2021)
# mlm_model = scorer.MaskedLMScorer('bert-base-uncased', 'cpu')
ilm_model = scorer.IncrementalLMScorer('distilgpt2', 'cpu')
stimuli = ["The keys to the cabinet are on the table.",
"The keys to the cabinet is on the table."]
# use sequence_score with different reduction options:
# Sequence Surprisal - lambda x: -x.sum(1)
# Sequence Log-probability - lambda x: x.sum(1)
# Sequence Surprisal, normalized by number of tokens - lambda x: -x.mean(1)
# Sequence Log-probability, normalized by number of tokens - lambda x: x.mean(1)
# and so on...
print(ilm_model.sequence_score(stimuli, reduction = lambda x: -x.sum(1)))
'''
[41.51601982116699, 44.497480392456055]
'''
Tutorials
Upcoming features:
- Explore attention distributions extracted from transformers.
- Contextual cosine similarities, i.e., compute a word's cosine similarity with every other word in the input context with batched computation.
- Open to suggestions!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
minicons-0.1.14.tar.gz
(14.9 kB
view hashes)
Built Distribution
minicons-0.1.14-py3-none-any.whl
(15.3 kB
view hashes)
Close
Hashes for minicons-0.1.14-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9d654972c93641dd2b18bbcb5e7fa53f5ad44333cb4e5ef4a6bd3ac4d83b49b3 |
|
MD5 | c87d12894a3f18c22ef25c8ce5041d8e |
|
BLAKE2b-256 | 8f2a5f94bc5bc80211022dab9b2eb8b367fbe79e4ea8cbd6a1552273da86d41f |