Grimmerie

Grimmerie is a small collection of high-level Python utilities for rapid NLP prototyping, vectorization, clustering review, and data inspection.

The package is built around a simple idea:

result = spell(data)

Each spell hides common setup while keeping useful control over its inputs and outputs.

Current Spells

• specterize: Generate SPECTER2 document embeddings.
• tfidfize: Generate configurable TF-IDF representations.
• tfidf_features: Inspect the learned TF-IDF vocabulary.
• tfidf_document_frequency: Count how many documents contain each feature.
• cluster_viz: Write an interactive HTML cluster inspector.

Installation

pip install grimmerie

Grimmerie supports Python 3.9 and newer.

Optional NLP Features

The base package includes TF-IDF and embedding dependencies. Install the NLP extra only if you want spaCy lemmatization:

pip install "grimmerie[nlp]"
python -m spacy download en_core_web_sm

The first call to specterize downloads the SPECTER2 model, tokenizer, and adapter from Hugging Face. These files are cached by the underlying libraries.

Quick Start

import pandas as pd

from grimmerie import cluster_viz, specterize, tfidfize

df = pd.DataFrame({
    "title": [
        "Pre-training of Deep Bidirectional Transformers",
        "Attention Is All You Need",
        "Document-level Representation Learning",
    ],
    "abstract": [
        "A language representation model based on bidirectional training.",
        "A sequence transduction architecture based on attention.",
        "A method for representing scientific documents.",
    ],
})

text = df["title"] + " " + df["abstract"]

tfidf = tfidfize(text, return_type="array")
embeddings = specterize(text, return_type="numpy")

Both vectorizers preserve input order: row i in the output corresponds to row i in the input.

Input Handling

The vectorization spells accept:

• A single string
• A dictionary
• A list
• A pandas Series
• Other iterables
• Scalar values, which are converted to strings

A dictionary is converted into one text item by joining its values:

record = {"title": "BERT", "abstract": "A language model"}
embedding = specterize(record, return_type="numpy")

Lists containing dictionaries are also supported:

records = [
    {"title": "BERT", "abstract": "A language model"},
    {"title": "SPECTER", "abstract": "A document model"},
]

embeddings = specterize(records, return_type="numpy")

Input normalization is intentionally convenient rather than schema-aware. If you need a specific field order or formatting, combine your fields into a string column before calling a spell.

specterize

specterize(
    input_data,
    return_type="list",
    max_length=512,
)

specterize uses the SPECTER2 base model and adapter:

• Base model: allenai/specter2_base
• Adapter: allenai/specter2

The input is tokenized, truncated to max_length, and passed through the model. The first token representation is returned as the document embedding. The standard output has 768 features per input item.

Return Types

return_type	Result
"list"	Python list[list[float]]
"numpy"	NumPy array with shape (n, 768)
"tensor"	PyTorch tensor with shape (n, 768)

Example:

texts = [
    {"title": "BERT", "abstract": "We introduce a new model."},
    {"title": "Attention", "abstract": "Transformers use attention."},
]

embeddings = specterize(texts, return_type="numpy")
print(embeddings.shape)
# (2, 768)

SPECTER2 Notes

• The first call may download model data.
• PyTorch must be installed separately or through grimmerie[specter].
• Models are cached after loading.
• Model and tokenizer objects are cached in the module process.
• Text longer than max_length is truncated.
• Inference runs under torch.no_grad().
• The current implementation runs inference on the default PyTorch device.

tfidfize

tfidfize(
    input_data,
    *,
    lemmatize=False,
    spacy_model="en_core_web_sm",
    batch_size=2000,
    n_process=1,
    progress_interval=None,
    min_df=1,
    max_df=1.0,
    stop_words="english",
    ngram_range=(1, 1),
    lowercase=True,
    max_features=None,
    norm="l2",
    use_idf=True,
    smooth_idf=True,
    sublinear_tf=False,
    return_type="sparse",
    return_vectorizer=False,
    vectorizer=None,
)

By default, tfidfize fits scikit-learn's TfidfVectorizer on the supplied text and returns a sparse matrix.

Return Types

return_type	Result
"sparse"	SciPy sparse matrix
"array"	Dense NumPy array
"list"	Python list of lists
"frame"	pandas DataFrame with vocabulary columns

Use sparse output for large vocabularies. Dense output is convenient for small experiments but can use substantially more memory.

X = tfidfize(
    df["title"] + " " + df["abstract"],
    return_type="array",
)

Reusing a Vectorizer

Use return_vectorizer=True to keep the fitted vectorizer:

X_train, vectorizer = tfidfize(
    train_text,
    return_type="sparse",
    return_vectorizer=True,
)

X_test = tfidfize(
    test_text,
    vectorizer=vectorizer,
    return_type="sparse",
)

When vectorizer is supplied, Grimmerie calls transform rather than fitting a new vocabulary.

Lemmatization

Set lemmatize=True to preprocess text with spaCy before TF-IDF:

X = tfidfize(
    text,
    lemmatize=True,
    spacy_model="en_core_web_sm",
    return_type="array",
)

The lemmatization pipeline keeps alphabetic, non-stopword lemmas in lowercase. Use batch_size, n_process, and progress_interval to control spaCy processing.

Important TF-IDF Parameters

Parameter	Purpose
min_df	Ignore terms appearing in fewer documents than this threshold.
max_df	Ignore terms appearing in more documents than this threshold.
stop_words	Stopword configuration passed to scikit-learn.
ngram_range	Minimum and maximum n-gram sizes.
max_features	Optional vocabulary size limit.
norm	Row normalization, usually "l1", "l2", or None.
use_idf	Enable inverse document-frequency weighting.
smooth_idf	Smooth inverse document-frequency values.
sublinear_tf	Use logarithmic term-frequency scaling.
lowercase	Lowercase text during vectorization.

TF-IDF Helpers

tfidf_features

Return the learned feature names from a fitted TfidfVectorizer:

from grimmerie import tfidf_features

X, vectorizer = tfidfize(
    text,
    return_vectorizer=True,
)

features = tfidf_features(vectorizer)
print(features[:10])

The feature order matches the columns of the TF-IDF matrix.

tfidf_document_frequency

Count the number of documents containing each feature:

from grimmerie import tfidf_document_frequency

document_frequency = tfidf_document_frequency(X)

The returned NumPy array has one value per feature and follows the same feature order as the matrix columns.

cluster_viz

cluster_viz writes a standalone interactive HTML cluster inspector from a pandas DataFrame. It does not modify the input DataFrame.

cluster_viz(
    dataframe,
    cluster_id_column,
    *,
    entry_columns=None,
    truth_value_column=None,
    truth_true_value=True,
    truth_false_value=False,
    truth_unknown_value=None,
    output_path,
    title=None,
    title_column=None,
)

output_path and at least one entry_columns value are required.

Basic Visualization

Use cluster_viz without a truth column when you only want to inspect cluster membership and row data:

from grimmerie import cluster_viz

cluster_viz(
    df,
    cluster_id_column="cluster",
    entry_columns=["title", "orcid", "url"],
    output_path="clusters.html",
    title="Paper Clusters",
)

Open clusters.html in a browser. Each bubble represents a cluster and its size represents the number of visible entries. Selecting a bubble opens its entries in the side panel.

A cluster ID of -1 is displayed as Noise.

Truth-Aware Visualization

The truth column is a ternary status column. It can use any values you choose for true, false, and unknown:

cluster_viz(
    df,
    cluster_id_column="cluster",
    entry_columns=["title", "orcid", "url"],
    truth_value_column="truth",
    truth_true_value=82829383,
    truth_false_value="santa",
    truth_unknown_value="",
    output_path="clusters.html",
    title="Author Resolution Review",
)

Values matching the configured true and false values are treated as known. Values that match neither are treated as unknown. Unknown entries remain available for inspection but are excluded from metric calculations.

When a truth column is configured, the header provides global filters:

• All: Show every entry.
• True: Show only true entries.
• False: Show only false entries.
• Unknown: Show only unknown entries.

The filters update visible entries, bubble sizes, entry counts, cluster counts, and noise counts. The filter does not change the metric calculations.

Every displayed entry has a copy button that copies the first configured entry column. The side panel also includes actions for copying all selected entries or copying the selected entries as TSV.

Truth-Aware Metrics

Metric cards appear only when a truth column is supplied. The cards display the global mean, median, and a compact distribution for:

• Precision
• Recall
• F1
• Split score
• True-reference B3 precision
• True-reference B3 recall
• True-reference B3 F1

Metrics use the following scope:

1. True entries define the target population.
2. Only predicted clusters containing at least one true entry are evaluated.
3. False entries inside those clusters affect precision and B3 precision.
4. False-only clusters are ignored.
5. Unknown entries are excluded from metric denominators.

The split score measures the share of true entries retained in the largest evaluated predicted cluster. The B3 metrics are evaluated for true entries, using the known entries in their predicted clusters for precision and the full true population for recall.

Visualization Titles

Use title for an explicit name:

cluster_viz(
    df,
    cluster_id_column="cluster",
    entry_columns=["title"],
    output_path="clusters.html",
    title="Block 17",
)

Alternatively, use the first non-null value from a DataFrame column:

cluster_viz(
    df,
    cluster_id_column="cluster",
    entry_columns=["title"],
    title_column="block_name",
    output_path="clusters.html",
)

title takes precedence over title_column. If neither is provided, the visualization is named Cluster Inspector.

Browser Requirements

The generated HTML loads D3.js and web fonts from public CDNs. The file is otherwise self-contained, but the browser needs network access to load those resources. Copy buttons use the browser Clipboard API, which may be restricted when opening a file directly in some browser configurations.

Saving Outputs

Dense NumPy Output

import numpy as np

X = tfidfize(text, return_type="array")
np.save("tfidf.npy", X)

Sparse TF-IDF Output

from scipy import sparse

X = tfidfize(text, return_type="sparse")
sparse.save_npz("tfidf.npz", X)

HTML Visualization

The visualization is written directly to the requested path:

cluster_viz(
    df,
    cluster_id_column="cluster",
    entry_columns=["title"],
    output_path="reports/clusters.html",
)

The parent directory must already exist.

Common Workflows

Compare TF-IDF and SPECTER2 Representations

text = df["title"] + " " + df["abstract"]

tfidf_matrix = tfidfize(text, return_type="array")
specter_embeddings = specterize(text, return_type="numpy")

# Row i in both outputs corresponds to df.iloc[i].

Review Clustering Results

from grimmerie import cluster_viz

df["cluster"] = predicted_cluster_labels

cluster_viz(
    df,
    cluster_id_column="cluster",
    entry_columns=["title", "orcid", "url"],
    truth_value_column="truth",
    truth_true_value="confirmed",
    truth_false_value="rejected",
    truth_unknown_value="unknown",
    output_path="cluster_review.html",
    title="Resolution Review",
)

Troubleshooting

spaCy is required for lemmatization

Install the optional dependency:

pip install "grimmerie[nlp]"

Then download the configured model:

python -m spacy download en_core_web_sm

Model Downloads Are Slow

The first SPECTER2 call downloads model files from Hugging Face. Later calls reuse the local cache. A network connection is required for the first call unless the model has already been cached.

empty vocabulary; perhaps the documents only contain stop words

This is a scikit-learn error. Check that the input contains usable text and that your stopword, min_df, max_df, and preprocessing settings do not remove every token.

Missing Cluster Columns

cluster_viz validates every configured column and raises an error listing missing columns. Check cluster_id_column, entry_columns, truth_value_column, and title_column against dataframe.columns.

Missing Cluster IDs

Cluster IDs cannot be missing. Fill or remove missing IDs before calling cluster_viz.

Large Outputs

Prefer sparse TF-IDF output for large vocabularies. Dense matrices can require substantial memory. The HTML cluster inspector embeds all selected entry data in the generated file, so very large DataFrames can produce large HTML files.

Design Principles

• One-call workflows for common experiments.
• Strong defaults for rapid prototyping.
• Consistent row alignment across vectorization spells.
• Explicit output formats where downstream code needs control.
• Interactive inspection when aggregate metrics are not enough.

Limitations

Grimmerie is designed for experimentation and review rather than fully managed production pipelines. You may want lower-level control when you need:

• Exact model device placement and batching controls.
• Fully reproducible model and dependency locking.
• Custom SPECTER2 pooling behavior.
• Very large-scale visualization.
• Custom browser asset hosting instead of CDN resources.

Public API

The package exports the following names:

from grimmerie import (
    cluster_viz,
    specterize,
    tfidf_document_frequency,
    tfidf_features,
    tfidfize,
)