LLM Fingerprinting System

A black-box fingerprinting system that estimates an LLM's model family (GPT, LLaMA, Mistral, etc.) by analysing response patterns across 31 prompts. It can return unknown when the observations do not support a known family. General identification of fine-tunes, distillations and their base models has not yet been validated.

Note: config.py lists configured families; recognition requires representative training data. Backend support alone does not provide it.

A pre-trained classifier is bundled inside the package (llm_fingerprinter/model/) and used automatically.

---

How It Works

Fingerprinting runs in three sequential layers:

1. 31 prompts across 3 layers (discriminative → behavioral → stylistic):

   • Discriminative (11): Identity, knowledge cutoff, architecture, reasoning — most separating power
   • Behavioral (7): Safety boundaries, jailbreak resistance, honesty, policy handling
   • Stylistic (13): Formatting, creativity, constraint following, default voice

2. Feature extraction per response: 384 embedding dimensions + 12 linguistic features + 6 behavioral features. Responses are averaged within each layer, giving 402 dimensions per layer and 1206 dimensions per complete fingerprint.

3. Embedding rebalancing: Per-layer PCA retains up to 64 embedding components, limited by the numerical rank of the original observations. PCA, scaling and optional global PCA are fitted before synthetic augmentation, using only the training observations. A shared scale within each embedding block preserves its geometry without amplifying weak components. Constant engineered features are excluded, so the working dimension depends on the training data.

4. Ensemble evidence: Random Forest (45%) + SVM (45%) + MLP (10%) provide a supporting family estimate and disagreement diagnostics.

5. Shared identification policy: Family templates decide the accepted family using cosine distance, ambiguity and distance rejection. The CLI and Python API use this same policy. An unknown family stays unknown even if the ensemble or an optional model-version template suggests a match. A version estimate is accepted only when it agrees with an accepted family. Displayed template scores are similarities, not calibrated probabilities.

6. Early stopping (opt-in): identify --early-stop <threshold> can skip remaining layers using the ensemble's provisional score. A fingerprint with skipped layers returns unknown from the final identification policy; full-fingerprint templates cannot validate a partial observation. The default, --early-stop 0, runs the full suite.

7. Collection quality gates: Empty responses count as failed queries. Every attempted layer must meet its coverage requirement: 60% for identification and 90% for training collection. Embedding failures, incorrect feature dimensions, nonfinite values and zero embedding blocks prevent fingerprint creation. Training imports also validate the available collection metadata and feature schema, skipping invalid or explicitly incomplete records with a logged reason. Only layers deliberately skipped by early stopping can be padded, and those records are excluded from training imports.

Newly saved fingerprints retain all successfully collected prompts and raw responses, the feature schema, prompt-suite hash, generation settings, layer coverage and query counts. Historical files remain usable when the checks supported by their recorded data pass; missing historical provenance cannot be reconstructed.

---

Supported Backends

Backend	Description	API Key Required
ollama	Local Ollama instance	❌ No
ollama-cloud	Ollama Cloud API	✅ OLLAMA_CLOUD_API_KEY
openai	OpenAI API (or compatible)	✅ OPENAI_API_KEY
gemini	Gemini API	✅ GEMINI_API_KEY
deepseek	DeepSeek API	✅ DEEPSEEK_API_KEY
custom	Any HTTP-based LLM API	Optional

About the Custom Backend

The custom backend is the most flexible option — use it with:

• Proprietary LLM APIs not natively supported
• Self-hosted LLMs behind HTTP endpoints
• API proxies and gateways
• Any HTTP-based LLM service

All you need is an HTTP request template file. See examples in ./example/.

Responses can use JSON, SSE, NDJSON or Content-Type: text/plain. Streamed text fragments preserve spaces and newlines. Structured error events, malformed framing and incomplete streams raise an error instead of becoming fingerprint text. For a known plain-text endpoint that omits Content-Type, opt in through the Python API with CustomClient(request_file="request.txt", allow_plain_text=True); the CLI requires the endpoint to return the appropriate content type.

---

Installation

From PyPI

# Core package
pip install llm-fingerprinter

# With OpenAI support
pip install llm-fingerprinter[openai]

# With Gemini support
pip install llm-fingerprinter[gemini]

# With all backends
pip install llm-fingerprinter[all]

Quick Start

1. Identify a Model (Pre-trained Classifier)

# Local Ollama
llm-fingerprinter identify -b ollama --model llama3.2

# OpenAI
export OPENAI_API_KEY="your-key"
llm-fingerprinter identify -b openai --model gpt-4o-mini

# Custom endpoint
llm-fingerprinter identify -b custom -r ./custom_request.txt

2. Train Your Own Classifier

# Step 1: Generate training fingerprints for each family
#         Temperature is automatically varied across simulations for diversity
llm-fingerprinter simulate -b ollama --model llama3.2 --family llama --num-sims 5
llm-fingerprinter simulate -b openai --model gpt-4o-mini --family gpt --num-sims 5

# Step 2: Train and save the ensemble plus matching family templates
llm-fingerprinter train

# Step 3: Optionally enable estimates of already represented model versions
llm-fingerprinter build-model-templates

# Step 4: Identify unknown models
llm-fingerprinter identify -b ollama --model some-unknown-model

train saves family templates both inside the classifier artifact and as the matching external template store. Python's LLMFingerprinter.identify() uses the embedded templates by default; callers can inject family_templates and model_templates explicitly. The CLI loads the configured external stores when present, allowing build-templates and add-family updates to take effect. Use the same stores in Python to reproduce those updated decisions.

Older classifier artifacts keep their original preprocessing when loaded. Run train to obtain the corrected representation and embedded family templates; loading an old artifact does not retrain it.

---

train --early-stop-variants — Off by Default

Early stopping produces a partial fingerprint: skipped layers are padded with the mean of completed layers. --early-stop-variants adds synthetic examples of those padding patterns when fitting the ensemble. The representation and family templates are still fitted only on complete, original observations.

Both early stopping and its training variants are off by default because the production family decision requires a complete fingerprint. Variants support experiments with provisional ensemble scores; they do not make partial runs eligible for an accepted family or version. The effect of variants on a future partial-observation policy needs its own independent evaluation.

llm-fingerprinter train                          # full-suite default
llm-fingerprinter train --early-stop-variants    # experimental partial scoring
llm-fingerprinter identify -b ollama --model llama3.2 --early-stop 0

identify --early-stop warns if the loaded classifier was trained without them.

---

train --cross-validate — Evaluation by Model

Cross-validation groups fingerprints by model, so every simulation of one model stays on the same side of the split. Groups are assigned within each family to keep every represented family in both training and validation. All transforms, augmentation and templates are fitted inside the training fold.

This matters because simulate -n 5 produces five runs of a single model at five temperatures. Those runs can be very similar. Ungrouped evaluation can therefore overstate performance on unseen models within a known family. The reported accuracy scores the shared final decision, counting rejected known samples as unsuccessful. The Python result also includes the ensemble's score before rejection, accepted coverage and accuracy among accepted results.

The fold count is capped by the family with the fewest distinct models, and the output says which one:

llm-fingerprinter train --cross-validate

   Grouping by model — 21 distinct models across 108 fingerprints
   Folds capped at 2 by family 'llama' (2 distinct model(s))

To get more folds — and a less noisy estimate — collect fingerprints for more distinct models within a family, not more simulations of the same one.

The corrected implementation was evaluated on the existing 108 fingerprints from 21 models, using the same seven two-fold model-grouped splits as the code review. Mean raw ensemble family accuracy was 94.84%; the shared final policy scored 94.58%, counting rejection as unsuccessful. In a separate challenge that excluded each family from training in turn, the shared policy rejected 105/108 excluded-family samples. These are exploratory results on a small, reused corpus, not a fresh test set or a general accuracy guarantee.

What to Collect Next

• Reserve new, independent models and collection sessions as a test set before changing prompts, thresholds or classifiers. Evaluate additional runs of known versions, unseen models within known families, and entirely unknown families separately.
• Broaden family coverage: Gemini is configured but currently has no samples in the reviewed corpus. Add more Llama/Qwen generations and independently sourced fine-tunes with known ancestry before adding many more runs of the existing models.
• Collect the same models across backends and system-prompt settings, and keep raw responses and model/deployment identifiers. This helps distinguish family signal from provider or configuration effects.
• Use the new raw-response and schema records for re-extraction and prompt ablation. Keep held-out data out of feature selection and threshold tuning.

---

build-templates — Build Family Template Classifier

Rebuild the external family template store from valid training fingerprints, for example after changing the collection or the rejection threshold. train already creates matching family templates, so this is not a required extra step after training.

llm-fingerprinter build-templates

The template classifier uses cosine distance to the nearest family mean and rejects distant or ambiguous queries. Its decision is authoritative in the shared policy. At least two distinct, usable family templates are required. Rebuilding this external store does not replace templates embedded in an older classifier artifact; Python callers should inject the updated store or retrain.

---

build-model-templates — Build Model-Level Templates

Build optional templates for model versions represented in the collection (e.g. gpt-4o-mini vs gpt-4.1). A model-version match is accepted only within an accepted, agreeing family; it cannot rescue an unknown family decision or establish a version absent from the store.

llm-fingerprinter build-model-templates

Requires fingerprints that contain model_name in their metadata (all fingerprints generated with simulate on this version do).

---

add-family — Add a New Family Without Retraining

Add a new model family to the template classifier from a few fingerprint samples, without retraining the full ensemble.

llm-fingerprinter add-family --model deepseek-chat --family deepseek --num-fps 3 -b deepseek

Start with at least 3 valid, complete fingerprints. More independent models and sessions are needed to assess whether the family template generalizes. Existing templates for another family must already be available. The CLI uses the updated external store; Python callers should inject that store explicitly.

---

Environment Variables

Variable	Backend	Description
OLLAMA_CLOUD_API_KEY	ollama-cloud	Ollama Cloud API key
OPENAI_API_KEY	openai	OpenAI API key
GEMINI_API_KEY	gemini	Gemini API key
DEEPSEEK_API_KEY	deepseek	DeepSeek API key
LOG_LEVEL	all	Logging level (DEBUG, INFO, WARNING)
LLM_FINGERPRINTER_DATA	all	Override data directory (fingerprints, logs)
LLM_FINGERPRINTER_MODEL	all	Override the model/templates directory

Where data and model artifacts live

How you run it	Data directory	Model directory
From a source checkout	the checkout root	llm_fingerprinter/model/
Installed from PyPI	~/Library/Application Support/llm-fingerprinter (macOS), $XDG_DATA_HOME/llm-fingerprinter (Linux), %LOCALAPPDATA%\llm-fingerprinter (Windows)	<data dir>/model

These locations are derived from where the package is installed, never from your current working directory. Model artifacts are joblib files, and loading one executes the code it contains — so an artifact is only loaded from the installed package, the per-user data directory, or a path you name explicitly via LLM_FINGERPRINTER_MODEL / LLM_FINGERPRINTER_DATA. Anything else is refused.

Upgrading from ≤ 0.4.1? Earlier versions located the data directory by walking up from the working directory to the nearest setup.py/.git, so fingerprints could end up inside whatever repository you happened to be in. Those files are still on disk; point LLM_FINGERPRINTER_DATA at that directory to keep using them. The CLI prints a reminder when it spots one.

---

License

MIT License