Skip to main content

Model Provenance Kit

Part of Cisco AI Defense - Open-source AI security scanners, developer tools, and research from Cisco.

License Python Discord Cisco AI Defense AI Security and Safety Framework

Model Provenance Kit is a Python toolkit and CLI for detecting model provenance. It determines whether a machine learning model derives from a known base model family by comparing multi-signal fingerprints extracted from weights, tokenizers, and architecture metadata.

Model Provenance Kit Demo

Key Features

  • Pairwise comparison: compare any two models head-to-head across 8 provenance signals.
  • Database scan: scan a model against a reference database of known base-model fingerprints.
  • Deep-signal fingerprints: download pre-computed weight fingerprints for weight-level matching.
  • Multi-signal pipeline: combines metadata (MFI), tokenizer (TFV, VOA), and weight signals (EAS, NLF, LEP, END, WVC) into a single pipeline score.
  • Weights decide: architecture metadata (MFI) supports the verdict but never confirms a match without weight evidence.
  • Two-layer caching: in-memory + disk JSON cache for fast repeat runs.
  • Multiple output formats: Rich terminal table (default), JSON, or plain text.
  • Streaming support: models over 20 GB are loaded via streaming to limit memory usage.

Reference Database

The bundled reference database contains fingerprints for ~175 base models spanning 45+ model families from 20+ publishers, ranging from 135M to 70B+ parameters. Covered publishers include:

Meta, Google, Alibaba, Microsoft, Mistral AI, DeepSeek, TII, Zhipu AI, NVIDIA, IBM, BigScience, OpenAI, Allen AI, Facebook AI, Stability AI, Hugging Face, Cohere, Databricks, Tencent, Moonshot AI, MiniMax, and more.

The database covers text generation, fill-mask, text-to-text, embedding, and translation architectures across four size buckets (<=1B, 1B–10B, 10B–40B, 40B+).

Documentation

For deeper technical details, see the guides in docs/:

Guide Description
Pipeline Architecture End-to-end data flow, compare vs scan modes, phase breakdown
Signal Reference Extraction, similarity, and behaviour of all 8 provenance signals
Scoring and Model Loading Identity/tokenizer scores, verdict rules, NaN handling, large-model streaming
Database and Caching Seed database layout, deep-signal download, two-layer cache, HMAC integrity

For the formal definition of model provenance — what counts as a provenance relationship and what does not — see the Model Provenance Constitution.

Installation

Requirements

  • Python 3.12+
  • uv (recommended) or pip

Install from source

git clone https://github.com/cisco-ai-defense/model-provenance-kit.git
cd model-provenance-kit
uv sync

Install as a CLI tool

uv tool install .

After installation the provenancekit command is available:

provenancekit --help

Quick Start

1. Download deep-signal fingerprints (one-time setup)

Deep-signal fingerprints are pre-computed weight-level features stored as parquet files. They enable the full weight-signal matching pipeline during scan. Without them, scan results rely only on metadata and tokenizer signals.

The fingerprints are hosted on Hugging Face: cisco-ai/model-provenance-kit.

provenancekit download-deepsignals-fingerprint

Check installation status at any time:

provenancekit download-deepsignals-fingerprint --status

To update to the latest fingerprints:

provenancekit download-deepsignals-fingerprint --update

2. Scan a model against known base models

provenancekit scan bigscience/bloom-560m

This extracts features from the model, runs a 3-stage lookup against the reference database, and returns ranked matches with scores and decision labels.

3. Compare two models head-to-head

provenancekit compare gpt2 distilgpt2

Usage

Commands

Command Purpose
provenancekit compare MODEL_A MODEL_B Pairwise comparison of two models
provenancekit scan MODEL_ID Scan one model against the reference database
provenancekit download-deepsignals-fingerprint Download/manage deep-signal weight fingerprints

Output Formats

All commands that produce results support three output modes:

# Rich terminal table (default)
provenancekit compare gpt2 gpt2

# JSON (machine-readable, suitable for piping)
provenancekit compare gpt2 gpt2 --json

# Plain text (no colour, CI-friendly)
provenancekit compare gpt2 gpt2 --plain

Verbose Logging

Enable structured logging to stderr with the top-level --verbose flag. It must come before the subcommand:

provenancekit --verbose scan bigscience/bloom-560m
provenancekit --verbose compare gpt2 distilgpt2

CLI Reference

provenancekit compare

provenancekit [--verbose] compare MODEL_A MODEL_B [options]
Option Description
MODEL_A First model: HuggingFace repo ID (e.g. gpt2) or local path
MODEL_B Second model: HuggingFace repo ID or local snapshot path
--json Output as JSON
--plain Output as plain key-value text (no colour)
--cache-dir PATH Override the default cache directory
--no-cache Disable feature caching entirely
--timing Show high-level phase timings
--trust-remote-code Allow model-hosted Python code (config/tokenizer); use only with models you trust

Examples:

# Basic comparison
provenancekit compare gpt2 distilgpt2

# Compare with JSON output
provenancekit compare bigscience/bloom-560m bigscience/bloomz-560m --json

# Compare with custom cache
provenancekit compare gpt2 gpt2 --cache-dir /tmp/pk-cache

# Compare without caching
provenancekit compare gpt2 gpt2 --no-cache

provenancekit scan

provenancekit [--verbose] scan MODEL_ID [options]
Option Default Description
MODEL_ID Model to scan: HuggingFace repo ID or local snapshot path
--json Output as JSON
--plain Output as plain key-value text (no colour)
--top-k N 3 Maximum number of matches to return
--threshold F 0.50 Minimum pipeline score for inclusion (0.0–1.0)
--db-root PATH bundled DB Override the provenance database root directory
--cache-dir PATH ~/.provenancekit/cache Override the default cache directory
--no-cache Disable feature caching
--timing Show phase-level timing breakdown
--trust-remote-code Allow model-hosted Python code (config/tokenizer); use only with models you trust

Examples:

# Basic scan
provenancekit scan bigscience/bloom-560m

# Scan with more results and lower threshold
provenancekit scan gpt2 --top-k 10 --threshold 0.30

# Scan with JSON output
provenancekit scan bigscience/bloom-560m --json

# Scan with a custom database
provenancekit scan gpt2 --db-root /path/to/my/database

Scan workflow:

  1. Extract model fingerprint and features (MFI, tokenizer, weight signals).
  2. Run a 3-stage lookup against the provenance database:
    • Stage 1 (Param Filter): size-bucket filtering (±1 adjacent bucket).
    • Stage 2 (Hash Check): annotate candidates with exact/family/none match.
    • Stage 3 (Similarity): full scoring per candidate (MFI tier + weight identity).
  3. Return ranked matches with scores, decision labels, and signal breakdowns.

provenancekit download-deepsignals-fingerprint

provenancekit download-deepsignals-fingerprint [options]
Option Description
--db-root PATH Override the provenance database root directory
--update Re-download and replace existing fingerprints with the latest
--no-verify Skip SHA-256 integrity check after download
--status Show current deep-signals installation status and exit

What it does:

  1. Downloads deep-signals-v2.zip from HuggingFace Hub (HTTPS only).
  2. Verifies SHA-256 integrity (unless --no-verify).
  3. Extracts parquet files with safety checks (size limits, path traversal protection, symlink rejection).
  4. Performs an atomic swap of the by-family/ directory.
  5. Writes an installation marker for subsequent status checks.

Examples:

# First-time install
provenancekit download-deepsignals-fingerprint

# Check what's installed
provenancekit download-deepsignals-fingerprint --status

# Force update to latest
provenancekit download-deepsignals-fingerprint --update

# Install to a custom database location
provenancekit download-deepsignals-fingerprint --db-root /data/provenance-db

Signals and Scoring

Model ProvenanceKit combines three categories of evidence into a single pipeline score.

Metadata Signal

Signal Full Name Description
MFI Metadata Family Identification 3-tier gate from config.json: Tier 1 (exact arch hash), Tier 2 (family hash + dimension check), Tier 3 (weighted soft match across 11 feature groups)

Tokenizer Signals

Signal Full Name Description
TFV Tokenizer Feature Vector 11-component structural similarity (class, vocab size, BOS/EOS, script distribution, merge rules, etc.)
VOA Vocabulary Overlap Analysis Jaccard similarity between vocabulary sets

Weight Signals

Signal Full Name Description
EAS Embedding Anchor Similarity Pairwise cosine of script-aware anchor embedding rows → self-similarity matrix → Pearson on upper triangle
NLF Norm Layer Fingerprint Concatenated LayerNorm/RMSNorm weight vectors → cosine similarity
LEP Layer Energy Profile Frobenius norm per layer → 1D profile → Pearson correlation
END Embedding Norm Distribution Row-wise L2 norms of embeddings → histogram → cosine similarity
WVC Weight Vector Correlation Per-layer statistical signature → mean cosine over common layers

Scoring

Identity score: NaN-aware weighted average of the 5 weight signals (EAS, NLF, LEP, END, WVC). Signal weights are calibrated via Cohen's d on a 111-pair benchmark. When a signal returns NaN, it is excluded and remaining weights are proportionally rescaled.

Tokenizer score: supplementary context, 25% TFV + 75% VOA. Reported alongside identity but not used in the pipeline decision. In scan, VOA is not computed, so the score is partial (at most 0.25); use the tfv field or compare for tokenizer similarity.

Pipeline score: final decision score. Weights decide, metadata supports:

  • Weight evidence available: pipeline score = identity score
  • No weight evidence: pipeline score = MFI score, and the decision is marked unverified

Score Interpretation

MFI Tier Identity Score (I) Verdict
1–2 I > 0.75 Confirmed Match
1–2 0.65 < I ≤ 0.75 Weak Match (metadata match, weak weight evidence)
1–2 I ≤ 0.65 Not Matched (metadata/weight conflict)
3 I > 0.75 High-Confidence Match
3 0.65 < I ≤ 0.75 Weak Match
3 I ≤ 0.65 Not Matched
any NaN Metadata Match (unverified) / Not Matched (unverified)

Two rules adjust this table:

  • Unrelated weights: when WVC is computed and below 0.01, the score is capped at 0.65 (Not Matched). Separately trained models of the same family share architecture and tokenizer, which can lift the other signals above the match bands.
  • Shallow evidence (no LEP and WVC, e.g. bitsandbytes, AWQ, GPTQ): a match needs MFI Tier 1/2 and matching tokenizers; otherwise the score is capped at 0.75, Weak Match (unverified).

A model whose config.json contradicts its own tensor shapes gets no MFI Tier 1/2; the fields are listed in config_inconsistent.

Every result includes a verification field: weights_agree, weights_weak, weights_conflict, weights_only, weights_shallow or unverified.

Caching

Model ProvenanceKit uses a two-layer feature cache to speed up repeat comparisons:

  1. In-memory cache: session-scoped Python dict for instant lookups within the same process.
  2. Disk cache: JSON files under ~/.provenancekit/cache/ (configurable) storing per-model MFI fingerprints, tokenizer features, vocabularies, and weight signals.

On a warm cache, Model ProvenanceKit skips expensive model loading and feature extraction, reducing comparison time from minutes to seconds.

Cache controls:

# Use a custom cache directory
provenancekit compare gpt2 gpt2 --cache-dir /tmp/pk-cache

# Disable caching entirely (always extract fresh)
provenancekit compare gpt2 gpt2 --no-cache

The HuggingFace Hub also caches downloaded model files and tokenizer assets locally. Both caches work together to minimize network usage and computation.

Environment Variables

All settings use the PROVENANCEKIT_ prefix and can be set as environment variables:

Variable Default Description
PROVENANCEKIT_CACHE_DIR ~/.provenancekit/cache Feature cache directory
PROVENANCEKIT_DB_ROOT bundled database Path to the provenance seed database
PROVENANCEKIT_SCAN_TOP_K 3 Max matches for scan
PROVENANCEKIT_SCAN_THRESHOLD 0.50 Min pipeline score for scan results

Example:

export PROVENANCEKIT_CACHE_DIR=/tmp/pk-cache
export PROVENANCEKIT_SCAN_TOP_K=10
provenancekit scan gpt2

Benchmark

The benchmarks/run_benchmark.ipynb notebook runs a structured evaluation across similar and dissimilar model pairs.

Jupyter Kernel Setup

cd model-provenance-kit
uv pip install ipykernel
uv run python -m ipykernel install --user --name provenancekit --display-name "ProvenanceKit (.venv)"

Then select "ProvenanceKit (.venv)" as the kernel in VS Code / Cursor / JupyterLab.

Configuration

The notebook exposes three knobs in the Configuration cell:

Parameter Default Description
MAX_WORKERS 2 Parallel comparison workers
PAIR_LIMIT None Max pairs to evaluate (None = all)
PAIR_FILTER "all" "all", "similar", or "dissimilar"

Development

Run tests

# Fast tests only
uv run pytest -m "not slow" --tb=short -q

# All tests (includes model downloads)
uv run pytest -m slow

# With coverage
uv run pytest -m "not slow" --cov=provenancekit --cov-report=term-missing

Lint and type-check

# Lint
uv run ruff check src/ tests/

# Format
uv run ruff format src/ tests/

# Type check
uv run mypy src/

Pre-commit hooks

# Install hooks
uv run pre-commit install

# Run all hooks against all files
uv run pre-commit run --all-files

Troubleshooting

Model downloads hang or stall

Recent versions of huggingface_hub (≥ 0.27) include Xet, a storage backend for large files. On some networks and VPNs the Xet transfer protocol can stall or produce errors when downloading model weights, tokenizer files, or other repository assets (e.g. Byte range not sequential, Can't load tokenizer).

If you experience download hangs, corrupted file errors, or tokenizer-loading failures, disable Xet and fall back to standard HTTPS:

# One-off
HF_HUB_DISABLE_XET=1 provenancekit scan bigscience/bloom-560m

# Persistent (add to your shell profile)
export HF_HUB_DISABLE_XET=1

This also applies when running tests:

HF_HUB_DISABLE_XET=1 uv run pytest tests/

Tip: If the issue persists, try clearing the cached files for the affected model and retrying:

rm -rf ~/.cache/huggingface/hub/models--<org>--<model>

Streaming extraction is slow

The first run downloads safetensors shards from the Hub. Subsequent runs reuse the HuggingFace cache and complete in seconds.

Deep-signal fingerprints not installed

If scan shows a hint about missing deep-signal fingerprints:

provenancekit download-deepsignals-fingerprint

This enables the full weight-signal matching pipeline. Without deep signals, scan results rely on metadata and tokenizer signals only.

Scan refuses the database after an upgrade

Fingerprints are stamped with the extractor version that produced them, and scan will not compare features from different versions. If scan reports that the installed deep-signal fingerprints are from a different release:

provenancekit download-deepsignals-fingerprint --update

Scan returns few or no matches

  • Try lowering the threshold: --threshold 0.30
  • Try increasing top-k: --top-k 10
  • Verify deep signals are installed: provenancekit download-deepsignals-fingerprint --status

Notes and Limitations

  • Model comparisons depend on available model artifacts and configs on HuggingFace.
  • The scan command uses a bundled seed database. For custom deployments, use --db-root or PROVENANCEKIT_DB_ROOT to point to your own database directory.
  • Results provide strong evidence of provenance but are not absolute proof.
  • Weight signals require loading model weights into memory (or streaming for large models). First-run performance depends on network speed and model size.
  • FP8 and MLX checkpoints return no weight evidence yet, so their verdicts are unverified.
  • A match requires shared weights. Knowledge transfer without shared weights (e.g. phi-1.5 to phi-2) and distillation that changes dimensions are not detected.
  • See Known Limitations for more.

Contributing

Contributions are welcome. Please read the following before submitting a pull request:

Security

To report a security vulnerability, please see SECURITY.md.

License

This project is licensed under the Apache License 2.0.

Metadata

Release files for cisco-ai-provenance-kit 1.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for cisco-ai-provenance-kit 1.2.0
File Size Uploaded Provenance Actions
cisco_ai_provenance_kit-1.2.0.tar.gz 795.5 kB With provenance Details for cisco_ai_provenance_kit-1.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for cisco-ai-provenance-kit 1.2.0
File Interpreter ABI Platform Actions
cisco_ai_provenance_kit-1.2.0-py3-none-any.whl Python 3 none any Details for cisco_ai_provenance_kit-1.2.0-py3-none-any.whl

Total release size: 1.3 MB

Release files / cisco_ai_provenance_kit-1.2.0.tar.gz

Download URL cisco_ai_provenance_kit-1.2.0.tar.gz
Size 795.5 kB
Tags Source
SHA-256 checksum
How to use checksums
4020c98c3e374e63884597417fbfb891a9010fb024426170fea77e99ee921175
BLAKE2b-256 checksum
How to use checksums
4201c877806f6fa02b55e5b5324845cf1f16e4397309ee37282e6a3df490617e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.

Transparency log

Release files / cisco_ai_provenance_kit-1.2.0-py3-none-any.whl

Download URL cisco_ai_provenance_kit-1.2.0-py3-none-any.whl
Size 476.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
4dc30bdc0784aff1b63767669539ba1294798b1b0603b17b13cd4a1833c3a5c6
BLAKE2b-256 checksum
How to use checksums
bab318a8f874d74ca5933cd29b0a273424bb7827cad6c5568a2afc4b43cfe4d0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.

Transparency log

Security for cisco-ai-provenance-kit 1.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for cisco_ai_provenance_kit-1.2.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.2.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page