mlprints 0.1.0a3

A state-of-the-art framework for LLM fingerprinting and adversarial testing

MLprints

A framework for generating, training, and verifying LLM fingerprints.

Quick start

1 - Install

uv sync

Then use uv run mlprints … for the examples below, or activate the environment uv created.

Alternatively, in a virtual environment (with Python ≥ 3.11):

pip install -e .

2 - Set experiments directory

export MLPRINTS_EXPERIMENTS_DIR=/path/to/experiments

Alternatively, pass --experiments-dir to mlprints generate.

Some assets are downloaded on demand and cached locally (no large bundled data tree).

The currently registered fingerprint algorithm is Perinucleus, with training support. Available verifiers are string matching and a watermark z-test.

CLI

Command	Role
mlprints generate CONFIG.yaml	Generate fingerprints; add --train to train in the same run when supported.
mlprints train CONFIG.yaml --fingerprints-dir DIR	Train on a previous generate output (DIR must contain fingerprints.yaml).
mlprints measure utility CONFIG.yaml	Measure model utility with LightEval tasks.
mlprints verify CONFIG.yaml --fingerprints DIR --model MODEL	Verify a model using saved fingerprints.

Run mlprints <command> --help for command-specific options.

Generate fingerprints example

mlprints generate src/mlprints/configs/fingerprint/perinucleus_config.yaml

Generate and train in one step (algorithms that support training):

mlprints generate src/mlprints/configs/fingerprint/perinucleus_config.yaml --train

Under your experiments root, each run writes:

• fingerprints/{algo}/{timestamp}/fingerprints.yaml
• fingerprints/{algo}/{timestamp}/config.yaml
• fingerprints/{algo}/{timestamp}/metadata.yaml

Train fingerprints example

mlprints train src/mlprints/configs/fingerprint/perinucleus_config.yaml \
  --fingerprints-dir /path/to/fingerprints/{algo}/{timestamp}

Checkpoints and training metadata are written under that fingerprint directory (e.g. trained/.../checkpoints/).

Measure utility example

mlprints measure utility src/mlprints/configs/utility/leaderboard_config.yaml

The config may use native LightEval task expressions in evaluation.tasks (comma-separated, optional |fewshot).

Verify fingerprints example

mlprints verify src/mlprints/configs/verify/match_config.yaml \
  --fingerprints /path/to/fingerprints/{algo}/{timestamp} \
  --model /path/to/model \
  --apply-chat-template

Verification results are written under verification/{timestamp}/.

Grid configs

Wrap any parameter in {grid: [...]} to run every combination:

learning_rate:
  grid: [1.0e-5, 2.0e-5]

Generation expands grids under algo.params; training expands algo.training; verification expands verifier.params and inference. Ordinary YAML lists remain unchanged. Use --skip-existing to reuse matching runs.

How to add a new fingerprint

1. Implement generation (and optional training) in src/mlprints/fingerprint/.
2. Register in FINGERPRINT_ALGOS in src/mlprints/common/fingerprints.py.
3. Add src/mlprints/configs/fingerprint/<name>_config.yaml with algo.name, algo.params, and optionally algo.training if training is required.
4. Implement and register a verifier when the scheme needs one.

Contributing

Issues and pull requests are very welcome. For questions, contact [e-mail].

Disclaimer: Most of the code pushed has been either handwritten, or written with AI on a first passage only then to be severly revised and edited for performance and clarity over countless hours. The end result is a repo that is meant to be very flexible, readable and usable by AI and humans alike. Therefore, any contributions that worsen the standard set will be asked to be revised. Any contributions that improve the standard are very welcome (and such is continuously improved). It is both a means as much as an ends (instead of only the first, in which case we would have spared ourselves plenty of development time).

License

Released under the GNU Affero General Public License v3.0 or later. Full text in LICENSE.