turnkey 0.1.0

---

Quick Start | Documentation | Notebooks | Examples | Ask DeepWiki | Community

About

Turnkey shortens the path from “I have a detector idea” to “I can run it, inspect what happened, and audit the result.” It gives detector authors one runtime for paired evaluation, typed model signals, inspectable event traces, and reproducible audit artifacts.

The first run is deliberately small: it runs on CPU, downloads no model, and requires no API key. GPU-backed methods and remote judges are optional layers, not prerequisites for understanding the system.

Core Features

• Bring your own detector: scaffold with turnkey init, then load path.py:build without forking Turnkey or editing a central registry.
• Compare paired paths: evaluate the reference and intervention paths on the same selected and attacked input.
• Ask for model state: typed providers expose prompt logprobs, hidden states, gradients, and other method-owned signals with cache and lifecycle accounting.
• Debug the failure: connect a case to its policy, target, provider, timing, cache, and measured-forward events in the CLI or a standalone HTML report.
• Audit the claim: recompute metrics and verify redaction, source identity, event relationships, and public artifacts instead of trusting a summary.
• Start without infrastructure: the dummy backend and fixture dataset make the complete loop runnable without GPU access.

dataset -> attack -> target -> judge -> detector policy -> metrics -> audit

Quick Start

Install the packaged release from PyPI with pip install turnkey, or install from source for development.

git clone https://github.com/lijrjyan/turnkey.git
cd turnkey
uv sync --extra dev
uv run turnkey run --config configs/runs/smoke.yaml

The run prints an output directory containing four public artifacts:

Artifact	Purpose
run.json	Resolved configuration, source identity, and environment
cases.jsonl	Redacted paired reference/intervention outcomes
events.jsonl	Target, policy, provider, cache, and timing events
metrics.json	Safety, utility, and measured runtime cost

Inspect one failure category and independently audit the bundle:

uv run turnkey inspect outputs/<run-id> --category missed_harm
uv run turnkey audit outputs/<run-id>
uv run turnkey report outputs/<run-id> --html report.html

An empty JSON list from turnkey audit means the bundle passed the current artifact checks. It is not a claim that a detector is scientifically validated or production-ready.

Start an external detector without copying boilerplate by hand:

uv run turnkey init my-detector
cd my-detector
uv run turnkey dev ./detector.py:build --max-samples 4

Docker

Run the same CPU smoke with nothing installed but Docker:

docker build -t turnkey https://github.com/lijrjyan/turnkey.git
docker run --rm -v "$PWD/outputs:/app/outputs" turnkey

The default command runs configs/runs/smoke.yaml and writes the four core artifacts under outputs/. Any CLI invocation works the same way, for example docker run --rm turnkey detector list. For GPU-backed HF models, build the large image with --target full and run with --gpus all.

Included Method Examples

Turnkey includes compact integrations that exercise different runtime capabilities:

Method	Signal or control pattern
allow_all	No-op baseline for pipeline and metric sanity
keyword	Local prompt rule for the first detector and CI smoke
smoothllm	Perturbation and repeated target calls
jailguard	Mutation and divergence
gradsafe	Gradient provider
perplexity	Typed prompt-logprob filter
self_exam	Output-side deterministic self-screening
llamaguard	Separate Llama Guard input classifier
rcs_toy	CPU-safe representation proxy
rcs	Hidden-state provider and calibration

These integrations demonstrate the harness. They are not leaderboard results or claims of strict paper reproduction; original model, dataset, calibration, and license conditions still apply.

Documentation

Choose the shortest path for what you are trying to do:

Goal	Guide
Install and verify the source checkout	Installation
Complete the first run-to-audit loop	Ten-minute quickstart
Add a detector without changing Turnkey	Build an external detector
Trace a missed or overblocked case	Debug a run
Understand policies and model signals	Runtime concepts
Compare model-based safety filters	Model-based filters
Learn the public artifact contract	Artifact concepts
Use the command line precisely	CLI reference
Learn interactively	Executable notebooks

Build the complete local documentation site with:

uv sync --extra docs
uv run mkdocs serve

Development

Development happens on dev; main is the stable branch and only receives reviewed changes.

uv sync --extra dev
uv run ruff check .
uv run pytest -q
uv run make smoke

Optional model and judge dependencies are isolated from the CPU-first path:

uv sync --extra dev --extra hf --extra judges

Community & Support

• Use GitHub Issues for confirmed bugs, documentation gaps, and scoped feature proposals.
• After the public repository has been indexed, use Ask DeepWiki to explore the architecture and ask code-grounded questions.
• Never include credentials, private prompts, unpublished datasets, or personal data in issues, logs, examples, or generated artifacts.
• Do not open a public issue for a vulnerability or suspected data leak. Use GitHub's private vulnerability reporting channel when it is available.

Project Status

Turnkey is an early source preview. Public interfaces, artifact schemas, and method integrations can change before the first functional PyPI release. Pin a Git commit for reproducible work and inspect the generated run.json before comparing runs.

The README badge and .devin/wiki.json prepare DeepWiki integration, but the free public index cannot read a private GitHub repository. Ask DeepWiki becomes active after the formal repository is public and indexed; private indexing requires a connected Devin account.

License

Turnkey is licensed under the Apache License 2.0. Third-party methods and datasets retain their own terms; see third_party/README.md.