FlightDeck

Ship AI agents safely with release diffs, runtime evidence, and policy gates.

Local-first CLI + SQLite. Optional flightdeck serve exposes a small web UI and /v1 JSON API—data stays on your machine unless you change that.

Try it now — runs in 30 seconds, no account needed:

pip install flightdeck-ai
flightdeck demo  # end-to-end release diff in a temp workspace

How it works: you tag an agent build as a release, collect run evidence (cost, latency, errors), diff a baseline against a candidate, and then promote or rollback only when a policy gate says the numbers are acceptable — with an optional human approval step before the ledger moves.

Product snapshot

Marketing composite. See the demo video and actual screenshots for the shipped UI. Policy is threshold-based on rollups from ingested runs—not built-in PII scanners. Theming notes

Demo

Click the image to play the demo video · Download WebM

In ~20 seconds

1. Register immutable agent releases (release.yaml + bundle checksum).
2. Ingest run evidence (RunEvent JSONL or POST /v1/events).
3. Diff baseline vs candidate: cost, latency, errors, and confidence (optional pricing catalog lines on top).
4. Promote only when policy passes; optional human approval (request → confirm) before the ledger moves.

Why it exists

Small prompt or model changes can move cost, latency, and error rate in ways that are easy to miss. FlightDeck turns those into explicit promote decisions backed by ingested runs—before production pointers advance.

Who should use this?

• Platform or ML engineering teams shipping multiple LLM agents to production—especially after a cost or regression tied to a prompt or model change—who want a governed promote path without treating a tracing SaaS as their release gate.
• Regulated or compliance-sensitive teams (healthcare, fintech, and similar) where data residency, audit trails, and control over evidence and pricing data matter; local-first defaults and optional self-hosted flightdeck serve fit that posture.
• Teams that version agent builds (prompts, tools, model pins) and need a durable audit trail for what shipped and what ran.
• Engineers who want a straightforward workflow to answer “is this candidate safe to roll forward?” with numbers and policy, not only gut feel or spreadsheet checklists.

Example outcome

You ship a candidate whose prompt or model shifts slightly; under your tariffs the diff shows cost per run rising while policy caps spend. flightdeck release promote (or the HTTP promote path) stays blocked until you change the model, adjust policy with intent, or gather more evidence — not because CI is slow, but because the ledger says no. The examples/quickstart/ scenario uses two custom pricing YAMLs (baseline and candidate tariffs) to show a concrete cost delta across releases; flightdeck init alone seeds a bundled snapshot so your first cost-aware diff is not empty.

How it fits your stack

FlightDeck sits next to your agent runtime (not in the inference hot path): emit evidence, run flightdeck from a laptop or CI, gate promote with policy.

flowchart LR
  subgraph runtime [Your agent runtime]
    agent[Agent or service]
  end
  subgraph fd [FlightDeck workspace]
    ingest[Ingest RunEvents]
    ledger[(SQLite ledger)]
    diff[release diff]
    promote[promote or rollback]
  end
  subgraph automation [Automation]
    ci[CI job or operator]
  end
  agent -->|"JSONL or HTTP events"| ingest
  ingest --> ledger
  ledger --> diff
  diff --> ci
  ci -->|"policy pass"| promote

Fast start

After pip install flightdeck-ai (or uv tool install flightdeck-ai):

flightdeck demo

flightdeck demo runs the full quickstart ledger flow in a disposable temp workspace—no sed, no fixture paths—using examples/quickstart from your checkout or packaged flightdeck/_bundled_quickstart from PyPI.

Web UI (needs a workspace in the current directory):

flightdeck init
flightdeck serve

Open http://127.0.0.1:8765/. Same end-to-end checks CI uses: flightdeck-quickstart-verify (contributors: uv run flightdeck-quickstart-verify).

Install and smoke-test

User install (recommended):

pip install flightdeck-ai
flightdeck demo

Contributor / development install (uv):

uv sync --extra dev
uv run flightdeck --help
uv run flightdeck demo
uv run flightdeck-quickstart-verify

pip, Windows, CI: DEVELOPMENT.md

Web UI: uv run flightdeck serve → http://127.0.0.1:8765/ · docs/web-ui.md

The UI may show a loopback / no Bearer status line—that is what the server is doing, not something you execute. Bearer mode = you set a shared secret on the server (FLIGHTDECK_LOCAL_API_TOKEN) and the same value for the UI (VITE_FLIGHTDECK_LOCAL_API_TOKEN or web/.env.local with npm run dev). SECURITY.md · docs/http-api.md

Example workflow (policy + two tariffs)

Substitute release IDs in the JSONL or rely on flightdeck-quickstart-verify for the same checks CI runs.

flightdeck init
flightdeck pricing import examples/quickstart/pricing-baseline.yaml
flightdeck pricing import examples/quickstart/pricing-candidate.yaml
flightdeck policy set examples/quickstart/policy.yaml

BASELINE=$(flightdeck release register examples/quickstart/baseline-release)
CANDIDATE=$(flightdeck release register examples/quickstart/candidate-release)

sed "s/__BASELINE_RELEASE_ID__/${BASELINE}/g" examples/quickstart/baseline-events.jsonl > baseline-events.jsonl
sed "s/__CANDIDATE_RELEASE_ID__/${CANDIDATE}/g" examples/quickstart/candidate-events.jsonl > candidate-events.jsonl

flightdeck runs ingest baseline-events.jsonl
flightdeck runs ingest candidate-events.jsonl

flightdeck release diff "$BASELINE" "$CANDIDATE" --window 7d
flightdeck release promote "$BASELINE" --env local --window 7d --reason "initial baseline"
flightdeck release history --agent agent_support --env local

Bundled pricing from init is a convenience snapshot—flightdeck pricing import for production. docs/release-artifact.md · RELEASE_NOTES.md

More examples: examples/quickstart/ · examples/ci/ · examples/deploy/ · examples/integration/

Documentation

Published site (GitHub Pages): flightdeckdev.github.io/flightdeck — built from docs/ on each push to main (.github/workflows/pages.yml). Enable Pages → GitHub Actions in the repository settings if the site is not live yet.

Webhooks

FlightDeck ships HMAC-signed outbound webhooks for promote.succeeded, rollback.succeeded, and promote.blocked. Point them at any HTTPS endpoint (Slack incoming webhook, Discord, PagerDuty Events v2, Linear inbound webhook, your own relay) — FlightDeck owns the signing, not the integrations.

flightdeck webhook add \
  --url https://hooks.slack.com/services/T000/B000/XXXX \
  --event promote.succeeded --event rollback.succeeded \
  --description "prod release alerts"
# Save the printed secret — it will not be shown again.

flightdeck webhook list
flightdeck webhook test wh_<id>

Receivers verify each delivery by recomputing HMAC-SHA256(secret, raw_body) and comparing against X-FlightDeck-Signature: sha256=<hex> (GitHub convention). Full details: docs/cli.md · docs/http-api.md.

Contributing (quick CI match)

uv sync --frozen --extra dev
uv run python -m ruff check src tests
uv run python -m pytest
uv run flightdeck demo
uv run flightdeck-quickstart-verify
uv run flightdeck --help

Full gates (web static, schemas, e2e): DEVELOPMENT.md

Screenshots

Overview — releases + promotion ledger	Diff — policy PASS verdict

Runs — forensics with datalist search	Dark mode

License

Apache-2.0 — LICENSE · NOTICE

Canonical: github.com/flightdeckdev/flightdeck