fsd-disengagement-studio 0.4.0

Catalog FSD disengagements from Tesla Dashcam clips with a per-firmware regression dashboard.

fsd-disengagement-studio

Catalog and classify FSD disengagements from your saved Tesla Dashcam clips. Newly viable in May 2026 because firmware 2025.44.25+ embeds frame-perfect SEI metadata (including AutopilotState) in saved clips, and Tesla published the proto schema in teslamotors/dashcam (Dec 4, 2025).

Install

# From PyPI (once published)
pip install fsd-disengagement-studio              # core CLI + dashboard
pip install 'fsd-disengagement-studio[sei]'       # +protobuf/pandas to parse Dashcam SEI
pip install 'fsd-disengagement-studio[dev]'       # +pytest, ruff, httpx

Then try the dashboard with no Tesla and no API keys:

fsd-studio demo                                   # seeds + serves at http://127.0.0.1:8000/

The wave-3 spec at ../ideas/v2/wave3/fsd_disengagement_studio.md walks through the full design rationale.

What it does

For each saved Dashcam event:

1. Parse the SEI metadata stream to find frames where AutopilotState transitions from SELF_DRIVING/AUTOSTEER to NONE while vehicle_speed_mps > 1. (The speed gate filters out parking-lot manual-takeover events.)
2. For each disengagement, sample keyframes from front + back cams in a ±5s window around the transition.
3. Classify the cause via VLM (gpt-4o-mini default, ~$0.0008/event) into a structured DisengagementCause enum (phantom_brake, lane_confusion, occluded_sign, hov_decision, driver_pre_empt, traffic_light_hesitation, construction_zone, sun_glare, weather, unclear, other).
4. Persist to SQLite with firmware version + GPS + clip path + confidence.
5. Serve a FastAPI + HTMX dashboard showing per-firmware regression: which cause categories are improving, which are worsening across firmware versions.

Composability with the v3 stack

This project is the second consumer of tesla-clip-tools (the first being sentrytriage). The library provides:

• tesla_clip_tools.sources.tesla.TeslaSource — walks the canonical TeslaCam folder layout
• tesla_clip_tools.sampler.sample_keyframes — frame extraction via imageio (bundled ffmpeg)
• tesla_clip_tools.vlm.openai.OpenAIBackend — generic VLM call parameterised on response_format
• tesla_clip_tools.sei.iter_sei_payloads / sei_dataframe — the SEI byte-walker

This repo adds: the FSD-specific schema, the transition detector, the per-firmware regression dashboard, and the disengagement-cause classifier prompt.

Status — v0.4 alpha

What works today:

• Path B auto-saves the dashcam clip (added v0.4): fsd-studio listen --auto-save POSTs to Tesla's manual_save_dashcam_event Fleet API the moment a disengagement is queued, so Path C is guaranteed to find the matching .mp4 in SavedClips/ later. Per-vehicle 30s cooldown stops a re-broadcast burst from spamming the API; Fleet API errors are logged but never crash the listen loop. --mock-fleet runs a realistic dry-run without contacting Tesla. The dashboard /pending page surfaces the running auto-save counter ("Auto-saves issued: N").
• All three paths collaborate (v0.3): Path A (manual save), Path B (real-time trigger), and Path C (bulk backfill) compose as one system.
  • fsd-studio listen queues real-time disengagements to a JSONL log
  • fsd-studio backfill --pending-log <path> reads the log, matches each newly-classified DisengagementEvent to a queued record by ±300s (configurable) timestamp window, marks the record with the matched event's ID, and atomically rewrites the log
  • Dashboard GET /pending?unmatched_only=true (default) shows queued records still waiting for a clip; GET /pending/stale?stale_after_hours=24 surfaces missing-clip warnings (Path B fired but no clip was ever saved)
• Path B real-time trigger (v0.2): fsd-studio listen --ha-url ... --ha-token ... subscribes to HA's tesla_fleet_telemetry event bus, runs a stateful DisengagementDetector with cooldown + low-speed gate, appends each detected transition to a JSONL log. Detector handles both explicit autopilot_state transitions and inferred-from-self_driving_miles plateaus. Fully testable with MockTelemetryStream.
• Pure-Python find_disengagements() finds engaged → disengaged transitions in any sequence of FrameTelemetry tuples (testable without a Tesla via mocked sequences)
• DisengagementClassification Pydantic schema with 11 cause categories
• DisengagementEvent SQLModel + Store with upsert, list, and per-firmware aggregation
• fsd-studio CLI with classify, backfill, serve commands
• fsd-studio serve starts a FastAPI + HTMX + ECharts dashboard at localhost:8000
• Test suite covers the schema, transition detector, store aggregation, and FastAPI routes (TestClient — no real server needed)

What needs a real Tesla + the [sei] extra:

• iter_disengagements_from_clip() — calls sei_dataframe(mp4_path) which requires the [sei] extra (pip install 'fsd-disengagement-studio[sei]') plus a one-time protoc --python_out=. dashcam.proto to generate dashcam_pb2.py in tesla-clip-tools.
• The backfill CLI command — needs real saved Dashcam clips with SEI to do meaningful work.

What's coming in v0.5+ (per wave-3 spec):

• Per-route map (Leaflet) showing where disengagements cluster
• Per-cause Sankey across firmware versions
• Regression alerts (push when a cause spikes vs trailing N-week baseline)
• Reviewer-feedback fine-tuning (thumbs-up/down on each event tunes a small classifier on top of the VLM embeddings)

Path B auto-save (v0.4) -- usage

export TESLA_FLEET_ACCESS_TOKEN=...   # from your Fleet API OAuth flow
export TESLA_VEHICLE_ID=...           # the vehicle id (or VIN) to target
export HA_URL=http://homeassistant.local:8123
export HA_LONG_LIVED_TOKEN=...

# Auto-save is on by default; the matching clip lands in SavedClips/.
uv run fsd-studio listen

# Dry-run (no real Fleet API call):
uv run fsd-studio listen --mock-fleet --vehicle-id rehearsal

# Disable entirely (v0.3 behavior):
uv run fsd-studio listen --no-auto-save

The dashboard's /pending page renders an "Auto-saves issued: N" stat panel fed by a sidecar <pending-log>.autosave-stats.json that the listener writes after each event. JSON consumers can hit GET /api/autosave-stats.

Quick start (without a Tesla)

git clone https://github.com/Raymondriter/fsd-disengagement-studio.git
cd fsd-disengagement-studio
uv sync --extra dev
uv run pytest                                  # 15+ tests, no fixtures needed
uv run fsd-studio serve --port 8000           # empty dashboard at http://localhost:8000

Quick start (with a Tesla)

uv sync --extra sei --extra dev
# Generate the SEI proto module (one-time):
cd ../tesla-clip-tools/src/tesla_clip_tools
protoc --python_out=. dashcam.proto
cd -

export OPENAI_API_KEY=sk-...
uv run fsd-studio backfill /path/to/SavedClips --firmware 2026.14.3
uv run fsd-studio serve

Important caveats

• SEI is only in Dashcam clips (SavedClips, RecentClips), not Sentry events. The wave-3 SEI deep-dive confirmed this. So this tool reads from SavedClips/ (and RecentClips/), NOT SentryClips/. The SavedClips/ folder layout is identical to SentryClips/ — tesla_clip_tools.sources.tesla.TeslaSource handles either.
• Path B (Fleet Telemetry triggered save) is fragile in v0.1. Tesla hasn't shipped a first-class autopilot-state transition signal; it has to be inferred from SelfDrivingMilesSinceReset rate-drop. v0.1 ships only Path A (manually-saved clips) + Path C (bulk historical backfill). Path B is v0.2 opt-in.
• Legal risk for individual-owner use is low. Owner owns vehicle-generated data per Tesla's Privacy Notice + Fleet API Terms; teslafsdtracker.com and MATT3R K3Y have published cross-owner FSD-disengagement data since 2020 with no Tesla legal action; Tesla itself publishes aggregate data in its FSD Vehicle Safety Report. Single-VIN by default. Footer disclaimer in the dashboard reminds users not to generalize from n=1 in public posts.

Screenshots

The dashboard rendered against synthetic demo data (fsd-studio demo):

Changelog

See CHANGELOG.md. Versions follow Keep a Changelog and the project uses SemVer.

License

MIT. See LICENSE.

CI

GitHub Actions runs ruff + pytest on Python 3.12 and 3.13 against every push and PR. See .github/workflows/ci.yml. Until tesla-clip-tools is published to PyPI, the standalone CI strips the [tool.uv.sources] table and resolves it as a regular dependency; the workspace-level monorepo CI at C:\Dev\tesla\.github\workflows\ci.yml keeps using the path-editable sibling.