puffin-finetune-studio

A visual studio for fine-tuning open LLMs, backed by a production MLOps engine.

Do the whole workflow (data, training, evals, deploy, monitoring) in your browser or by chatting with an AI. Under the hood is a reproducible, config-driven engine (SFT / LoRA / DPO with eval gates, a model registry, and cloud-portable serving) that you can also drive headless from the CLI.

Quickstart · The Studio · Why · Under the hood · Docs · Contributing

TL;DR

pip install "puffin-finetune-studio[copilot]"
finetune-copilot          # opens Puffin Studio in your browser

That one command starts the engine and the web UI, waits for both, and drops you into a point-and-click (or chat-driven) workspace for the whole fine-tuning lifecycle. Everything the studio does, the engine underneath also exposes as the puffin CLI and a Python API, so you can automate the exact same workflow in CI.

The reusable contract: a new project is a new config plus a new dataset plus new evals. Not a new platform every time.

Why a studio?

Fine-tuning usually lives in a pile of one-off scripts and notebooks only the ML engineer can drive, on top of a training/serving stack that quietly drifts out of sync. Puffin makes it a studio anyone on the team can use, backed by an engine that keeps every run reproducible and production-safe.

Quickstart

Prerequisites

• Python 3.11+
• Node.js 18.18+ (for the studio web UI)
• ~1 GB free disk for the smoke model; no GPU required for the smoke path

Open the studio

pip install "puffin-finetune-studio[copilot]"

# optional: unlock the AI chat (the studio itself works without it)
export ANTHROPIC_API_KEY="sk-ant-..."     # PowerShell: $env:ANTHROPIC_API_KEY="sk-ant-..."

finetune-copilot                          # starts everything, opens your browser

finetune-copilot installs the frontend's npm deps on first run, launches the engine + UI, and opens the studio. finetune-copilot doctor checks your environment; finetune-copilot --prod serves a prebuilt UI from a single port with no Node.js at runtime.

No ANTHROPIC_API_KEY? The chat also drives any local agent CLI you already have installed and authed (Claude Code, Codex, Gemini, Qwen, OpenCode, Cursor, GitHub Copilot).

Or drive the engine headless (CLI / CI)

Prefer YAML and a terminal, or automating in CI? Clone the repo and run the same golden path without the UI:

git clone https://github.com/m-ahmed-elbeskeri/puffin-finetune-studio
cd puffin-finetune-studio
cp .env.example .env
make setup           # engine + dev extras
make train-smoke     # tiny CPU smoke train (SmolLM2-135M, < 1 min)
make gate            # exits non-zero if thresholds are missed
make serve           # FastAPI on :8080

(Windows: the same targets via .\make.ps1 <target>.) The project ships with no training data. Drop your JSONL into data/raw/, or upload it in the studio's Data page; a 20-row reference dataset lives at tests/fixtures/example.jsonl.

The Studio

Puffin Studio (a Next.js + FastAPI app, opened with finetune-copilot) is the main way to use the platform: the whole fine-tuning lifecycle as pages you click through, with an AI copilot that can do any of it for you.

• Train Studio (/train): pick a curated recipe (smoke → style tune → domain adaptation → QLoRA → full fine-tune → DPO) or open the full knob editor, with a Beginner/Intermediate/ Advanced toggle, YAML preview, GPU-aware warnings, and a smoke-first launch.
• A page for every stage: Data, Train, Runs, Evaluate, Deploy, Monitor, Playground, each with one-click actions (audit data, run the pipeline, run evals + gate, push/promote, diagnose drift). Ctrl/Cmd+K opens a page-aware command bar.
• Ask the AI instead: the copilot has tool-use access to the whole engine, so you can just say "run a smoke train then the gate." Chat through the Anthropic or OpenAI APIs, or through any local agent CLI you already have (Claude Code, Codex, Gemini, Qwen, OpenCode, Cursor, GitHub Copilot), auto-detected in the model picker.
• One command, clean teardown: finetune-copilot starts the engine + UI and opens your browser; Ctrl+C stops both.

Train Studio: recipes or a full knob editor, GPU-aware

Monitor: latency, throughput, quality, drift

Under the hood: the engine

The studio is a thin, friendly surface over a config-driven engine. Every button, chat action, and CLI command drives the same YAML-configured pipeline below, and training and serving share one feature layer so they can never drift apart (the single most common fine-tuning failure).

┌──────────────────────────────────────────────────────────┐
│                       Configs (YAML)                     │
│   data.yaml │ train.yaml │ eval.yaml │ deploy.yaml       │
│        observability.yaml  +  profiles/<provider>.yaml   │
└─────────────────┬─────────────────────┬──────────────────┘
                  ▼                     ▼
        ┌──────────────────┐  ┌──────────────────┐
        │  Data pipeline   │  │ Features (shared │
        │  ingest→validate │  │ prompt/chat/     │
        │  →redact→dedupe  │  │ tokenizer/RAG/   │
        │  →split→card     │  │ schemas)         │
        └────────┬─────────┘  └────────┬─────────┘
                 ▼                     │
        ┌──────────────────┐           │
        │    Training      │ ◀─────────┘
        │  SFT / LoRA / DPO│
        │  (TRL + PEFT)    │
        └────────┬─────────┘
                 ▼
        ┌──────────────────┐
        │   Evaluation     │
        │  task / safety / │
        │  regression /    │
        │  latency  → gate │
        └────────┬─────────┘
                 ▼
        ┌──────────────────┐
        │  Model registry  │  MLflow by default; Vertex / SageMaker / AzureML adapters
        └────────┬─────────┘
                 ▼
        ┌──────────────────┐
        │   Serving        │ ◀─── shared features ─┐
        │  FastAPI +       │                       │
        │  Transformers /  │                       │
        │  vLLM            │                       │
        └────────┬─────────┘                       │
                 ▼                                 │
        ┌──────────────────┐                       │
        │   Monitoring     │                       │
        │  logs / metrics  │                       │
        │  / drift / judge │ ──── feedback loop ───┘
        └──────────────────┘

Dataset format

{
  "id": "ticket-00001",
  "source": "support-zendesk-2024-q4",
  "messages": [
    {"role": "system",    "content": "You are a helpful customer support agent."},
    {"role": "user",      "content": "How do I reset my password?"},
    {"role": "assistant", "content": "Click 'Forgot password' on the sign-in page..."}
  ],
  "quality_score": 0.92,
  "license": "internal",
  "contains_pii": false
}

{
  "prompt": "Explain transformers to a 5-year-old.",
  "chosen": "Imagine a robot that pays attention to what's important...",
  "rejected": "Transformers are a deep learning architecture introduced in...",
  "reason": "chosen is age-appropriate"
}

JSON Schemas live in data_contracts/. Records that fail validation are blocked before training.

Fine-tuning methods

Default recommendation: SFT + LoRA. Switch by editing configs/train.yaml, no code change.

Evaluation gates

The promotion gate runs four eval layers and fails the build if thresholds are missed:

• Task exact match, F1, ROUGE-L, JSON validity, tool-call correctness, custom rubric.
• Safety prompt injection, jailbreak, data leakage, toxicity, memorization. OWASP-LLM-aligned.
• Regression a golden set of previously-fixed bugs and high-value queries.
• Latency / cost p50 / p95 / p99 latency, tokens/sec, cost per 1k requests.

# configs/eval.yaml
gates:
  min_task_score: 0.85
  min_improvement_over_baseline: 0.05
  max_safety_failures_critical: 0
  max_regression_failures: 0
  min_json_validity: 0.995
  max_p95_latency_ms: 2500

make gate (or the Copilot's one-click gate) exits non-zero on failure.

Deployment

Pick a profile in configs/deploy.yaml (platform.provider); the rest of the code is unchanged.

Repository layout

puffin-finetune-studio/
├── configs/            YAML configs (data, train, eval, deploy, observability)
├── profiles/           Provider profiles (local, gcp_vertex, aws_sagemaker, ...)
├── data_contracts/     JSON schemas for SFT / preference data
├── eval_sets/          Golden, safety, regression, latency JSONL
├── src/llmops/
│   ├── common/         Config, logging, tracking, versioning
│   ├── data/           Ingest, validate, redact, dedupe, split, card
│   ├── features/       SHARED with serving: chat template, prompt builder, schemas
│   ├── training/       SFT/LoRA, DPO, merge, push
│   ├── evaluation/     Task / safety / regression / latency / gate
│   ├── serving/        FastAPI + OpenAI-compatible + guardrails
│   ├── monitoring/     Logs, quality, drift
│   ├── providers/      local | gcp | aws | azure | kubernetes
│   └── cli.py          `puffin` entry point
├── copilot/            Next.js + FastAPI dashboard (optional, install with [copilot])
│   ├── frontend/       Next.js 15 / React 19 / Tailwind / Recharts
│   └── backend/        FastAPI tool-use loop + the finetune-copilot launcher
├── infra/              Dockerfiles + per-cloud Terraform
├── pipelines/          DVC + GitHub Actions + Vertex + Argo
├── tests/              unit / data_quality / evaluation / serving / security
└── pyproject.toml

Documentation

Contributing

Contributions are very welcome. Please read CONTRIBUTING.md for the dev setup, coding standards (ruff + mypy + pytest), and PR checklist, and CODE_OF_CONDUCT.md for community expectations.

make setup && make lint && make test-fast   # green before you push

Good first issues are labelled good first issue.

Security

Found a vulnerability? Please do not open a public issue. See SECURITY.md for private disclosure.

Roadmap

See open issues and the Discussions board. Near-term: GRPO/KTO recipe polish, hosted demo, one-click cloud submit from the Copilot.

License

Apache 2.0. Use it, fork it, ship it.

Acknowledgements

Built on the shoulders of Transformers, TRL, PEFT, FastAPI, Next.js, and MLflow.