quickdistill 0.1.4

Fast and easy toolkit for distilling AI models

QuickDistill

Fast and easy toolkit for evaluating AI models (and training them in the future)

QuickDistill provides an intuitive web UI for the complete model distillation workflow:

• 📊 View and filter Weave traces from your LLM calls
• 🎯 Export strong model outputs as test sets
• 🔬 Run weak models on strong outputs
• ⚖️ Evaluate similarity using LLM judges
• 📥 Download evaluation datasets for analysis

Installation

pip install quickdistill

Quick Start

Launch the UI in your current directory:

quickdistill launch

This will:

• Start the QuickDistill server on http://localhost:5001
• Open the Trace Viewer in your browser
• Create local directories for projects, exports, and results

Requirements

Set these environment variables:

export WANDB_API_KEY="your_wandb_key"          # Required for W&B Inference
export OPENROUTER_API_KEY="your_openrouter_key"  # Optional for OpenRouter models

Get your keys:

• W&B: https://wandb.ai/settings
• OpenRouter: https://openrouter.ai/keys

Usage

1. Fetch Weave Traces

Enter your Weave project name (e.g., username/project-name) and click "Fetch New Project" to load traces from W&B.

2. Export Strong Model Outputs

• Filter traces by model or operation
• Select traces to use as ground truth
• Export to create a test set

3. Run Weak Models

• Select a strong model export
• Choose W&B models from the list or enter custom OpenRouter models
• Run inference to generate weak model outputs

4. Create Judges

Navigate to /judge to create LLM judges:

• Scalar judges: Rate similarity on a numeric scale (1-5)
• Boolean judges: Determine if outputs are correct/incorrect

5. Run Evaluations

• Select weak model results
• Choose a judge
• Run evaluation and view results in Weave

6. Download Datasets

Click "Download" next to any weak model result to get a clean JSON dataset with:

[
  {
    "input": "question text...",
    "strong_model": "model-name",
    "strong_output": "strong response...",
    "weak_model": "model-name",
    "weak_output": "weak response..."
  }
]

CLI Options

quickdistill launch                    # Launch on default port 5001
quickdistill launch --port 8080        # Launch on custom port
quickdistill launch --no-browser       # Don't open browser automatically
quickdistill launch --debug            # Run in debug mode

Project Structure

When you run quickdistill launch, it creates these directories ~/.cache/quickdistill:

your-project/
├── projects/                  # Cached Weave traces by project
│   └── username_project/
│       └── traces_data.json
├── strong_exports/            # Exported strong model test sets
│   └── model-name_10traces.json
├── weak_model_*.json          # Weak model inference results
├── judges.json                # Saved judge configurations
└── evaluations/               # Evaluation results

Features

• Multi-provider support: Works with W&B Inference and OpenRouter
• Flexible judging: Create custom LLM judges or use pre-built ones
• Trace filtering: Filter by model, operation, or custom criteria
• Batch operations: Run multiple models and evaluations in parallel
• Export formats: Download clean datasets for further analysis
• Project isolation: Each Weave project is cached separately

Development

Install in development mode:

git clone https://github.com/byyoung3/quickdistill.git
cd quickdistill
pip install -e .

License

MIT

Author

Brett Young ([e-mail])