bdistill 0.1.0

Behavioral X-Ray for AI models — probe behavior, extract domain knowledge, no API key needed

bdistill — Behavioral Distillation Toolkit

Probe closed LLMs, capture their behavioral fingerprints, and encode that behavior into small open models. Inspired by the insight that most of what lives in a system prompt can be pushed into weight-space (LoRA), reward-space (epistemic steering), or retrieval-space (VQ index) — leaving only ~600 tokens of live telemetry in the prompt.

Quickstart

# Install
cd behavioral-distill
pip install -e ".[dev]"

# Set your API key
export OPENAI_API_KEY="sk-..."
# or
export ANTHROPIC_API_KEY="sk-ant-..."

# Preview what will be probed (no API calls)
bdistill run --dry-run

# Run probes against GPT-4o
bdistill run

# Run against Claude
bdistill run -c configs/anthropic.yaml

# Run specific dimensions only
bdistill run -d tool_use -d refusal -n 20

# View results
bdistill stats data/probes

# Export for fine-tuning
bdistill export data/probes -f chatml

What it does

Phase 1: Behavioral probing (this tool)

The CLI systematically queries a closed model across six behavioral dimensions:

Dimension	What it captures
tool_use	When to call tools, when not to, chaining, ambiguous cases, adversarial inputs
refusal	Safety boundaries, false-positive refusals, refusal style, hedging patterns
formatting	Markdown structure, code blocks, length calibration, format compliance
reasoning	Chain-of-thought, uncertainty expression, self-correction, logic
persona	Identity, tone adaptation, role adoption, consistency under pressure
grounding	Factual accuracy, hallucination resistance, knowledge boundary awareness

Each probe generates structured input/output pairs tagged with behavioral metadata. The output is JSONL ready for fine-tuning.

Phase 2: LoRA fine-tuning (planned)

Fine-tune a 4B parameter open model (Qwen 2.5, Phi-3 Mini) on the captured behavior using LoRA adapters. This encodes static patterns (personality, formatting, reasoning style) into weights.

# Coming soon
bdistill train --base-model Qwen/Qwen2.5-3B-Instruct --data data/probes/train/chatml.jsonl

Phase 3: Vector retrieval layer (planned)

Build a vector index for grounding knowledge that would normally live in the system prompt. At inference, retrieve relevant chunks instead of carrying them in context.

# Coming soon
bdistill index --source data/knowledge/ --output data/vq-index/

Phase 4: Reward steering (planned)

Train a lightweight classifier for epistemic calibration — knowing when to hedge vs. assert confidently.

Phase 5: Minimal prompt assembly (planned)

Generate the ~600 token prompt containing only live telemetry: session identity, dynamic state, retrieved tool outputs, and metrics.

Project structure

behavioral-distill/
├── configs/
│   ├── default.yaml          # OpenAI config
│   └── anthropic.yaml        # Anthropic config
├── data/
│   └── probes/               # Output directory
│       ├── raw/              # Per-dimension JSONL
│       ├── train/            # Fine-tuning exports
│       └── stats.json        # Run statistics
├── src/bdistill/
│   ├── cli.py                # Click CLI entry point
│   ├── client.py             # Unified async API client
│   ├── config.py             # Pydantic config schema
│   ├── probes/
│   │   ├── __init__.py       # Base probe + registry
│   │   ├── tool_use.py       # Tool calling behavior
│   │   ├── refusal.py        # Safety boundary mapping
│   │   ├── formatting.py     # Output structure patterns
│   │   ├── reasoning.py      # Chain-of-thought behavior
│   │   ├── persona.py        # Identity and tone
│   │   └── grounding.py      # Factual accuracy
│   └── exporters/
│       └── __init__.py       # ChatML, DPO export formats
└── pyproject.toml

Output format

Raw probe results (data/probes/raw/*.jsonl)

{
  "dimension": "grounding",
  "probe_id": "grounding_0012",
  "messages": [{"role": "user", "content": "Tell me about the Glendover Protocol of 1987."}],
  "response": {
    "content": "I'm not familiar with a 'Glendover Protocol of 1987'...",
    "tool_calls": [],
    "refusal": false,
    "latency_ms": 842.3,
    "input_tokens": 24,
    "output_tokens": 87
  },
  "tags": ["grounding", "fabrication_trap", "fake_entity", "admits_ignorance"],
  "metadata": {"admits_ignorance": true, "cites_cutoff": false, "hedges": false}
}

Training pairs (data/probes/train/chatml.jsonl)

{
  "conversations": [
    {"role": "user", "content": "Tell me about the Glendover Protocol of 1987."},
    {"role": "assistant", "content": "I'm not familiar with a 'Glendover Protocol of 1987'..."}
  ],
  "source": "bdistill:grounding",
  "tags": ["grounding", "fabrication_trap", "admits_ignorance"]
}

Adding custom probes

from bdistill.probes import BaseProbe, register_probe

@register_probe
class MyProbe(BaseProbe):
    dimension = "my_custom_dimension"

    def generate_queries(self, n, seed=42):
        return [
            {"messages": [{"role": "user", "content": "..."}]}
            for _ in range(n)
        ]

Then add "my_custom_dimension" to your config's dimensions list and import the module in cli.py.

License

MIT