Trajectory Distiller

Convert agent traces from multiple formats into training datasets for fine-tuning.

Installation

pip install trajectory-distiller

Supported Input Formats

Supported Output Formats

Quick Start

Distill Traces

# Convert glint traces to OpenAI chat format (auto-detected)
distill input.jsonl --format openai_chat --output train.jsonl

# Convert armand0e format explicitly
distill input.jsonl --input-format armand0e --format sharegpt -o train.jsonl

# Convert to alpaca format
distill input.jsonl --format alpaca -o alpaca_train.jsonl

Filter Traces

# Filter to records using specific tools
distill filter traces.jsonl --tool bash --tool edit

# Filter by error rate and quality
distill filter traces.jsonl --min-errors 0.1 --min-quality 0.5

# Filter by session length
distill filter traces.jsonl --min-turns 5 --max-turns 50

# Combine filters and save
distill filter traces.jsonl --tool bash --min-quality 0.3 -o filtered.jsonl

Split Dataset

# Split into 95/5 train/val
distill split traces.jsonl --train-ratio 0.95 --val-ratio 0.05

# Stratify by tool distribution
distill split traces.jsonl --stratify-by tool --output-dir splits/

# Split with test set
distill split traces.jsonl --train-ratio 0.8 --val-ratio 0.1 --test-ratio 0.1

Fable5 Dataset Usage

# Glint dataset
distill glint_traces.jsonl --format openai_chat -o glint_openai.jsonl

# armand0e dataset
distill armand0e_data.jsonl --input-format armand0e --format alpaca -o armand0e_alpaca.jsonl

# vfable dataset
distill vfable_traces.jsonl --input-format vfable --format sharegpt -o vfable_sharegpt.jsonl

# opencoven dataset
distill opencoven_pairs.jsonl --input-format opencoven --format openai_chat -o opencoven_openai.jsonl

# victor dataset
distill victor_pairs.jsonl --input-format victor --format conversation -o victor_conv.jsonl

Programming API

from trajectory_distiller import Distiller, FormatConverter, TraceFilter, DataSplitter

# Distill traces
distiller = Distiller()
records = distiller.distill("traces.jsonl", output_format="openai_chat")

# Filter traces
trace_filter = TraceFilter()
filtered = trace_filter.filter_by_tool(records, tools=["bash", "edit"])
filtered = trace_filter.filter_by_quality(filtered, min_quality_score=0.5)

# Convert formats
converter = FormatConverter()
alpaca_records = converter.to_alpaca(records)
sharegpt_records = converter.to_sharegpt(records)

# Split data
splitter = DataSplitter()
splits = splitter.split(records, train_ratio=0.95, stratify_by="tool")
splits.save("output/")
print(splits.stats())

License

MIT

Ecosystem

Part of the FableForge ecosystem — 21 open-source projects built from 210K real agent traces: