🔥 Z-Temper

Autonomous dataset tempering engine for synthetic data scaling.

Z-Temper closes the loop between diagnostics and training data. It reads diagnostic reports and blueprints produced by Project Peony (peony-core), pulls seed exemplars through DataFlux (dataflux-core), and drives Google Gemini through a generate → critique → filter loop to synthesize clean, ready-to-train datasets in ShareGPT or Alpaca format.

---

How it fits together

Peony diagnostic report / blueprint
            │
            ▼
   BlueprintParser  ──►  target capabilities, data recipes, edge cases
            │
            ▼
  SeedDatasetLoader  ◄── dataflux-core / local .json / .jsonl
            │
            ▼
      BatchGovernor
            │
   ┌────────┴────────┐
   ▼                 ▼
LoopGenerator    QualityCritic
(Gemini calls,   (structural checks,
 retry/backoff)   min length, dedup)
   │                 │
   └────────┬────────┘
            ▼
  ready_to_train.jsonl  (ShareGPT / Alpaca)

• BlueprintParser — validates and parses a Peony orchid_dataset_blueprint.json (or a full diagnostic report containing one) into target capabilities, synthesis recipes, and edge cases to cover.
• SeedDatasetLoader — loads seed exemplars via dataflux-core when available, falling back to local .json/.jsonl files.
• LoopGenerator — calls the Gemini API with a structured Pydantic response schema (ShareGPTBatch / AlpacaBatch) and retries transient API errors with exponential backoff.
• QualityCritic — rejects malformed samples, enforces a minimum character length, and suppresses duplicates via content hashing.
• BatchGovernor — orchestrates the loop end-to-end: generate a batch, filter it, stream valid samples to disk immediately, repeat until the target count is hit. Shows a live progress bar via rich.
• TemperEngine — the top-level facade tying all of the above together; this is what you'd typically import.

---

Installation

Requires Python ≥ 3.14.

pip install z-temper

From source (development)

Dependency management is via uv.

git clone https://github.com/JustZeo/Z-Temper.git
cd Z-Temper
uv sync
uv pip install -e .

Configuration

Z-Temper needs a Gemini API key. Either export it in your shell:

export GEMINI_API_KEY="your-api-key-here"

or drop it in a .env file in your working directory (loaded automatically):

GEMINI_API_KEY=your-api-key-here

You can also pass api_key= explicitly when constructing TemperEngine, which takes priority over both.

---

Quick Start

Python API

from z_temper import TemperEngine

engine = TemperEngine(model="gemini-2.5-flash")

engine.synthesize_dataset(
    blueprint_path="path/to/orchid_dataset_blueprint.json",
    output_file="output/ready_to_train.jsonl",
    target_count=100,
    batch_size=5,
    seed_path="path/to/seed_data.jsonl",  # optional
    format_type="sharegpt",               # or "alpaca"
)

CLI

ztemper \
  --blueprint path/to/orchid_dataset_blueprint.json \
  --output output/tempered_dataset.jsonl \
  --count 500 \
  --batch-size 10 \
  --format sharegpt \
  --model gemini-2.5-flash

Flag	Short	Default	Description
--blueprint	-b	required	Path to a Peony orchid_dataset_blueprint.json.
--output	-o	tempered_dataset.jsonl	Destination path for the generated dataset.
--count	-c	100	Target number of valid samples to produce.
--batch-size		5	Samples requested per generation call.
--seed	-s	None	Optional seed dataset path or identifier.
--format	-f	sharegpt	Output schema: sharegpt or alpaca.
--model	-m	gemini-2.5-flash	Gemini model used for generation.

---

Output formats

ShareGPT

{"conversations": [{"from": "human", "value": "..."}, {"from": "gpt", "value": "..."}]}

Alpaca

{"instruction": "...", "input": "", "output": "..."}

Each line is validated by QualityCritic (structure, minimum length, no duplicates) before being written, and results are streamed to disk as they're generated — no data is lost if the run is interrupted partway through.

---

Testing

uv run pytest tests/ -v

---

Related projects

• Project Peony (pip install peony-core) — mechanistic interpretability and diagnostics framework that produces the blueprints Z-Temper consumes.
• DataFlux (pip install dataflux-core) — unified dataset loading library spanning multiple providers, used here for seed exemplars.

---

License

MIT License. Built for the Project Peony / Z-Temper research pipeline.