ProgramSmith

Convert GitHub repos into Harbor tasks automatically.

Overview

Automates task creation from real open-source repos. Input any GitHub repo, and it produces a long-horizon ProgramBench-style reverse engineering task.

Every task is difficulty-calibrated and audited before export. If reward-hacking, task environment problems, or verifier design issues are found, the pipeline continues iterating through a loop until all QA and difficulty calibration is met. Task execution and grading are containerized, with required dependencies installed when each task image is built.

Quick Start

# Install
uv pip install programsmith

# Generate a task from a repo
programsmith create --repo d5/tengo

# Or farm multiple tasks at once
programsmith farm repos.txt

Installation

uv pip install programsmith

Ensure Docker is running and at least one model credential is set:

export CLAUDE_CODE_OAUTH_TOKEN=<token>   # (run `claude setup-token` in your CLI)
# or
export ANTHROPIC_API_KEY=<api-key>

Trials can run on any provider:

export OPENAI_API_KEY=<api-key>          # codex CLI + any openai/* model
export GEMINI_API_KEY=<api-key>          # gemini CLI + any gemini/* model
export ZAI_API_KEY=<api-key>             # GLM models via the mini-SWE-agent harness

Usage

Alias: psmith <cmd> is automatically installed

Commands:

• programsmith create — Generate a task from a repo
• programsmith farm — Farm a list of repos into multiple tasks
• programsmith serve — Start the browser dashboard in the background
• programsmith stop — Stop the background dashboard
• programsmith doctor — Preflight checks (Docker, credentials, disk)
• programsmith status — Pipeline status

Generate a Task

programsmith create --repo <owner/repo>

Runs the full pipeline (see below). Accepted tasks are exported to out/tasks/, tasks that fail calibration but are sound are exported to out/easy/.

Re-running the same command resumes the run from wherever it parked. Ctrl-C stops active model trials, keeps completed trials, and leaves unfinished work ready to resume.

Farm Multiple Repositories

programsmith farm repos.txt

Streams through a repo list (one owner/name[@sha] per line) and drives every run to completion in the foreground. Re-running the command picks up existing runs where they stopped. A curated starter list of known-good repos is included (pb10-repos.txt).

Dashboard

programsmith serve
# The dashboard runs in the background and survives Ctrl-C / closing this shell.
programsmith stop

Serves the local dashboard at http://localhost:8765: live pipeline DAG for each run, agent output, sweep results, file explorer, and optional review gates. Evaluation sweeps remain parked unless serve --spend is used. programsmith serve returns after the dashboard is healthy; programsmith stop is the explicit shutdown command.

Task Requirements

Pipeline

License

Apache License 2.0