oddish 0.1.0

Postgres-backed eval scheduler for Harbor agent tasks — queuing, retries, and monitoring

Oddish

Run evals on Harbor tasks at scale with queuing, retries, and monitoring.

Overview

Oddish extends Harbor with:

• Provider-aware queuing and automatic retries for LLM providers
• Real-time monitoring via dashboard or CLI
• Postgres-backed state plus S3 for artifacts

Harbor compatibility: replace harbor run with oddish run.

Quick Start

1. Install

uv pip install -e .

2. Generate an Oddish API key here

export ODDISH_API_KEY="ok_..."

3. Submit a job

# Run a single agent
oddish run -d terminal-bench@2.0 -a codex -m gpt-5.2-codex --n-trials 3

# Or sweep multiple agents
oddish run -d terminal-bench@2.0 -c sweep.yaml

Example sweep.yaml

agents:
  - name: claude-code
    model_name: anthropic/claude-sonnet-4-5
    n_trials: 3
  - name: codex
    model_name: openai/gpt-5.2-codex
    n_trials: 3
  - name: gemini-cli
    model_name: google/gemini-3-flash-preview
    n_trials: 3

4. Monitor Progress

oddish status

Commands

• oddish run — submit a job
• oddish status — monitor progress
• oddish clean — cleanup jobs

Documentation

Technical documentation lives in AGENTS.md.

Instructions for self-hosting are in SELF_HOSTING.md.

License

Apache License 2.0