ai-almanac

Local-first benchmarking platform for AI weather and climate models.

Pick a region and an event type (e.g. monsoon onset), select one or more models, submit a benchmark, and see per-grid-point skill maps (MAE, FAR, miss rate, RMSE, ACC, bias) rendered in your browser. The whole thing — web UI, API, benchmark runner, and database — is a single Python process you launch on your own GPU workstation (NVIDIA DGX Spark, a lab box, your laptop for the LLM-driven UI).

pipx install ai-almanac      # or `uv tool install ai-almanac`
ai-almanac serve             # opens http://localhost:8765 in your browser

That's it.

What you get

One Python package. One process. One port. One data directory.

Benchmark jobs run in detached local supervisor processes. Closing or restarting the web server does not stop active work. Restarting ai-almanac serve reconciles queued and running jobs from SQLite, and active jobs can be canceled from the benchmark UI.

Install

# Recommended — uv tool install gives you a managed venv per CLI
uv tool install ai-almanac

# Or pipx
pipx install ai-almanac

# Or pip (less isolated)
pip install ai-almanac

Once a homebrew tap / .deb is published, brew install ai-almanac and apt install ai-almanac will be available too. The PyPI package is the source of truth — those channels are thin wrappers.

Prerequisites for running real benchmarks and blends

Actual benchmark and model-blending execution need separate scientific Python stacks. Those dependencies live in isolated Pixi-managed environments to keep the core install small:

# Install pixi (one-time): https://pixi.sh
curl -fsSL https://pixi.sh/install.sh | bash

# Materialize both workload environments (takes a few minutes the first time)
ai-almanac env prepare

This also checks out the blending workflow at the version pinned by AI Almanac. Subsequent ai-almanac serve runs reuse both environments. ai-almanac env info prints the installed benchmark package versions.

Usage

ai-almanac serve                       # default: 127.0.0.1:8765, opens browser
ai-almanac serve --port 9000           # alternate port
ai-almanac serve --no-open             # don't auto-launch a browser tab
ai-almanac serve --reload              # dev mode (uvicorn auto-reload)
ai-almanac env prepare                 # install / update workload environments
ai-almanac env info                    # show installed package versions

ai-almanac reset --confirm             # wipe ~/.local/share/ai-almanac/
ai-almanac version

Test shared mode locally

Run the complete multi-user architecture locally without DNS, TLS, an OIDC provider, or a GPU:

pixi run self-host-local

Open http://localhost:18080. The stack uses PostgreSQL, shared ownership and authorization, persistent Docker volumes, and synthetic benchmark outputs. Switch between the built-in administrator and regular user at http://localhost:18080/__dev.

On a host with NVIDIA Container Toolkit, run real benchmarks instead:

pixi run self-host-local-gpu

The same shared stack can exercise GCS artifact storage or the deployed Modal runner. GCS requires three bucket names and a service-account or Application Default Credentials JSON file:

export GCS_DATA_BUCKET=my-data-bucket
export GCS_UPLOADS_BUCKET=my-uploads-bucket
export GCS_OUTPUTS_BUCKET=my-outputs-bucket
export GOOGLE_APPLICATION_CREDENTIALS="$HOME/.config/gcloud/application_default_credentials.json"

pixi run self-host-local-gcs

To submit jobs to Modal, also export MODAL_TOKEN_ID and MODAL_TOKEN_SECRET, deploy the Modal app, then run:

pixi run self-host-local-modal

Modal mode uses GCS because remote workers cannot access the local Compose volume. The local runner supports either local storage or GCS.

Stop the stack with pixi run self-host-local-down. Use pixi run self-host-local-reset to also delete its PostgreSQL and application data volumes.

For an internet-accessible or multi-user installation, follow the deployment guide. Do not expose personal mode to untrusted users.

Where data lives

Everything goes under $AI_ALMANAC_DATA_DIR, defaulting to:

• Linux: ~/.local/share/ai-almanac/
• macOS: ~/Library/Application Support/ai-almanac/
• Windows: %LOCALAPPDATA%\ai-almanac\

$AI_ALMANAC_DATA_DIR/
├── almanac.db          ← SQLite (WAL mode)
├── uploads/            ← user-uploaded obs datasets
├── jobs/<job_id>/      ← run logs, NetCDF outputs, figures
├── benchmark-env/      ← Pixi environment (ROMP + scientific/geo dependencies)
├── blending-env/       ← Pixi environment and pinned blending workflow
└── cache/              ← weight cache (HuggingFace), ARCO chunks

Override with AI_ALMANAC_DATA_DIR=/some/path ai-almanac serve to move one instance's private state. Do not point multiple running instances at the same application data directory. Researchers may register the same read-only input dataset directories from separate AI Almanac instances.

Adding data

Open Data in the web UI and register local observation or model-output directories. AI Almanac checks the file pattern and configured NetCDF variable before making a source available to benchmark workflows.

Development

Install Pixi, then start the complete development stack:

pixi run dev

This runs FastAPI with Python auto-reload on http://localhost:8765 and the SvelteKit Vite server with hot module replacement on http://localhost:5173. See DEVELOPMENT.md for the full task list.

License

MIT.