Onyx Developer Script

ods is onyx.app's devtools utility script. It is packaged as a python wheel and available from PyPI.

Installation

A stable version of ods is provided in the default python venv which is synced automatically if you have pre-commit hooks installed.

While inside the Onyx repository, activate the root project's venv,

source .venv/bin/activate

Prerequisites

Some commands require external tools to be installed and configured:

• Docker - Required for compose, logs, and pull commands

  • Install from docker.com

• uv - Required for backend commands

  • Install from docs.astral.sh/uv

• GitHub CLI (gh) - Required for run-ci, cherry-pick, and trace commands

  • Install from cli.github.com
  • Authenticate with gh auth login

• AWS CLI - Required for screenshot-diff commands (S3 baseline sync)

  • Install from aws.amazon.com/cli
  • Authenticate with aws sso login or aws configure

Autocomplete

ods provides autocomplete for bash, fish, powershell and zsh shells.

For more information, see ods completion <shell> --help for your respective <shell>.

zsh

Linux

ods completion zsh | sudo tee "${fpath[1]}/_ods" > /dev/null

macOS

ods completion zsh > $(brew --prefix)/share/zsh/site-functions/_ods

bash

ods completion bash | sudo tee /etc/bash_completion.d/ods > /dev/null

Note: bash completion requires the bash-completion package be installed.

Commands

compose - Launch Docker Containers

Launch Onyx docker containers using docker compose.

ods compose [profile]

Profiles:

• dev - Use dev configuration (exposes service ports for development)
• multitenant - Dev configuration plus the docker-compose.multitenant.yml overlay (multi-tenant mode)

Flags:

Examples:

# Start containers with default configuration
ods compose

# Start containers with dev configuration
ods compose dev

# Start containers in multi-tenant mode
ods compose multitenant

# Stop running containers
ods compose --down
ods compose dev --down

# Start without waiting for services to be healthy
ods compose --wait=false

# Force recreate containers
ods compose --force-recreate

# Use a specific image tag
ods compose --tag edge

logs - View Docker Container Logs

View logs from running Onyx docker containers. Service names are available as arguments to filter output, with tab-completion support.

ods logs [service...]

Flags:

Examples:

# View logs from all services (follow mode)
ods logs

# View logs for a specific service
ods logs api_server

# View logs for multiple services
ods logs api_server background

# View last 100 lines and follow
ods logs --tail 100 api_server

# View logs without following
ods logs --follow=false

pull - Pull Docker Images

Pull the latest images for Onyx docker containers.

ods pull

Flags:

Examples:

# Pull images
ods pull

# Pull images with a specific tag
ods pull --tag edge

backend - Run Backend Services

Run backend services (API server, model server) with environment loaded from .vscode/.env. On first run, copies .vscode/env_template.txt to .vscode/.env if the .env file does not already exist.

Enterprise Edition features are enabled by default with license enforcement disabled, matching the compose command behavior.

ods backend <subcommand>

Subcommands:

• api - Start the FastAPI backend server (uvicorn onyx.main:app --reload)
• model_server - Start the model server (uvicorn model_server.main:app --reload)

Flags:

Shell environment takes precedence over .env file values, so inline overrides work as expected (e.g. S3_ENDPOINT_URL=foo ods backend api).

Examples:

# Start the API server
ods backend api

# Start the API server on a custom port
ods backend api --port 9090

# Start without Enterprise Edition
ods backend api --no-ee

# Start the model server
ods backend model_server

# Start the model server on a custom port
ods backend model_server --port 9001

web - Run Frontend Scripts

Run bun scripts from web/package.json without manually changing directories.

ods web <script> [args...]

Script names are available via shell completion (for supported shells via ods completion), and are read from web/package.json.

Examples:

# Start the Next.js dev server
ods web dev

# Run web lint task
ods web lint

# Forward extra args to the script
ods web test --watch

test - Run Tests

Run the repo's test suites without changing directories or remembering which suite owns a file.

ods test <suite|path> [args...]

The first argument is a suite name or a path inside a suite. A path selects the suite that covers it, so you can pass a file straight from your editor. All later arguments go to the suite's test runner.

The Go suites run with -race, the same as pr-golang-tests.yml. A runner that takes packages rather than files, such as go test, runs the package that holds a file argument. <file>::<TestName> runs one test.

Examples:

# Run a whole module
ods test ods

# Run one package, one file's package, or one test
ods test tools/ods/internal/testsuite
ods test tools/ods/internal/testsuite/testsuite_test.go
ods test tools/ods/internal/testsuite/testsuite_test.go::TestResolveGoTargets

# Forward arguments to go test
ods test cli -run TestChat -v

coverage - Measure Go Coverage Against a Baseline

Measure Go statement coverage per package and hold it against a committed baseline, so coverage can go up but not down.

ods coverage <suite|module-dir> [flags]

The baseline is a .coverage-baseline.yaml at the module root recording each package's floor. --check fails when a package drops below its floor, which is what pr-golang-tests.yml runs on every PR. After adding tests, --update raises the floors.

The floors are the gate, but they are not always a good comparison: they go stale, so the report shows gains the current change never made. Give --base a commit and the report compares against the coverage snapshot of that commit instead. A PR then sees only what it changed.

Coverage is measured per package with go test -coverprofile, so a package's number counts only its own tests. That is a number the package's owner can act on; a cross-package -coverpkg total would credit a package for statements its own tests never assert on.

The suites are the same Go modules ods test knows: ods, cli, and terraform. A module directory such as tools/ods is accepted in place of a suite name, which is what CI passes.

Flags:

Examples:

# Report where each package stands
ods coverage ods

# Fail on a regression (what CI runs)
ods coverage ods --check

# Record today's numbers as the new floors
ods coverage ods --update

# Compare your branch against main (fetch first, so origin/main is current)
git fetch origin
ods coverage ods --base origin/main

# Keep the profile and browse the uncovered lines
ods coverage ods --profile /tmp/cover.out
go tool cover -html=/tmp/cover.out

Comparing against the base

A snapshot records the exact per-package statement counts of one commit. Each snapshot is a YAML object in S3 at s3://<bucket>/coverage/<module>/<commit sha>.yaml, where the module directory keeps its shape with / replaced by -: tools/ods becomes tools-ods. Runs on main and on release/** write them with --publish, which needs AWS credentials, refuses a dirty tree, and only records a fully successful run.

--base <commit-ish> resolves the base commit first: the merge base of that revision and HEAD when there is one, which is the fork point of your branch, else the revision itself. That is what CI passes, where the given SHA is already the base. The command then walks the first-parent history from the base, at most 25 commits, and reports against the first snapshot it finds. It logs the distance when the snapshot is not on the base commit itself.

With a snapshot, the report column reads Base instead of Floor. Without one, or when the revision cannot be fetched, the command prints a warning and reports against the floors, which is the behavior without --base.

--base never changes what the check does. The committed floors stay the only gate.

Raising the baseline

The gate never fails on an improvement, so a baseline goes stale as tests are added. ods coverage ods reports how many packages have risen; commit the gain with --update so the new level becomes the floor.

pr-golang-tests.yml runs ods coverage <module> --check for every Go module. Without a baseline the tests still run and the report prints, but nothing is gated. A module opts into the gate by committing a baseline, so cli and terraform-provider-onyx join by running ods coverage <suite> --update once.

CI measures and reports in two steps. The first runs the check with --profile and no AWS credentials: the test code comes from the PR, and a process can read the environment it starts with. The second step gets credentials and reports from that profile with --from-profile. A PR run adds --base <base sha>. A merge queue run on main, and a push to a release/** branch, add --publish instead, so the commit that lands gets its snapshot. A fork PR cannot assume the AWS role, so it reads no snapshot and falls back to the floors. After this change lands, run the workflow once by hand (workflow_dispatch on main) to publish the first snapshots.

Against a base, a module without a baseline still reports what moved, so cli and terraform-provider-onyx can now appear in the PR comment. The check still gates nothing for them.

In CI, each module's --markdown report goes to the job summary, and its --html page is uploaded as an artifact and published to the reports bucket. One PR comment, updated in place, lists the modules where a package moved, each with a link to its page.

Floors are rounded down to one decimal, and a package may sit --tolerance below its floor without failing. That absorbs the jitter from suites that depend on ports or timing; a real regression is far larger.

The package floors are the gate. The module total is reported with its delta but never fails the check: a package added without tests, or a well-covered package deleted, moves the total without any package regressing.

type-coverage - Measure Type Coverage Against a Baseline

Measure type coverage per directory and hold it against a committed baseline. Type coverage is the share of identifiers whose type is not any. Each type cast (x as T or <T>x) and each non-null assertion (x!) also counts as one uncovered item. as const and as unknown do not count, because they do not override the checker.

ods type-coverage <checker> [flags]

The only checker is typescript (alias ts), which measures web/. Python is not supported yet, because ty does not report types.

ods web types:check type-checks web/ with the TypeScript 7 API. From the same program, it counts the identifiers in each file. A type error fails the command before the coverage is compared. The count does not include tests: tests/ and __tests__/ directories, and *.test.* and *.spec.* files. Tests are still type-checked. This command groups the files into directories three levels deep, such as src/app/admin, and compares each directory with its floor in web/.type-coverage-baseline.yaml. The flags and the baseline format are the same as for ods coverage.

The default tolerance is 0, because the measurement does not change between runs. The TypeScript 7 API is marked unstable, so a TypeScript upgrade can move the floors. After an upgrade, run --update.

Flags:

Examples:

# Report where each directory stands
ods type-coverage ts

# Fail on a regression (what CI runs)
ods type-coverage ts --check

# Record the new floors after you remove `any` types
ods type-coverage ts --update

# Type-check and print the total only
ods web types:check

The typescript-check pre-commit hook runs --check when a .ts or .tsx file in web/ changes. pr-quality-checks.yml runs the same hook on every PR.

dev - Devcontainer Management

Manage the Onyx devcontainer. Also available as ods dc.

Requires the devcontainer CLI (bun install -g @devcontainers/cli).

ods dev <subcommand>

Subcommands:

• up - Start the devcontainer (pulls the image if needed)
• into - Open a zsh shell inside the running devcontainer
• exec - Run an arbitrary command inside the devcontainer
• restart - Remove and recreate the devcontainer
• rebuild - Pull the latest published image and recreate
• stop - Stop the running devcontainer

The devcontainer image is published to onyxdotapp/onyx-devcontainer and referenced by tag in .devcontainer/devcontainer.json — no local build needed.

Examples:

# Start the devcontainer
ods dev up

# Open a shell
ods dev into

# Run a command
ods dev exec -- bun test

# Restart the container
ods dev restart

# Pull latest image and recreate
ods dev rebuild

# Stop the container
ods dev stop

# Same commands work with the dc alias
ods dc up
ods dc into

db - Database Administration

Manage PostgreSQL database dumps, restores, and migrations.

ods db <subcommand>

Subcommands:

• dump - Create a database dump
• restore - Restore from a dump
• upgrade/downgrade - Run database migrations
• drop - Drop a database

Run ods db --help for detailed usage.

openapi - OpenAPI Schema Generation

Generate OpenAPI schemas and client code.

ods openapi all

check-lazy-imports - Verify Lazy Import Compliance

Check that specified modules are only lazily imported (used for keeping backend startup fast).

ods check-lazy-imports

check-getattr - Forbids the getattr Builtin

Checks that backend Python code does not reference the getattr builtin, which hides attribute access from the type checker. Genuinely dynamic lookups are suppressed inline with # ods: ignore[getattr] plus a brief justification. String literal contents and comments never match; replacement fields inside f-strings are scanned as code.

ods check-getattr [paths...]

--annotate appends the ignore marker to every violating line (baseline maintenance).

fmt - Format Sources

Also available as ods format.

fmt tf - Format Terraform

Rewrite Terraform files into the canonical HCL style. Also available as ods fmt terraform.

ods fmt tf [paths...]

This applies the same formatter as terraform fmt, so the output matches terraform byte for byte, but no terraform binary is needed. A file that does not parse is reported and left alone.

Files and directories may be given to limit the run. With no arguments, the whole repository is scanned. Vendored trees (.terraform, node_modules, .venv) are always skipped.

Flags:

Examples:

# Format every .tf file in the repository
ods fmt tf

# Format one subtree
ods fmt tf deployment/terraform

# Report unformatted files, change nothing
ods fmt tf --check

The command exits non-zero when a file was rewritten or failed to parse, which is how the terraform-fmt pre-commit hook gates a commit.

lint - Run Linters

lint tf - Check Published Terraform

Check published Terraform modules for values that must stay internal. Also available as ods lint terraform.

ods lint tf [paths...]

The modules under deployment/terraform are published, but they stay in sync with the infrastructure Onyx runs. That makes it easy to carry an internal value across by accident -- an office IP in a variable default is the case this check was written for.

The check looks for objective patterns only:

It cannot screen for customer names, because listing them here would leak them; that stays a review step.

Add a trailing # public-safe: ok comment to accept a specific line.

Files and directories may be given to limit the check. With no arguments, deployment/terraform is scanned.

Examples:

# Check all published modules
ods lint tf

# Check one subtree
ods lint tf deployment/terraform/modules/aws

# Check a single file
ods lint tf deployment/terraform/modules/aws/vpc/main.tf

audit - Audit Dependencies for Vulnerabilities

Install the audit extra first. The scanner is about 50 MB, most of the download, so it ships as a separate onyx-devtools-audit wheel that provides the ods-audit binary. ods audit forwards to it and prints an install hint when it is missing.

# Install alongside ods
uv tool install 'onyx-devtools[audit]'

# Or run it without installing (how CI runs the gate)
uv run --with 'onyx-devtools[audit]' ods audit

ods-audit takes the same arguments, so ods audit --python and ods-audit --python are the same command.

Scan the JavaScript (bun.lock) and Python (uv.lock) lockfiles via osv-scanner (vendored as a library, no external binary required) and open GitHub Dependabot security alerts for known vulnerabilities. With no selector flags, all sources are audited.

Accepted advisories are suppressed via an allowlist fetched from S3 at runtime (s3://onyx-internal-tools/audit/ignores.json by default), so a release can be unblocked without a code change. The command exits non-zero when an unignored finding at or above --fail-on (default critical) remains, which is how it gates deploys.

ods audit [--web] [--python] [--dependabot] [--format text[,json][,sarif]] [--fail-on critical|high|moderate|low] [--ignore-url s3://...]

--format takes a comma-separated list. The machine-readable formats (json, sarif) are written to stdout, while the human-readable text report is written to stderr when combined with one of them. This lets a single run produce a SARIF file for upload and a readable report in the log: ods audit --format=sarif,text > audit.sarif sends SARIF to the file and the report (plus, when the gate fails, a runbook explaining how to resolve or suppress each finding) to the terminal. A lone format always goes to stdout, so --format=sarif > audit.sarif is unchanged. At most one machine-readable format may be requested.

Examples:

# Audit everything; fail on unignored criticals
ods audit

# Only the Python lockfile
ods audit --python

# Emit a SARIF report (used by the nightly GitHub code-scanning job)
ods audit --python --format=sarif > audit.sarif

# SARIF to a file for upload, readable report to the log (used by CI gates)
ods audit --format=sarif,text > audit.sarif

Managing the allowlist

Suppress a reviewed-and-accepted advisory so it stops blocking the gate:

# Interactive editor (add/edit/delete rows, then upload after a confirmation)
ods audit ignore

# Non-interactive add of a single suppression (a --reason is required)
ods audit ignore add GHSA-xxxx-xxxx-xxxx --ecosystem npm \
  --reason "not reachable in our usage" --expires 2026-09-01

ods audit ignore add stamps added_by from your git email, shows a diff, and uploads the updated allowlist to S3 after a confirmation prompt (--yes skips it). Suppress only advisories you've assessed — the allowlist gates every deploy.

The allowlist is a JSON document of the form:

{
  "ignores": [
    {
      "id": "GHSA-xxxx-xxxx-xxxx",
      "ecosystem": "npm",
      "reason": "not reachable in our usage",
      "added_by": "you@onyx.app",
      "expires": "2026-09-01"
    }
  ]
}

id matches a finding's id or any of its aliases (case-insensitive); ecosystem and expires are optional (an expired entry stops suppressing).

run-ci - Run CI on Fork PRs

Pull requests from forks don't automatically trigger GitHub Actions for security reasons. This command creates a branch and PR in the main repository to run CI on a fork's code.

ods run-ci <pr-number>

Example:

# Run CI for PR #7353 from a fork
ods run-ci 7353

cherry-pick - Backport Commits to Release Branches

Cherry-pick one or more commits to release branches and automatically create PRs. Cherry-pick PRs created by this command are labeled cherry-pick 🍒.

ods cherry-pick <commit-sha> [<commit-sha>...] [--release <version>]

Examples:

# Cherry-pick a single commit (auto-detects release version)
ods cherry-pick abc123

# Cherry-pick to a specific release
ods cherry-pick abc123 --release 2.5

# Cherry-pick to multiple releases
ods cherry-pick abc123 --release 2.5 --release 2.6

# Cherry-pick multiple commits
ods cherry-pick abc123 def456 ghi789 --release 2.5

screenshot-diff - Visual Regression Testing

Compare Playwright screenshots against baselines and generate visual diff reports. Baselines are stored per-project and per-revision in S3:

s3://<bucket>/baselines/<project>/<rev>/

This allows storing baselines for main, release branches (release/2.5), and version tags (v2.0.0) side-by-side. Revisions containing / are sanitised to - in the S3 path (e.g. release/2.5 → release-2.5).

ods screenshot-diff <subcommand>

Subcommands:

• compare - Compare screenshots against baselines and generate a diff report
• upload-baselines - Upload screenshots to S3 as new baselines

The --project flag provides sensible defaults so you don't need to specify every path. When set, the following defaults are applied:

The S3 bucket defaults to onyx-playwright-artifacts and can be overridden with the PLAYWRIGHT_S3_BUCKET environment variable.

compare Flags:

upload-baselines Flags:

Examples:

# Compare local screenshots against the main baseline (default)
ods screenshot-diff compare --project admin

# Compare against a release branch baseline
ods screenshot-diff compare --project admin --rev release/2.5

# Compare two revisions directly (both sides fetched from S3)
ods screenshot-diff compare --project admin --from-rev v1.0.0 --to-rev v2.0.0

# Compare with explicit paths
ods screenshot-diff compare \
  --baseline ./baselines \
  --current ./web/output/screenshots/ \
  --output ./report/index.html

# Upload baselines for main (default)
ods screenshot-diff upload-baselines --project admin

# Upload baselines for a release branch
ods screenshot-diff upload-baselines --project admin --rev release/2.5

# Upload baselines for a version tag
ods screenshot-diff upload-baselines --project admin --rev v2.0.0

# Upload with delete (remove old baselines not in current set)
ods screenshot-diff upload-baselines --project admin --delete

The compare subcommand writes a summary.json alongside the report with aggregate counts (changed, added, removed, unchanged). The HTML report is only generated when visual differences are detected.

trace - View Playwright Traces from CI

Download Playwright trace artifacts from a GitHub Actions run and open them with playwright show-trace. Traces are only generated for failing tests (retain-on-failure).

ods trace [run-id-or-url]

The run can be specified as a numeric run ID, a full GitHub Actions URL, or omitted to find the latest Playwright run for the current branch.

Flags:

When multiple traces are found, an interactive picker lets you select which traces to open. Use arrow keys or j/k to navigate, space to toggle, a to select all, n to deselect all, and enter to open. Falls back to a plain-text prompt when no TTY is available.

Downloaded artifacts are cached in /tmp/ods-traces/<run-id>/ so repeated invocations for the same run are instant.

Examples:

# Latest run for the current branch
ods trace

# Specific run ID
ods trace 12345678

# Full GitHub Actions URL
ods trace https://github.com/onyx-dot-app/onyx/actions/runs/12345678

# Latest run for a PR
ods trace --pr 9500

# Latest run for a specific branch
ods trace --branch main

# Only download admin project traces
ods trace --project admin

# List traces without opening
ods trace --list

Testing Changes Locally (Dry Run)

Both run-ci and cherry-pick support --dry-run to test without making remote changes:

# See what would happen without pushing
ods run-ci 7353 --dry-run
ods cherry-pick abc123 --release 2.5 --dry-run

Upgrading

To upgrade the stable version, upgrade it as you would any other requirement.

Building from source

Generally, go build . or go install . are sufficient.

go build . will output a tools/ods/ods binary which you can call normally,

./ods --version

while go install . will output to your GOPATH (defaults ~/go/bin/ods),

~/go/bin/ods --version

Typically, GOPATH is added to your shell's PATH, but this may be confused easily during development with the pip version of ods installed in the Onyx venv.

To build the wheel,

uv build --wheel

To build and install the wheel,

uv pip install .

Build constraints

The release workflows build every wheel against tools/requirements/build-constraints.txt, shared by cli, tools/ods and tools/ods-audit. They pass it as UV_BUILD_CONSTRAINT with UV_REQUIRE_HASHES, so the build environment (hatchling, go-bin, manygo and their dependencies) is pinned and verified.

A plain local uv build resolves those dependencies freely. Set the same variables to build against the pinned closure:

UV_BUILD_CONSTRAINT="$(git rev-parse --show-toplevel)/tools/requirements/build-constraints.txt" \
  UV_REQUIRE_HASHES=true GOTOOLCHAIN=local \
  uv build --wheel

tools/requirements/build-constraints.txt is compiled from its .in by the pip-compile pre-commit hook, so a plain pre-commit run pip-compile refreshes it. To upgrade a pinned build dependency, edit build-constraints.in and the matching [build-system] requires in all three pyproject.toml files; the build-constraints-drift hook fails if they disagree.

go-bin ships the Go toolchain used to compile the binary. Bumping Go means moving it everywhere at once: the go directive in go.mod, cli/Dockerfile, the setup-go and GO_VERSION pins across .github/workflows/, .devcontainer/Dockerfile, the go-bin pin in all three pyproject.toml files and in tools/requirements/build-constraints.in, and the compiled build-constraints.txt.

Deploy

Releases are deployed automatically when git tags prefaced with ods/ are pushed to GitHub.

ods release ods calculates and pushes the next tag for you,

ods release ods              # bumps the patch version
ods release ods --bump minor
ods release ods --version 0.14.0
ods release ods --dry-run    # computes the version, tags nothing

The workflow also opens a PR moving every onyx-devtools== pin in the repo to the new version, so no follow-up commit is needed.

See also, .github/workflows/release-devtools.yml.