datadog slo overrides CLI

CLI to set overrides idempotently for multiple SLO's

Usage

Set Datadog SLO corrections ("SLO overrides") on many SLOs at once, selected by tag. A correction excludes a time window from an SLO's error budget (e.g. for planned downtime); this tool is the bulk, idempotent, scriptable way to apply them.

Install

uv tool install datadog_slo_overrides_cli   # global `datadog-slo-overrides` command
# or run without installing:
uvx --from datadog_slo_overrides_cli datadog-slo-overrides --help
# or, from a checkout:
uv run datadog-slo-overrides --help

Check the installed version with datadog-slo-overrides --version.

Commands

Credentials

The tool never stores credentials. They are resolved in this order (first wins):

1. --api-key / --app-key flags
2. DD_API_KEY / DD_APP_KEY environment variables
3. an optional .envrc in the config dir, loaded via direnv

direnv-loaded values can never override a flag or a real environment variable.

Optional direnv setup (keep secret-fetching logic in a file that direnv's approval model governs, rather than the tool executing shell itself):

datadog-slo-overrides init-envrc                 # writes ~/.config/datadog-slo-overrides/.envrc
# it pre-fills DD_API_KEY / DD_APP_KEY exports; review and adjust the source if needed, then:
direnv allow ~/.config/datadog-slo-overrides

If the .envrc is present but unapproved, or doesn't export the keys, the tool prints an actionable hint instead of failing silently. The config dir honours XDG_CONFIG_HOME.

Selecting SLOs

• --tag key:value — repeat to require several tags. One tag is sent to Datadog's (single-tag) server query; the rest are ANDed client-side.
• --tags-query "<raw>" — a raw single-tag Datadog query, used as-is instead of --tag.

Idempotency strategies

set is idempotent: re-running the same command never creates duplicate corrections. --strategy controls when an existing correction counts as already covering your window:

Examples

Preview which SLOs would be corrected (dry run — nothing is written):

datadog-slo-overrides set --tag app:gitlab --tag customer:sbp \
    --start 2026-06-10T22:00 --end 2026-06-11T00:00

Apply a 2-hour scheduled-maintenance correction:

datadog-slo-overrides set --tag app:gitlab --tag customer:sbp \
    --start 2026-06-10T22:00 --end 2026-06-11T00:00 \
    --category "Scheduled Maintenance" --description "DB maintenance" \
    --apply

When a matched SLO is already covered, it is skipped rather than duplicated:

Already satisfied under --strategy skip-if-covered (will skip): 1
  SBP - SLO monitor for the sbp gitlab Website  (fbb8a2c3…)  -> correction d9e08dd2-…

DRY RUN — would create 0, skip 1 already present. Re-run with --apply to write.

Configuration file

init-config writes non-secret defaults (site, timezone, category, strategy) to ~/.config/datadog-slo-overrides/config.toml. CLI flags override the config, which overrides the built-in defaults. Credentials are never read from this file.

datadog-slo-overrides init-config

Run datadog-slo-overrides set --help for the full list of options.

Reporting downtime

list reports, for each matching SLO, the downtime that still counts against it. Each backing monitor's alerting periods are reconstructed from its Datadog alert/recovery events, the SLO is treated as impaired whenever any of its monitors is alerting, and the windows of every correction (override) on that SLO — including the occurrences of a recurring one — are subtracted. What remains is reported with its uptime percentage for the window. The window defaults to the start of the current month through now; set --start/--end to change it. Tag selection is optional (omit both --tag and --tags-query to report every SLO).

# Net downtime this month, for SLOs tagged app:gitlab:
datadog-slo-overrides list --tag app:gitlab

# A specific window, showing only the SLOs that still have downtime left:
datadog-slo-overrides list --start 2026-08-01T00:00 --end 2026-09-01T00:00 --only-downtime

Tags query : app:gitlab
Window     : 2026-08-01 00:00:00 → 2026-08-31 00:00:00 UTC
Rule       : Datadog monitor alert periods (union: any backing monitor alerting), minus SLO corrections — for a multi-monitor SLO this need not equal Datadog's own SLI
Matched    : 2 SLO(s)
Events     : 4 fetched, 4 usable transition(s)

SBP - SLO monitor for the sbp gitlab Website  (fbb8a2c3…)
  monitors: 12345
  raw 2h 4m 58s - excluded 1h = net 1h 4m 58s
  uptime 99.850% (99.711% before corrections)
  • 2026-08-10 03:00:00 → 2026-08-10 04:00:00  (1h)
  • 2026-08-24 17:47:00 → 2026-08-24 17:51:58  (4m 58s)

Instants are rendered to the second, because monitor transitions land on arbitrary seconds. A correction that only partly overlaps an outage trims it rather than excusing it whole, and an SLO whose downtime was entirely excused says so explicitly — it is not silently identical to one that never failed. --only-downtime hides the clean SLOs and reports how many were hidden.

How to read it, and what it cannot tell you

The percentage answers "was the service impaired". It is not a reproduction of the figure on the SLO's own Datadog page: for an SLO backed by several monitors this unions their bad time, where Datadog aggregates its SLI differently. Treat a disagreement there as expected, not as a bug.

Because a silently empty result is indistinguishable from a flawless month, the report states its own evidence and refuses to certify uptime it cannot back:

• The Events line reports how many events were fetched and how many became usable transitions. If events came back but none were readable as state changes, the command says so on stderr and exits 1 rather than printing a page of 100%s.
• A Monitors line appears when some backing monitors produced no transitions at all in the window. Those read as fully up by absence of evidence — worth knowing if a monitor has been red longer than the week of history the report looks back over.
• Events are fetched in day-sized slices, and a slice that comes back at the API's page cap is halved and retried; if one still saturates, a warning says the report understates downtime.
• Ignored events are counted by reason (no monitor, no timestamp, unrecognised transition, out-of-range timestamp) and reported on stderr.

Other limits worth knowing: only type: monitor SLOs link to monitors (via monitor_ids), so metric and time-slice SLOs show no monitor downtime; a monitor in WARN is not counted as down; a future --end is pulled back to now so unelapsed time cannot pad the percentage; and a recurring correction with no duration is skipped with a warning rather than having one inferred.

Command tree

datadog-slo-overrides commands list

Prints a tree of every command the CLI provides.

Developing further

Development flow as Paleofuturistic Python

Prerequisite: uv

Setup

• Fork and clone this repository.
• On first run of any workflow command, the bootstrap step will prompt to install pre-commit hooks.

Workflow

All commands are invoked via ./workflow.cmd <namespace>.<task>:

Development cycle

• Add dependencies: uv add some_lib_you_need
• Develop (optional, tinker: uvx --with-editable . ptpython)
• Format: ./workflow.cmd format
• Lint: ./workflow.cmd lint
• Test: ./workflow.cmd test
• Build: ./workflow.cmd build
• Review docs: ./workflow.cmd document
• Make a pull request.