Data Navigator

Members-only catalogue of skill files that teach AI agents to query public databases, plus the executor that runs the queries. Three faces — web UI, the navigator CLI, and (Phase 3) MCP — all over one HTTP API. See SPEC.md for the full design; this README covers running and validating the local app.

Data Navigator finds and queries individual structured public-record sources. Workflows Navigator uses complex OSINT workflows to return structured data assembled from several sources.

Layout

The skill tree is the single source of truth:

skills/<jurisdiction>/<source-slug>/
├── SKILL.md     # agent-facing playbook: inputs, output shape, gotchas
├── meta.yaml    # validated catalogue record (app/skills_registry.py)
└── adapter.py   # run(input, ctx) — executes the query

• skills/_template/ — copy this to add a source; every field is documented.
• skills/_meta/navigator-data/ — the meta-skill members drop into ~/.claude/skills/ so their agent knows the find → show → query loop.
• app/skills_registry.py — walks the tree, validates every meta.yaml (pydantic), builds the registry. Invalid skill dirs fail the app at startup.
• app/runner.py — dynamic adapter loading + AdapterContext.get_key() (OS keychain / NAVIGATOR_KEY_<NAME> env; keys never enter agent context).

Source runtimes: local (public or BYO-key, runs on the member's machine, unmetered) and hosted (our key, runs server-side, metered — e.g. OpenSanctions).

Run locally

uv sync
uv run uvicorn app.main:app --reload --port 8000

Open http://localhost:8000/data. Set OPENROUTER_API_KEY to enable the natural-language query bar (/api/query/natural returns 503 without it — there is no fallback planner).

Run with Docker

docker compose up --build

CLI

One unified navigator CLI for the whole product: navigator data … is Data Navigator, navigator tools … (Phase 3) is osint-navigator.

navigator auth login                   # magic link → PAT in the OS keychain
navigator auth status                  # tier + quota state
navigator data find company            # search the catalogue
navigator data show no/brreg/enheter   # print the source's SKILL.md + meta
navigator query no/brreg/enheter --input '{"navn":"Equinor","size":3}'
navigator query no/brreg/enheter --input '{"navn":"Equinor"}' --out companies.csv
navigator query <source-id> --input-file request.json --out response.json
navigator tools find "satellite imagery"   # OSINT tool search (osint-navigator API)
navigator tools show <tool-id>             # full tool record + documentation
navigator keys missing                 # BYO keys you still need to set
navigator keys set companies-house     # store a key in the OS keychain

For a headless, Docker, or CI installation, sign in at https://navigator.indicator.media/data, open Connect an Agent → API Keys, generate a key, and inject it through the runtime's secret manager as NAVIGATOR_PAT. Do not put the key in an agent prompt, CLAUDE.md, shell arguments, or source control. NAVIGATOR_PAT takes precedence over the OS keychain. navigator auth import --stdin is an installer-to-keychain bridge, not a keychainless authentication path; environment credentials must be revoked from the API Keys modal rather than with navigator auth logout.

--input-file is the safe path for arbitrary text or larger request bodies; it avoids interpolating user content into a shell command. --out <file>.json|.csv writes the records to the file and prints a compact summary instead of the full payload. Hosted sources route to the Data Navigator server (DATANAV_BASE_URL) with the stored PAT.

There is no install step: in remote mode the CLI fetches skill bundles from the authed API and caches them under ~/.navigator/cache/ (re-fetched when a source's meta changes). Mode is auto-detected — a packaged install (no local skills/ tree) defaults to remote so navigator query … works out of the box for members; an in-repo checkout defaults to local for dev and smoke tests. Set DATANAV_CATALOGUE=remote|local to override. The only thing a member installs is the meta-skill.

Membership (Phase 2)

Everything is paid-member-gated in production (DATANAV_AUTH=on; default off for local dev). Auth is owned by osint-navigator — this service validates Bearer PATs / the shared osint_session cookie via its internal introspection API and requires the shared data_sources capability for Data Navigator's web, CLI/remote source-skill catalogue, and MCP surfaces. Through 2026-09-05 that capability is promotional for Pro and Lab members; it then returns to Lab-only alongside desktop_download. Metering: UI NL queries + try-it burn data_ui; hosted-source queries burn data_hosted; public + BYO local execution is never metered. Quota storage lives in osint-navigator.

Cross-service proof: uv run python scripts/integration_check_membership.py (boots osint-navigator's internal API from the sibling checkout and verifies 401/403/tier/quota behavior end-to-end).

MCP, meta-skill, rollout

• MCP: data_find_source / data_show_source / data_query live on osint-navigator's remote MCP and proxy here over the service channel. Local sources are unmetered over MCP (the member's agent reasons; upstream is free).
• Meta-skill: members download it from the UI ("Agent setup") or GET /api/meta-skill and drop it into ~/.claude/skills/navigator-data/.
• Rollout: the full Tools/Data/Skills/Workflows toggle is public. Data remains fail-closed behind the member gate; Skills and Workflows are disabled as coming soon. /data is reverse-proxied from osint-navigator when DATA_NAVIGATOR_URL is set.
• Deploy and September rollback: ../docs/deploy-runbook.md.

API

curl -s "http://localhost:8000/api/registry"
curl -s "http://localhost:8000/api/sources/no/brreg/enheter"   # meta + skill_md
curl -s -X POST "http://localhost:8000/api/query/no/brreg/enheter" \
  -H "Content-Type: application/json" -d '{"navn":"Equinor","size":3}'

Validation

First, validate the skill tree (both smoke scripts assume this passes):

uv run python scripts/validate_skills.py

The validation gate is three live public APIs returning records:

• no/brreg/enheter — Norway Brønnøysund companies
• us/usaspending/awards — USAspending federal awards
• global/gleif/lei-records — GLEIF LEI records

uv run pytest
DATANAV_BASE_URL=http://localhost:8000 uv run python scripts/smoke_public_apis.py
NAVIGATOR_CMD="uv run navigator" uv run python scripts/smoke_cli_public_apis.py
bash scripts/smoke_docker_image.sh

Deployment

render.yaml defines a Docker web service (health check /health, port from $PORT). Server secrets: OPENROUTER_API_KEY (NL planner) and NAVIGATOR_KEY_THINKPOL (hosted partner source). Arbiter is a BYO-key source: members register through Indicator's attributed partner link in the Arbiter source metadata, then store their own key with navigator keys set arbiter. ARBITER_API_BASE optionally selects the matching Arbiter deployment and defaults to production.

Phase 2 (membership wiring: auth introspection against osint-navigator, metered quotas, submissions form) and Phase 3 (MCP, catalogue growth) are specified in SPEC.md §9.

The implemented global/arbiter/case-studies partner source runs locally with the member's Arbiter key. Spotlight uses the Navigator CLI and saves its top-level-compatible report response for deterministic Arbiter analytics rendering. Charged or mutating Arbiter operations must use an explicit structured query; the natural-language planner cannot approve them.