skills-telemetry

Telemetry and metadata capture for standards-as-skills — attributes token cost and outcomes to individual skills across Claude Code and GitHub Copilot, joined to policy (OPA/Rego), delivery (Linear/GitHub) and quality data. Design rationale: docs/design-proposal.md.

Metadata only. No prompt, response or file content is ever emitted; the collector drops it again as defence in depth. If it runs on your machine, docs/for-developers.md lists every field that leaves it — and export STDTEL_DISABLED=1 turns it off entirely.

When can I trust these numbers?

Telemetry starts producing plausible-looking ratios on day one. Most of them mean nothing yet. The phases below are set by data volume, not elapsed time — how long each takes depends entirely on team size, and a small team may never leave the first one.

The hard floor: below 30 merged PRs per arm, or fewer than 5 developers, report descriptively and make no comparative claim.

Early on, the most valuable thing this data does is find its own faults. A high share of unversioned skills or unattributed tickets bounds every later conclusion, and both are fixable now — see stdtel-onboard and ticket-prefixed branches.

The failure mode this exists to prevent: reading a scorecard after two weeks, seeing a skill "underperform" across nine PRs, and deprecating it. Nine PRs cannot distinguish a bad skill from a quiet fortnight. Full derivations and the assumptions behind every figure: docs/evaluation-power.md. Worked examples of asking these questions, including the ones the data cannot answer: docs/insight-walkthroughs.md.

Documentation

Layout

skills/<name>/SKILL.md      skill catalogue with validated front-matter (the contract)
stdtel/manifest.py          front-matter parser + `stdtel-validate` CI gate
stdtel/hooks/cli.py         Claude Code hooks: session-start | pre-tool-use | post-tool-use | stop
stdtel/transcript.py        incremental JSONL reader + token attribution (tail rule, first-only sensitivity)
stdtel/exporter.py          std.skill.invocation spans via OTLP/HTTP (content scrubbed)
stdtel/enrich.py            join keys: std.ticket.id from branch, std.repo, std.team, std.harness
stdtel/skillmap.py          generates collector/copilot-skill-map.yaml for Copilot tool-call mapping
collector/otel-collector.yaml  drop content → normalise gen_ai.* → map Copilot skills → pseudonymise → spanmetrics
deploy/docker-compose.yml   collector + Tempo + Prometheus + Grafana + Postgres; `langfuse` profile optional
deploy/smoke.sh             eight-hop verification ladder (`mise run smoke`)
collector/overlay-*.yaml    merged over the base config; `none` is the default no-op, `langfuse` adds an exporter
warehouse/schema.sql        skill_invocation, session_cost, ticket, pull_request, policy_result, defect, skill_eval
warehouse/scorecard.sql     weekly per-skill scorecard → keep / refine / review-merge / deprecate
warehouse/load_traces.py    Tempo → Postgres loader
eval/run_eval.py            offline with/without-skill eval, real OPA grading (`--dry-run` is a smoke test)
eval/power.py               generates every table in docs/evaluation-power.md; CI checks it is current
policies/                   Rego behind the primary metric: logging.*, telemetry.manifest_valid
agents/                     skill-scorecard-analyst: keep / refine / merge / deprecate from the data
warehouse/load_delivery.py  GitHub → ticket / pull_request / defect; policy_result from a CI artefact
examples/settings.*.json    hook + OTel wiring: `global` installs once, `project` overrides per repo
docs/adrs/                  six ADRs: distribution, delivery data, hook constraints, identity, integrity, Langfuse
plugin.json                 Agent Plugins v1 manifest (portable `skills/` is the shared half)
.claude-plugin/             Claude Code plugin + marketplace manifest
hooks/, com.github.copilot/ per-harness hook manifests, generated from stdtel/install.py::EVENTS
stdtel/install.py           `stdtel-install`: absolute-path resolution + settings merge
mise.toml                   toolchain (Python 3.13) + `.venv` + tasks wrapping the Makefile

Quick start

Two separate jobs: install the harness wiring once for your user, then onboard each project you want attributed telemetry from. Skipping the second step still gives you spans — the skills just come through as unversioned, with std.team=unknown.

1. Install once, globally

uv tool install stdtel        # or: pipx install stdtel
stdtel-install settings       # merges hooks into ~/.claude/settings.json

Two halves, and neither updates the other. The plugin ships hook registration, the launcher, the skills and the agent. The package ships everything that runs. A plugin without the package produces no data and no error, because the launcher exits 0 in silence — it says so once at session start, which is the only place it can, since stdtel-doctor is part of the package too. Keep them in step:

uv tool install stdtel --force --refresh          # the package
claude plugin update stdtel@amiable-standards     # the plugin

stdtel-doctor reports drift between them. To run from a checkout instead: uv tool install /path/to/skills-telemetry.

stdtel-install resolves the absolute path of the stdtel-hook it was installed alongside and writes that into the config. This is not cosmetic: hook processes get a non-login sh -c and inherit whatever PATH launched the harness, so a bare stdtel-hook is unresolvable whenever a version manager (mise, asdf, pyenv) or an activated venv is what put it there. The same reason pre-commit bakes sys.executable into the git hook it generates. stdtel-install where prints the path it will use; --dry-run shows the JSON without writing.

Claude Code also strips OTEL_* from every subprocess it spawns, so point the exporter at your collector with the STDTEL_-namespaced variables, which survive:

export STDTEL_OTLP_ENDPOINT=http://collector.internal:4318   # default: http://localhost:4318
export STDTEL_OTLP_TIMEOUT=2                                 # seconds; bounds a dead-collector stall

With no STDTEL_SKILLS_ROOT set, the catalogue is read from ~/.claude/skills. Symlink your standards there and every project gets versioned spans:

ln -s "$PWD/skills/structured-logging" ~/.claude/skills/structured-logging
stdtel-validate ~/.claude/skills      # same contract gate CI runs

As a plugin

The repo is laid out for three loaders at once (ADR-001), so it installs as a plugin without a separate packaging step:

/plugin marketplace add amiable-dev/skills-telemetry
/plugin install stdtel@amiable-standards

The shipped hooks/hooks.json carries the bare command name, because a distributed manifest cannot know your install path — run stdtel-install settings afterwards to bind it to an absolute one.

Copilot

Copilot needs no code from us. It emits first-party OpenTelemetry with per-tool-call spans and token counts; point it at the same collector (user settings, COPILOT_OTEL_* env vars, or the enterprise managed-settings.json telemetry block for a fleet).

If you do run our hooks on Copilot as well, install them from com.github.copilot/hooks/hooks.json, which sets STDTEL_HARNESS per hook. That is load-bearing: VS Code Copilot reads ~/.claude/settings.json, and its snake_case payload dialect is indistinguishable from Claude Code's — without that env block, Copilot activity is recorded as claude-code.

2. Onboard a project

Per-project overrides go in <project>/.claude/settings.json — Claude Code merges them over the user file, so repeat only what differs. Do not repeat the hooks block: it is already registered globally and a second copy fires each hook twice.

cd ~/projects/payments-api
mkdir -p .claude && cp ~/projects/skills-telemetry/examples/settings.project.json .claude/settings.json
$EDITOR .claude/settings.json      # STDTEL_TEAM is the one you must set
git checkout -b feature/PLAT-123-add-audit-log   # ticket prefix -> std.ticket.id join key

Then verify the loop end to end:

claude                                     # invoke a skill in the project
cat ~/.stdtel/sessions/*.json              # a window with skill, version, tool_use_id
curl -s 'http://localhost:3200/api/search?tags=name%3Dstd.skill.invocation' | jq '.traces[0]'

A std.skill.version of unversioned means the name in the transcript matched no SKILL.md in any root — check STDTEL_SKILLS_ROOT and that the skill's front-matter name matches what you invoked.

Copilot: enable managed OTel export (VS Code / CLI) pointing at the same collector with resource attributes std.harness=copilot-vscode, std.team=<team>; collector/otel-collector.yaml maps catalogued skill tool-calls onto std.skill.*.

Span schema

Two span types, emitted at Stop. They overlap by design and must never be summed: std.session.cost is the session's total spend, std.skill.invocation attributes a share of that total to one skill. Use session cost as the denominator for cost-per-PR; use invocation tail tokens to compare skills with each other.

std.session.cost

Emitted once per turn that made any LLM request, whether or not a skill was loaded — a session that never loads a skill is still spend, and excluding it would silently understate cost per PR.

std.skill.invocation

Known limitations

• Tool counts are counts only. PostToolUse fires for every tool, so tool_input and tool_response carry commands, file contents and diffs — scrub() refuses those attribute names outright and a test asserts nothing leaks.
• Skill name is parsed from the Skill tool input in hooks (the field is skill, verified against 120 real invocations). Parsing is isolated in hooks/cli.py::_skill_from_payload.
• std.skill.trigger reports the transcript's caller.type where present, else unknown — it is never guessed.
• Tail attribution splits by load order; when several skills load in one turn compare against tail_tokens_first_only.
• Copilot granularity is per turn; use Claude Code's finer data for within-harness tuning only.
• load_tokens uses a chars/4 heuristic on the Skill tool result.
• gen_ai.* conventions are still Development upstream; extend transform/normalise as names move.

Contributing

Bug reports, questions and PRs are all welcome. Start with CONTRIBUTING.md — it covers the setup (mise run ci is the whole gate) and the conventions that the tests enforce, including the ones that will surprise you.

This tool runs inside your coding sessions, so SECURITY.md is also where the honest answer to "what does it collect" lives — as a list, not an assurance.

License

MIT.