skills-telemetry

Telemetry and metadata capture for standards-as-skills — attributes token cost and outcomes to individual skills across Claude Code and GitHub Copilot, joined to policy (OPA/Rego), delivery (Linear/GitHub) and quality data. Design rationale: docs/design-proposal.md.

Metadata only. No prompt, response or file content is ever emitted; the collector drops it again as defence in depth. If it runs on your machine, docs/for-developers.md lists every field that leaves it — and export STDTEL_DISABLED=1 turns it off entirely.

When can I trust these numbers?

Telemetry starts producing plausible-looking ratios on day one. Most of them mean nothing yet. The phases below are set by data volume, not elapsed time — how long each takes depends entirely on team size, and a small team may never leave the first one.

The hard floor: below 30 merged PRs per arm, or fewer than 5 developers, report descriptively and make no comparative claim.

Early on, the most valuable thing this data does is find its own faults. A high share of unversioned skills or of work joined to no change request bounds every later conclusion, and both are fixable now — see stdtel-onboard, and stdtel-doctor's branch-identity check. No branch-naming convention is needed (ADR-013).

The failure mode this exists to prevent: reading a scorecard after two weeks, seeing a skill "underperform" across nine PRs, and deprecating it. Nine PRs cannot distinguish a bad skill from a quiet fortnight. Full derivations and the assumptions behind every figure: docs/evaluation-power.md. Worked examples of asking these questions, including the ones the data cannot answer: docs/insight-walkthroughs.md.

Documentation

Layout

skills/<name>/SKILL.md      skill catalogue with validated front-matter (the contract)
stdtel/manifest.py          front-matter parser + `stdtel-validate` CI gate
stdtel/hooks/cli.py         Claude Code hooks: session-start | pre-tool-use | post-tool-use | stop
stdtel/transcript.py        incremental JSONL reader: requests, usage, skill loads, turns, sub-agents
stdtel/artefact.py          the capture contract: kinds and their attribute allowlists
stdtel/exporter.py          std.artefact.activation spans via OTLP/HTTP (content scrubbed)
stdtel/enrich.py            join keys: std.branch.hash, commit patch-ids, std.repo, std.team, std.harness
stdtel/skillmap.py          generates collector/copilot-skill-map.yaml for Copilot tool-call mapping
collector/otel-collector.yaml  drop content → normalise gen_ai.* → map Copilot skills → pseudonymise → spanmetrics
deploy/docker-compose.yml   collector + Tempo + Prometheus + Grafana + Postgres; `langfuse` profile optional
deploy/smoke.sh             eight-hop verification ladder (`mise run smoke`)
collector/overlay-*.yaml    merged over the base config; `none` is the default no-op, `langfuse` adds an exporter
warehouse/schema.sql        capture tables, change_request (+ commits, tickets), activation_change_request view
warehouse/scorecard.sql     weekly per-skill scorecard → keep / refine / review-merge / deprecate
warehouse/load_traces.py    Tempo → Postgres loader
eval/run_eval.py            offline with/without-skill eval, real OPA grading (`--dry-run` is a smoke test)
eval/power.py               generates every table in docs/evaluation-power.md; CI checks it is current
policies/                   Rego behind the primary metric: logging.*, telemetry.manifest_valid
agents/                     skill-scorecard-analyst: keep / refine / merge / deprecate from the data
warehouse/load_delivery.py  the GitHub adapter → change_request / ticket / defect; policy_result from CI
examples/settings.*.json    hook + OTel wiring: `global` installs once, `project` overrides per repo
docs/adrs/                  six ADRs: distribution, delivery data, hook constraints, identity, integrity, Langfuse
plugin.json                 Agent Plugins v1 manifest (portable `skills/` is the shared half)
.claude-plugin/             Claude Code plugin + marketplace manifest
hooks/, com.github.copilot/ per-harness hook manifests, generated from stdtel/install.py::EVENTS
stdtel/install.py           `stdtel-install`: absolute-path resolution + settings merge
mise.toml                   toolchain (Python 3.13) + `.venv` + tasks wrapping the Makefile

Quick start

Two separate jobs: install the harness wiring once for your user, then onboard each project you want attributed telemetry from. Skipping the second step still gives you spans — the skills just come through as unversioned, with std.team=unknown.

1. Install once, globally

uv tool install stdtel        # or: pipx install stdtel
stdtel-install settings       # hooks and Claude Code's own telemetry, into ~/.claude/settings.json
# with the plugin installed, it registers the hooks; add only the telemetry, or each hook fires twice:
# stdtel-install settings --native-only

Two halves, and neither updates the other. The plugin ships hook registration, the launcher, the skills and the agent. The package ships everything that runs. A plugin without the package produces no data and no error, because the launcher exits 0 in silence — it says so once at session start, which is the only place it can, since stdtel-doctor is part of the package too. Keep them in step:

uv tool install stdtel --force --refresh          # the package
claude plugin update stdtel@amiable-standards     # the plugin

stdtel-doctor reports drift between them. To run from a checkout instead: uv tool install /path/to/skills-telemetry.

stdtel-install resolves the absolute path of the stdtel-hook it was installed alongside and writes that into the config. This is not cosmetic: hook processes get a non-login sh -c and inherit whatever PATH launched the harness, so a bare stdtel-hook is unresolvable whenever a version manager (mise, asdf, pyenv) or an activated venv is what put it there. The same reason pre-commit bakes sys.executable into the git hook it generates. stdtel-install where prints the path it will use; --dry-run shows the JSON without writing.

Claude Code also strips OTEL_* from every subprocess it spawns, so point the exporter at your collector with the STDTEL_-namespaced variables, which survive:

export STDTEL_OTLP_ENDPOINT=http://collector.internal:4318   # default: http://localhost:4318
export STDTEL_OTLP_TIMEOUT=2                                 # seconds; bounds a dead-collector stall

With no STDTEL_SKILLS_ROOT set, the catalogue is read from ~/.claude/skills. Symlink your standards there and every project gets versioned spans:

ln -s "$PWD/skills/structured-logging" ~/.claude/skills/structured-logging
stdtel-validate ~/.claude/skills      # same contract gate CI runs

As a plugin

The repo is laid out for three loaders at once (ADR-001), so it installs as a plugin without a separate packaging step:

/plugin marketplace add amiable-dev/skills-telemetry
/plugin install stdtel@amiable-standards

The shipped hooks/hooks.json carries the bare command name, because a distributed manifest cannot know your install path — run stdtel-install settings afterwards to bind it to an absolute one.

Copilot

Copilot needs no code from us. It emits first-party OpenTelemetry with per-tool-call spans and token counts; point it at the same collector (user settings, COPILOT_OTEL_* env vars, or the enterprise managed-settings.json telemetry block for a fleet).

If you do run our hooks on Copilot as well, install them from com.github.copilot/hooks/hooks.json, which sets STDTEL_HARNESS per hook. That is load-bearing: VS Code Copilot reads ~/.claude/settings.json, and its snake_case payload dialect is indistinguishable from Claude Code's — without that env block, Copilot activity is recorded as claude-code.

2. Onboard a project

Per-project overrides go in <project>/.claude/settings.json — Claude Code merges them over the user file, so repeat only what differs. Do not repeat the hooks block: it is already registered globally and a second copy fires each hook twice.

cd ~/projects/payments-api
mkdir -p .claude && cp ~/projects/skills-telemetry/examples/settings.project.json .claude/settings.json
$EDITOR .claude/settings.json      # STDTEL_TEAM is the one you must set
git checkout -b add-audit-log     # any name: work joins its PR by branch identity (ADR-013)

Then verify the loop end to end:

claude                                     # invoke a skill in the project
cat ~/.stdtel/sessions/*.json              # a window with skill, version, tool_use_id
curl -s 'http://localhost:3200/api/search?tags=name%3Dstd.artefact.activation' | jq '.traces[0]'

A std.skill.version of unversioned means the name in the transcript matched no SKILL.md in any root — check STDTEL_SKILLS_ROOT and that the skill's front-matter name matches what you invoked.

Copilot: enable managed OTel export (VS Code / CLI) pointing at the same collector with resource attributes std.harness=copilot-vscode, std.team=<team>; collector/otel-collector.yaml maps catalogued skill tool-calls onto std.skill.*.

Span schema

Two span types, emitted at Stop. std.session.cost is the session's total spend, from the transcript. std.artefact.activation records that an artefact ran, and for a turn, sub-agent or compaction what the transcript observed it spend. A skill activation carries no cost since 0.9.0 (#117): what a skill cost is the harness's own requests named for it, in skill_activation_cost (ADR-014). Never add session cost to the per-request figures; it is the harness's total, kept to check them against.

std.session.cost

Emitted once per turn that made any LLM request, whether or not a skill was loaded — a session that never loads a skill is still spend, and excluding it would silently understate cost per PR.

std.artefact.activation

One span name for four artefacts, discriminated by std.artefact.kind. The per-kind attribute allowlist in stdtel/artefact.py is the contract — an attribute belonging to another kind is dropped and reported rather than emitted (ADR-009).

Every kind also carries std.artefact.source: hook when the harness reported it, transcript when it was read from the session directory because the hook has never fired on this machine. An inference and an observation are not the same measurement and a query can tell them apart.

kind=skill additionally carries:

Known limitations

• Tool counts are counts only. PostToolUse fires for every tool, so tool_input and tool_response carry commands, file contents and diffs — scrub() refuses those attribute names outright and a test asserts nothing leaks.
• Skill name is parsed from the Skill tool input in hooks (the field is skill, verified against 120 real invocations). Parsing is isolated in hooks/cli.py::_skill_from_payload.
• std.skill.trigger reports the transcript's caller.type where present, else unknown — it is never guessed. A skill you typed as /name is user-slash; a catalogued skill named first in a /loop argument (/loop /epic-loop …) is loop. Both are read from the transcript (std.artefact.source=transcript), because neither goes through the Skill tool (#144). loop is an inference — the harness activates only /loop — so filter it out when you want only what the harness activated.
• Copilot granularity is per turn; use Claude Code's finer data for within-harness tuning only.
• Per-skill cost needs Claude Code's own telemetry on (stdtel-install settings, or --native-only with the plugin). Without it a skill's runs are counted as unmeasured, never as free.
• A turn emits one span per Stop carrying that slice's delta, so two Stops inside one turn produce two rows that sum correctly. Count turns with count(DISTINCT prompt_id), never count(*).
• subagent and compaction are Claude Code only. Copilot has no confirmed equivalent, so those hooks are not registered there rather than registered and silently never firing.
• gen_ai.* conventions are still Development upstream; extend transform/normalise as names move.

Contributing

Bug reports, questions and PRs are all welcome. Start with CONTRIBUTING.md — it covers the setup (mise run ci is the whole gate) and the conventions that the tests enforce, including the ones that will surprise you.

This tool runs inside your coding sessions, so SECURITY.md is also where the honest answer to "what does it collect" lives — as a list, not an assurance.

License

MIT.