eng-effectiveness-metrics-tools

Tools for measuring and tracking engineering effectiveness, with a focus on agentic coding tool adoption.

agentic-adoption-scan

A CLI tool that scans all repositories in a GitHub organization to detect adoption of agentic coding tools (Claude Code, GitHub Copilot, Cursor, MCP servers, evals frameworks, and more). Produces tidy-format CSV data suitable for analysis and visualization.

Install

macOS (Intel and Apple Silicon):

curl -fsSL "https://github.com/posit-dev/eng-effectiveness-metrics-tools/releases/latest/download/agentic-adoption-scan_darwin_$(uname -m | sed 's/x86_64/amd64/').tar.gz" | tar -xz && sudo mv agentic-adoption-scan /usr/local/bin/

Linux (x86_64 and arm64):

curl -fsSL "https://github.com/posit-dev/eng-effectiveness-metrics-tools/releases/latest/download/agentic-adoption-scan_linux_$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/').tar.gz" | tar -xz && sudo mv agentic-adoption-scan /usr/local/bin/

Via pip/pipx (macOS, Linux, Windows):

pipx install agentic-adoption-scan

Prerequisites

The tool uses the gh CLI for GitHub authentication. Make sure you have it installed and authenticated:

gh auth login

Usage

Scan an organization for agentic tool indicators:

agentic-adoption-scan scan --org your-org --output results.csv

Fetch and analyze content of detected indicators:

agentic-adoption-scan inspect --org your-org --scan-results results.csv --output inspect-results.csv

Generate a customizable indicators config file:

agentic-adoption-scan init-config --output indicators.yaml

Run as an MCP server for Claude Code integration:

agentic-adoption-scan serve

Scan flags

What it detects

The tool checks for 25+ built-in indicators across 7 categories:

You can extend or override these indicators via a YAML config file (init-config generates a starter).

Output format

Scan results are written as tidy CSV — one row per (repo × indicator) observation:

scan_timestamp,org,repo,repo_visibility,repo_language,repo_pushed_at,category,indicator,found,file_path,details

This format makes it easy to filter, pivot, and visualize in R, Python, or any BI tool.

MCP server

The tool can run as an MCP server, exposing tools for use directly within Claude Code:

{
  "mcpServers": {
    "agentic-adoption-scan": {
      "command": "agentic-adoption-scan",
      "args": ["serve"]
    }
  }
}

Available MCP tools: scan_org, inspect_repo, list_indicators, get_repo_summary, get_adoption_summary.

Deploying to Posit Connect

You can deploy the MCP server to Posit Connect so that AI clients across your organization can access it without running anything locally. The connect/ directory contains a Python entry point that wraps the binary for Connect's ASGI runtime.

Prerequisites

• rsconnect-python installed: pip install rsconnect-python
• The gh CLI installed and available in PATH on the Connect server (the binary uses it for GitHub API calls)
• A Posit Connect server URL and API key

1. Set environment variables

In your Connect content's Vars settings (or pass via --environment at deploy time), set:

GITHUB_TOKEN=<your-github-pat>

The gh CLI reads GITHUB_TOKEN automatically, so no interactive gh auth login is needed on the server.

2. Write the manifest

cd connect/
rsconnect write-manifest fastapi --overwrite --entrypoint server:mcp .

This creates connect/manifest.json (and connect/requirements.txt if not already present) for later or CI-driven deployments.

3. Deploy

rsconnect deploy fastapi \
  --server https://your-connect-server.example.com \
  --api-key YOUR_API_KEY \
  --entrypoint server:mcp \
  --title "agentic-adoption-scan" \
  .

Or if you have already saved your server with rsconnect add:

rsconnect deploy fastapi \
  --name your-server-nickname \
  --entrypoint server:mcp \
  --title "agentic-adoption-scan" \
  .

4. Use from Claude Code

Once deployed, add the Connect-hosted MCP server to your Claude Code configuration:

{
  "mcpServers": {
    "agentic-adoption-scan": {
      "type": "streamable-http",
      "url": "https://your-connect-server.example.com/content/<content-id>/mcp"
    }
  }
}

Replace <content-id> with the numeric ID shown in the Connect dashboard for this content item.

Performance tip

Set Min processes to 1 in the content's runtime settings so the server is always warm and avoids cold-start delays when clients connect.

Other scripts

Record a deployment event into GitHub:

./record-deployment.sh posit-dev/eng-effectiveness-metrics-tools main testing

Releases

Releases are fully automated via python-semantic-release on every merge to main. The version bump is determined automatically from Conventional Commit PR titles (enforced by the pr-title workflow):

On every merge to main, semantic-release analyzes commits since the last tag. If there are releasable changes, it creates a CHANGELOG.md entry, commits it, tags the new version (e.g. v0.2.0), and publishes a GitHub release — all in one step. That tag push then triggers the publish workflow, which:

1. Builds native binaries for macOS (amd64/arm64) and Linux (amd64/arm64) via GoReleaser
2. Publishes Python wheels for all platforms to PyPI via go-to-wheel

One-time setup required

Deploy key — semantic-release pushes back to main and needs to trigger the publish workflow. The default GITHUB_TOKEN cannot do this, so an SSH deploy key is required:

ssh-keygen -t ed25519 -C "semantic-release" -f deploy_key -N ""

1. Add deploy_key.pub as a repo Deploy key with write access: Settings → Deploy keys
2. Add deploy_key (private) as an Actions secret named DEPLOY_KEY: Settings → Secrets and variables → Actions

PyPI Trusted Publisher — the publish workflow uses OIDC Trusted Publishing (no API token needed). Configure a Trusted Publisher on PyPI for this repository pointing to the publish.yml workflow and the pypi environment.