repo-auditor

Async CLI that audits Python repositories on GitHub for engineering quality — tests, CI, typing, packaging, docs and security — and explains every finding.

Problem

Judging whether a Python repository is well engineered is manual and inconsistent. Everyone looks at the same things — is there a README? are there tests? is CI running? is the package installable? is a secret committed? — but without explicit criteria the verdict changes with who is looking and on which day.

Existing badges and services hand back a number without saying how they reached it, which makes the number useless for acting on. Here, a score without a justification is a bug: the report is a list of findings, each with the evidence behind it and a concrete recommendation. The number is only a summary of that list. The exact method is public in docs/scoring.md.

Features

• Audit a single repository or every public repository of a user
• Eight dimensions: security, documentation, testing, CI/CD, static quality, packaging, activity and community
• Every finding carries evidence and, when not OK, an actionable recommendation
• Bounded concurrency, proactive rate-limit handling, retry with exponential backoff
• Local SQLite cache with TTL and conditional (ETag) revalidation — a warm cache is nearly free
• Text, JSON and Markdown output; severity-based exit codes for CI use
• Never executes the audited repository's code; a detected secret is reported by file and type, never by value

Requirements

• Python 3.11+
• Optionally a GITHUB_TOKEN — without one the GitHub API allows 60 requests/hour instead of 5000, which caps a single run at roughly 14 repositories and prints a warning

Installation

uv tool install gh-repo-auditor      # or: pipx install gh-repo-auditor
repo-auditor --help

The distribution is published as gh-repo-auditor, but the command it installs is repo-auditor. The prefix exists because PyPI rejects repo-auditor as too similar to the unrelated project RepoAuditor; it also states what is audited — GitHub repositories.

For local development:

uv sync
uv run repo-auditor --help

Usage

export GITHUB_TOKEN=...            # optional but recommended

repo-auditor audit psf/requests            # audit one repository
repo-auditor audit-user octocat            # audit all of a user's public repos
repo-auditor compare psf/requests urllib3/urllib3
repo-auditor audit psf/requests --json     # stable JSON (see docs)
repo-auditor audit psf/requests --markdown --output report.md
repo-auditor rules                         # show active dimensions and weights
repo-auditor cache clear

Real output, auditing this author's own ubuntu-doctor (the tool applied to its own portfolio):

david-oliveira-dev/ubuntu-doctor  score 92.5  INFO
  [INFO] activity.has_release: no published release was found
          → Tag a release so users can depend on a specific version.
  [INFO] community.security_policy: no SECURITY.md was found
          → Add a SECURITY.md describing how to report vulnerabilities.
  [OK] security.no_secret_patterns: no content matching a known secret pattern was found
  [OK] documentation.readme_substance: README has 6736 characters and 18 headings
  [OK] testing.suite_present: found 18 test file(s), e.g. tests/integration/test_cli.py
  [OK] ci.runs_lint_and_tests: a workflow runs both linting and tests
  [OK] packaging.src_layout: the package uses a src/ layout
  ...

Exit codes

This makes repo-auditor usable as a CI gate: fail the build if a critical finding appears.

JSON contract

--json emits a versioned, stable schema (schema_version). It is the tool's public contract — field names and nesting change only with a version bump. See docs/scoring.md for what each field means.

Configuration

The token is read only from the environment. It is never accepted as a command-line argument, never logged, and never written to the cache. See .env.example.

Testing

make check      # ruff + mypy (strict) + pytest with coverage

Every test runs offline. HTTP is exercised with respx against sanitised fixtures of real API responses; rules are exercised against hand-built snapshots. Minimum coverage: 85%.

Design decisions

• docs/architecture.md — the layers and the collect/evaluate boundary, with a diagram
• docs/scoring.md — the public scoring method
• docs/api-budget.md — measured API limits, the endpoints used, and the cost of an audit
• Architecture decision records in docs/adr/: snapshot separates collection from evaluation, UNKNOWN never penalizes, SQLite cache over an external service

Limitations

Stated up front, because a quality tool that overstates its reach is its own counter-example:

• Does not analyse source code — no AST, no complexity metrics
• Does not run the audited repository's tests, and never executes its code
• GitHub only; Python repositories only
• Community and activity signals are heuristics, not measurements
• Secret detection reports a pattern compatible with a secret, never a certainty — and never prints the value it matched

Roadmap

v1.1 adds audit history over time; v1.2 user-defined rules; v2.0 would require abstracting the provider layer to support GitLab.

Contributing

Issues and pull requests are welcome. Please read the PR template first.

License

MIT — see LICENSE.

Contact

David Oliveira — davidoliveira.devbr@gmail.com