Source-Code Metabase

A structured, human-readable knowledge base of the analysed source-code ecosystem. Designed to be loaded as context for LLM SAST so that cross-repository taint analysis becomes possible — sources in one repo, sinks in another, internal libraries that act as transparent pass- throughs to dangerous APIs.

Canonical field definitions: SCHEMA.md
Roadmap / gaps: NEXT_STEPS.md

Why this exists

Conventional SAST tools work one repository at a time and cannot follow:

1. Cross-repo SQL injection — service A constructs a SQL fragment and forwards it as a payload to service B which executes it.
2. Internal-library black boxes — shared wrappers hide JDBC/JPA sinks.
3. Cross-repo PII flow — ingress → queue → store → log → third party.
4. Crypto agility — algorithms and keys decided in config or shared libs.
5. Dangerous request payloads — e.g. raw SQL accepted on an HTTP endpoint (acme/sql-runner-api /query).

The metabase captures cross-repo facts so an LLM (or human) can reason about the full path, not a single repo in isolation.

Contents at a glance

Per-repo (gitignored after generation)

Cross-repo catalogues (mostly gitignored)

Hand-authored (committed)

Scripts (committed)

Building the metabase

Prerequisites: source repos cloned to your --repos-root path, uv sync, corp network.

# Incremental update — skips repos whose git SHA hasn't changed (~fast after first run)
uv run src2sink-build \
  --repos-root repos --metabase-root metabase \
  --internal-groups-file internal-groups.json --api-clients api-clients.json

# Force full re-extract of all repos (use after downloading new repo snapshots
# that are not git checkouts, or after changing internal-groups/api-clients config)
uv run src2sink-build \
  --repos-root repos --metabase-root metabase --force \
  --internal-groups-file internal-groups.json --api-clients api-clients.json

# Single repo — re-extracts only that repo, but re-aggregates taint/graphs/phase3
# from all existing JSONs so cross-repo outputs stay consistent
uv run src2sink-build \
  --repos-root repos --metabase-root metabase \
  --repo acme/sql-runner-api \
  --internal-groups-file internal-groups.json --api-clients api-clients.json

# Re-aggregate taint + graphs from existing v2 JSONs only (fast, no re-extraction)
uv run src2sink-build \
  --repos-root repos --metabase-root metabase --graphs-only \
  --internal-groups-file internal-groups.json --api-clients api-clients.json

# Phase 3 only (PII lifecycle, ROPA, auth/crypto, cross-repo phone flows)
uv run src2sink-build \
  --repos-root repos --metabase-root metabase --phase3-only \
  --internal-groups-file internal-groups.json --api-clients api-clients.json

# Resolve flagged library-source-map entries from on-disk pom.xml only
# (fast, no re-extraction). Walks every pom.xml under --repos-root, reads each
# groupId/artifactId, and for any mapping still flagged (status pending/ambiguous
# or with an empty clone_path) whose coordinate matches a pom, fills in clone_path
# and sets status: cloned. excluded and already-cloned entries are left alone.
uv run src2sink-build \
  --repos-root repos --metabase-root metabase --fix-source-map

# Tests (Phase 4 regression suite)
uv run pytest tests/ -q
uv run pytest tests/ -q -m fleet   # needs metabase/repos v2 JSONs

Build gates

.github/workflows/ci.yml runs six gates on every push, pull request, and weekly (advisories move even when the code does not):

make ci runs everything except opengrep, which needs the external ruleset checkout. Known false positives carry inline # nosec / # nosemgrep annotations with a stated reason rather than blanket rule exclusions.

Publishing a version to PyPI: docs/releasing.md. Known small gaps: docs/todo.md.

Verifying a release

Releases from 1.0.2 carry signed build provenance (SLSA Build L2) tying each artefact to the source commit, tag, and workflow run that produced it — so a file claiming to be src2sink can be checked rather than trusted:

# downloaded from the GitHub release
gh attestation verify src2sink-1.0.2-py3-none-any.whl --repo mimecast/src2sink

# installed from PyPI (PEP 740 attestation)
python -m pypi_attestations verify pypi --repo mimecast/src2sink src2sink-1.0.2-*.whl

1.0.0 and 1.0.1 predate this and have no provenance. What the claim does and does not cover — and the plan for Build L3 — is in docs/slsa.md.

Incremental behaviour: each re-run compares the repo's current git HEAD SHA against the SHA stored in the existing per-repo JSON. Repos that haven't changed are skipped (skip group/name (unchanged)); only changed repos are re-extracted. Repos without a .git/ directory (bare downloads) are always re-extracted. Cross-repo graphs and taint catalogues are always re-aggregated at the end regardless of how many repos were skipped, so the metabase stays consistent. Use --force to bypass the SHA check and re-extract everything.

Traces are not run automatically. src2sink-build produces a graphs/traces/INDEX.md that lists existing trace files and shows which endpoints still lack one, but it does not generate traces itself. Run src2sink-trace-batch (or src2sink-trace) as a separate step after the build completes.

Generating traces (separate step)

Traces give a full bidirectional view of a dangerous-payload endpoint: inbound routes, raw-code-payload nodes, SQL sinks, config stores, and upstream callers. They are written to graphs/traces/<name>.md.

Batch — trace all raw-code-payload endpoints at once (recommended):

# First run: generates a report for every endpoint in
# taint/raw-code-payload-endpoints.jsonl
uv run src2sink-trace-batch --metabase-root metabase \
  --internal-groups-file internal-groups.json --api-clients api-clients.json

# Subsequent runs: skip endpoints that already have a report
uv run src2sink-trace-batch --metabase-root metabase \
  --internal-groups-file internal-groups.json --api-clients api-clients.json --skip-existing

Single endpoint:

uv run src2sink-trace \
  --metabase-root metabase \
  --target acme/sql-runner-api \
  --path /query \
  --scan-repos repos \
  --internal-groups-file internal-groups.json \
  --api-clients api-clients.json \
  --output metabase/graphs/traces/sql-runner-api-query.md

--scan-repos is optional; it adds a literal import/URL scan of the cloned source trees to find callers that are not yet in the graph.

After adding new traces, refresh the index:

uv run src2sink-build --repos-root repos --metabase-root metabase --graphs-only \
  --internal-groups-file internal-groups.json --api-clients api-clients.json

Fixing flagged internal-library mappings

After the initial scan, library-source-map.json may contain entries the build couldn't resolve — coordinates with status: pending/ambiguous or an empty clone_path (an internal dependency whose source repo it couldn't locate). Every normal build already tries to repair these automatically at the end, but if you clone the missing repos after a scan you can re-run just the fix without re-extracting anything:

uv run src2sink-build --repos-root repos --metabase-root metabase --fix-source-map

This walks every pom.xml under --repos-root, reads each groupId / artifactId, and for any still-flagged mapping whose coordinate matches a pom it sets clone_path to that repo's path (relative to the repos root) and flips status to cloned. Matching is by exact groupId:artifactId, falling back to a unique artifactId-only match. Entries marked excluded, and those already cloned with a clone_path, are left untouched. It prints how many entries it resolved; anything still flagged afterwards has no matching pom.xml on disk (clone the repo, or set clone_path/status by hand). Once resolved, curate_internal_libraries.py can seed taint tables from that source.

Register another API client library

Edit src2sink/known_api_clients.py (ApiClientBinding), then re-run src2sink-build (or --graphs-only).

How to use this metabase as LLM context

Minimum pack for SAST on one repo

1. repos/<group>/<repo>.md and .json (check schema_version).
2. For each internal dependency, internal-libraries/<coord>.md.
3. Relevant taint rows (filter jsonl by repo column).
4. graphs edges involving that repo (service-call-edges.jsonl, payload-endpoint-producers.jsonl, queue-graph.jsonl).

Dangerous-payload / raw SQL in HTTP body

1. taint/raw-code-payload-endpoints.md (sample) + .jsonl (full).
2. graphs/payload-endpoint-producers.md for known client libraries.
3. graphs/traces/<service>-<path>.md if generated (see graphs/traces/INDEX.md).
4. Target repo v2 JSON: family=raw-code-payload, http-in, sql sinks.

Use phone numbers (not email) as the canonical PII lifecycle example when illustrating field flow — email is ambient in an email-security company (conventions/pii-classification.md).

PII and crypto

• PII lifecycle: graphs/pii-lifecycle.md, graphs/pii-phone-cross-repo.md.
• ROPA draft: ropa/categories-of-personal-data.md.
• Field catalogues: taint/pii-sources.md, taint/pii-sinks.md.
• Crypto: taint/crypto-operations.md, conventions/crypto-agility.md.

Updating and hand-curation

Per-repo files are regenerated wholesale by build_metabase_v2.py. Put durable notes in internal-libraries/ or conventions files, not inside generated output.

Confidence levels

Service-call and producer-index edges label confidence explicitly. Treat low as “investigate”, not “confirmed”.

Dependencies

v2 requires tree-sitter and per-language grammars (via uv sync). See metabase/docs/phase0-probe.md if resolution fails. Do not add public PyPI fallbacks.

See also

• SCHEMA.md — complete v2 node/edge vocabulary.
• docs/implementation-plan.md — phased plan and progress record.
• docs/architecture.md — components, data flow, trust boundaries.
• docs/threat-model.md — STRIDE risk register.
• docs/security-privacy-gap-analysis.md — requirements, abuse cases, SRTM.
• docs/operations-security.md — running it safely; data classification and retention.
• docs/sast-report.md — self-review of this tool's own code.
• docs/api-clients-json.md — the internal-service binding file.
• metabase-usage.md — using the output as LLM context.
• docs/releasing.md — publishing a version to PyPI.
• docs/slsa.md — build-provenance plan (SLSA Build L2 → L3).
• CHANGELOG.md — what shipped in each version.
• NEXT_STEPS.md — roadmap · docs/todo.md — known small gaps.
• /ai-sast-scanner skill — SAST prompt this metabase feeds.