Source-Code Metabase

A structured, human-readable knowledge base of the analysed source-code ecosystem. Designed to be loaded as context for LLM SAST so that cross-repository taint analysis becomes possible — sources in one repo, sinks in another, internal libraries that act as transparent pass- throughs to dangerous APIs.

Canonical field definitions: SCHEMA.md

Why this exists

Conventional SAST tools work one repository at a time and cannot follow:

1. Cross-repo SQL injection — service A constructs a SQL fragment and forwards it as a payload to service B which executes it.
2. Internal-library black boxes — shared wrappers hide JDBC/JPA sinks.
3. Cross-repo PII flow — ingress → queue → store → log → third party.
4. Crypto agility — algorithms and keys decided in config or shared libs.
5. Dangerous request payloads — e.g. raw SQL accepted on an HTTP endpoint (acme/sql-runner-api /query).

The metabase captures cross-repo facts so an LLM (or human) can reason about the full path, not a single repo in isolation.

Contents at a glance

Per-repo (gitignored after generation)

Cross-repo catalogues (mostly gitignored)

Hand-authored (committed)

Scripts (committed)

Building the metabase

Prerequisites: source repos cloned to your --repos-root path, uv sync, corp network.

# Incremental update — skips repos whose git SHA hasn't changed (~fast after first run)
uv run src2sink-build \
  --repos-root repos --metabase-root metabase \
  --internal-groups-file internal-groups.json --api-clients api-clients.json

# Force full re-extract of all repos (use after downloading new repo snapshots
# that are not git checkouts, or after changing internal-groups/api-clients config)
uv run src2sink-build \
  --repos-root repos --metabase-root metabase --force \
  --internal-groups-file internal-groups.json --api-clients api-clients.json

# Single repo — re-extracts only that repo, but re-aggregates taint/graphs/phase3
# from all existing JSONs so cross-repo outputs stay consistent
uv run src2sink-build \
  --repos-root repos --metabase-root metabase \
  --repo acme/sql-runner-api \
  --internal-groups-file internal-groups.json --api-clients api-clients.json

# Re-aggregate taint + graphs from existing v2 JSONs only (fast, no re-extraction)
uv run src2sink-build \
  --repos-root repos --metabase-root metabase --graphs-only \
  --internal-groups-file internal-groups.json --api-clients api-clients.json

# Phase 3 only (PII lifecycle, ROPA, auth/crypto, cross-repo phone flows)
uv run src2sink-build \
  --repos-root repos --metabase-root metabase --phase3-only \
  --internal-groups-file internal-groups.json --api-clients api-clients.json

# Resolve flagged library-source-map entries from on-disk pom.xml only
# (fast, no re-extraction). Walks every pom.xml under --repos-root, reads each
# groupId/artifactId, and for any mapping still flagged (status pending/ambiguous
# or with an empty clone_path) whose coordinate matches a pom, fills in clone_path
# and sets status: cloned. excluded and already-cloned entries are left alone.
uv run src2sink-build \
  --repos-root repos --metabase-root metabase --fix-source-map

# Tests (Phase 4 regression suite)
uv run pytest tests/ -q
uv run pytest tests/ -q -m fleet   # needs metabase/repos v2 JSONs

Build gates

.github/workflows/ci.yml runs six gates on every push, pull request, and weekly (advisories move even when the code does not):

make ci runs everything except opengrep, which needs the external ruleset checkout. Known false positives carry inline # nosec / # nosemgrep annotations with a stated reason rather than blanket rule exclusions.

Release gates

A separate workflow, .github/workflows/release.yml, runs when a v* tag is pushed — and on the 1st of each month as an unattended rehearsal that builds and attests without publishing, so upstream breakage surfaces before a release depends on it.

Publishing runs only after provenance succeeds, so a failure there stops a release before the irreversible step. Permissions are split per job: only publish can mint an OIDC token, only release can write to the repository.

Publishing a version to PyPI: docs/releasing.md. Known small gaps: docs/todo.md.

Verifying a release

The SLSA badge above is a claim, like every badge — it links to this repository's attestations, which is the evidence. Do not take either on trust; the commands below check the artefact you actually downloaded.

Releases carry signed build provenance tying each artefact to the source commit, tag, and workflow run that produced it — so a file claiming to be src2sink can be checked rather than trusted. From 1.0.3 the provenance is generated by an isolated builder (SLSA Build L3); 1.0.2 is L2; 1.0.0 and 1.0.1 have none.

# SLSA provenance, attached to the GitHub release (L3)
slsa-verifier verify-artifact src2sink-1.0.3-py3-none-any.whl \
  --provenance-path multiple.intoto.jsonl \
  --source-uri github.com/mimecast/src2sink --source-tag v1.0.3

# GitHub attestation, for the same artefact
gh attestation verify src2sink-1.0.3-py3-none-any.whl --repo mimecast/src2sink

# installed from PyPI instead (PEP 740 attestation)
python -m pypi_attestations verify pypi --repo mimecast/src2sink src2sink-1.0.3-*.whl

What the claim does and does not cover is in docs/slsa.md — the Build track says nothing about dependencies or source trustworthiness.

Incremental behaviour: each re-run compares the repo's current git HEAD SHA against the SHA stored in the existing per-repo JSON. Repos that haven't changed are skipped (skip group/name (unchanged)); only changed repos are re-extracted. Repos without a .git/ directory (bare downloads) are always re-extracted. Cross-repo graphs and taint catalogues are always re-aggregated at the end regardless of how many repos were skipped, so the metabase stays consistent. Use --force to bypass the SHA check and re-extract everything.

Traces are not run automatically. src2sink-build produces a graphs/traces/INDEX.md that lists existing trace files and shows which endpoints still lack one, but it does not generate traces itself. Run src2sink-trace-batch (or src2sink-trace) as a separate step after the build completes.

Generating traces (separate step)

Traces give a full bidirectional view of a dangerous-payload endpoint: inbound routes, raw-code-payload nodes, SQL sinks, config stores, and upstream callers. They are written to graphs/traces/<name>.md.

Batch — trace all raw-code-payload endpoints at once (recommended):

# First run: generates a report for every endpoint in
# taint/raw-code-payload-endpoints.jsonl
uv run src2sink-trace-batch --metabase-root metabase \
  --internal-groups-file internal-groups.json --api-clients api-clients.json

# Subsequent runs: skip endpoints that already have a report
uv run src2sink-trace-batch --metabase-root metabase \
  --internal-groups-file internal-groups.json --api-clients api-clients.json --skip-existing

Single endpoint:

uv run src2sink-trace \
  --metabase-root metabase \
  --target acme/sql-runner-api \
  --path /query \
  --scan-repos repos \
  --internal-groups-file internal-groups.json \
  --api-clients api-clients.json \
  --output metabase/graphs/traces/sql-runner-api-query.md

--scan-repos is optional; it adds a literal import/URL scan of the cloned source trees to find callers that are not yet in the graph.

After adding new traces, refresh the index:

uv run src2sink-build --repos-root repos --metabase-root metabase --graphs-only \
  --internal-groups-file internal-groups.json --api-clients api-clients.json

Fixing flagged internal-library mappings

After the initial scan, library-source-map.json may contain entries the build couldn't resolve — coordinates with status: pending/ambiguous or an empty clone_path (an internal dependency whose source repo it couldn't locate). Every normal build already tries to repair these automatically at the end, but if you clone the missing repos after a scan you can re-run just the fix without re-extracting anything:

uv run src2sink-build --repos-root repos --metabase-root metabase --fix-source-map

This walks every pom.xml under --repos-root, reads each groupId / artifactId, and for any still-flagged mapping whose coordinate matches a pom it sets clone_path to that repo's path (relative to the repos root) and flips status to cloned. Matching is by exact groupId:artifactId, falling back to a unique artifactId-only match. Entries marked excluded, and those already cloned with a clone_path, are left untouched. It prints how many entries it resolved; anything still flagged afterwards has no matching pom.xml on disk (clone the repo, or set clone_path/status by hand). Once resolved, curate_internal_libraries.py can seed taint tables from that source.

Register another API client library

Add a binding to api-clients.json (see docs/api-clients-json.md for the field reference and the --discover-api-clients drafting flow), then re-run src2sink-build --api-clients api-clients.json (or --graphs-only).

This is how the tool sees hops that regular SAST cannot: a repo that calls a service through its published client library has no URL, host, or service name anywhere in its source, so the binding is the only thing that knows the call crosses a service boundary. If a supplied api-clients.json loads zero bindings the build fails rather than quietly producing a client-blind metabase; check api_clients_binding_count in run-manifest.json and the "API-client binding coverage" table in graphs/service-call-graph.md to confirm detection is live.

How to use this metabase as LLM context

Minimum pack for SAST on one repo

1. repos/<group>/<repo>.md and .json (check schema_version).
2. For each internal dependency, internal-libraries/<coord>.md.
3. Relevant taint rows (filter jsonl by repo column).
4. graphs edges involving that repo (service-call-edges.jsonl, payload-endpoint-producers.jsonl, queue-graph.jsonl).

Dangerous-payload / raw SQL in HTTP body

1. taint/raw-code-payload-endpoints.md (sample) + .jsonl (full).
2. graphs/payload-endpoint-producers.md for known client libraries.
3. graphs/traces/<service>-<path>.md if generated (see graphs/traces/INDEX.md).
4. Target repo v2 JSON: family=raw-code-payload, http-in, sql sinks.

Use phone numbers (not email) as the canonical PII lifecycle example when illustrating field flow — email is ambient in an email-security company (conventions/pii-classification.md).

PII and crypto

• PII lifecycle: graphs/pii-lifecycle.md, graphs/pii-phone-cross-repo.md.
• ROPA draft: ropa/categories-of-personal-data.md.
• Field catalogues: taint/pii-sources.md, taint/pii-sinks.md.
• Crypto: taint/crypto-operations.md, conventions/crypto-agility.md.

Updating and hand-curation

Per-repo files are regenerated wholesale by build_metabase_v2.py. Put durable notes in internal-libraries/ or conventions files, not inside generated output.

Confidence levels

Service-call and producer-index edges label confidence explicitly. Treat low as “investigate”, not “confirmed”.

Dependencies

v2 requires tree-sitter and per-language grammars (via uv sync). Do not add public PyPI fallbacks.

See also

• SCHEMA.md — complete v2 node/edge vocabulary.
• docs/architecture.md — components, data flow, trust boundaries.
• docs/threat-model.md — STRIDE risk register.
• docs/security-privacy-gap-analysis.md — requirements, abuse cases, SRTM.
• docs/operations-security.md — running it safely; data classification and retention.
• docs/sast-report.md — self-review of this tool's own code.
• docs/api-clients-json.md — the internal-service binding file.
• metabase-usage.md — using the output as LLM context.
• docs/releasing.md — publishing a version to PyPI.
• docs/slsa.md — build provenance: what is claimed, how to verify it, and what it does not cover.
• CHANGELOG.md — what shipped in each version.
• docs/todo.md — known small gaps.
• /ai-sast-scanner skill — SAST prompt this metabase feeds.