repo-knowledge-graph 0.2.0

Clone a GitHub repo and emit an RDF knowledge graph plus AI context graph JSON.

repo-knowledge-graph

Turn any GitHub repository into a structured knowledge graph — in seconds — so AI tools, RAG pipelines, and onboarding assistants can understand your codebase without reading every file.

---

Why this accelerator exists

Every AI coding tool, onboarding assistant, or RAG system faces the same cold-start problem: it needs to understand a codebase before it can help with it. The conventional approach is to dump raw files into a context window, hope the model navigates the noise, and repeat for every new repo or new engineer.

repo-knowledge-graph eliminates that cold start. It takes a single GitHub URL and produces a machine-readable map of the repository — files, languages, classes, functions, imports, dependencies, and directory structure — in both a semantic RDF graph and a LLM-optimised JSON graph. The outputs are immediately consumable by:

• AI agents and Copilots (via context_chunks.jsonl or the Python API)
• RAG / vector search pipelines (each JSONL chunk is self-contained and has a stable id)
• Graph databases (Turtle / JSON-LD loads directly into any SPARQL / RDF store)
• Code intelligence dashboards (the manifest is a structured file index with language counts and dependency inventory)
• Onboarding docs generators (the README snippet + dependency nodes give instant project overview)

---

Value and time savings

Activity	Without this accelerator	With this accelerator
AI context prep for a new repo	Engineer manually curates files, writes summaries — 2–4 hours	Run one CLI command — under 30 seconds
RAG ingestion pipeline	Custom crawlers, chunkers, and deduplication — 1–2 weeks dev	Drop context_chunks.jsonl straight into your vector DB loader — ready immediately
Onboarding a new developer	"Read the code, ask Slack" — days to get mental model	Share manifest.json + repo graph; structured summary of every module and dependency — minutes
Dependency audit / compliance	Manual inspection of each package.json / composer.json — hours	All declared packages extracted as typed edges in the graph — automatic
Reproducible AI answers	AI tool has no idea which version of the code it answered about	Every graph is stamped with commit SHA + branch — fully reproducible
Multi-language codebases	Different scripts needed per language	Single command handles Python, PHP, JS, TS, Go, Java, Rust, and more

Dynamic, need-sized context windows (especially for large repos)

You do not need the entire repository in the model’s context. After generation, context_graph.json, manifest.json, and context_chunks.jsonl act as a retrieval index plus graph: at inference time you can assemble only the chunks that match the current task (feature area, package, path, or developer question), instead of loading millions of lines into one prompt.

In short: That model is correct — dynamic, need-sized context windows at inference time, not “everything in context” — as long as you treat the outputs as a retrieval index + graph, plan re-generation or incremental updates when you need freshness against the latest commit, and plug in a retrieval/orchestration layer on top (vector search, graph traversal, or an agent that fetches nodes by id). The graphs themselves are pinned to the scanned commit SHA; query-time selection is dynamic; repo freshness is a separate pipeline concern.

---

What it produces

Running repo-kg <github-url> on an-prog/sample_billing_app — a PHP/JS billing system — produces graph artifacts and, optionally, a wiki in under 10 seconds:

kg_out/
├── knowledge_graph.ttl       ← RDF/Turtle (semantic graph, triple stores, SPARQL)
├── knowledge_graph.jsonld    ← JSON-LD (linked data, semantic web tooling)
├── context_graph.json        ← Full LLM context graph  (nodes + typed edges)
├── manifest.json             ← Lightweight file index  (RAG, dashboards, search)
├── context_chunks.jsonl      ← Per-entity JSONL chunks (one per line, RAG-ready)
└── wiki/                     ← Karpathy-style markdown knowledge base
  ├── index.md
  ├── overview.md
  ├── schema.md
  ├── log.md
  ├── concepts/
  ├── entities/
  └── queries/

Sample run output:

Wrote outputs to kg_out/
  Graph: 3,165 nodes, 3,164 edges
  Files: 2,576 (PHP/JS/JSON/YAML/…)
  Dependencies: 2 manifest(s)
  Wiki pages: 143 (kg_out/wiki)
  Commit: 68fcb3787f2dcbb42631c177e77042c5acc675ea  branch: master

The Knowledge Graph (knowledge_graph.ttl / .jsonld)

The RDF graph uses stable, widely-recognised vocabularies:

Vocabulary	Used for
W3C PROV-O	Provenance: wasDerivedFrom, derivation chain
Schema.org	SoftwareSourceCode, codeRepository, name
code: (custom, open namespace)	Repository, SourceFile, ClassDeclaration, FunctionDeclaration, dependsOn, declares, contains, gitCommitSHA, gitBranch

This makes it interoperable with any SPARQL endpoint, RDF triplestore (Apache Jena, Oxigraph, GraphDB), or linked-data pipeline — no custom schema lock-in.

The Context Graph (context_graph.json)

A plain JSON document with nodes and edges arrays, designed for LLM-friendly consumption:

{
  "meta": {
    "schema_version": 2,
    "source_url": "https://github.com/an-prog/sample_billing_app",
    "commit_sha": "68fcb3787f2dcbb42631c177e77042c5acc675ea",
    "branch": "master",
    "github": { "owner": "an-prog", "repo": "sample_billing_app" }
  },
  "nodes": [
    {
      "id": "repo:an-prog/sample_billing_app",
      "type": "repository",
      "summary": "GitHub repository an-prog/sample_billing_app. README: BillRun is Open-Source Billing..."
    },
    {
      "id": "dep:composer:mongodb/mongodb",
      "type": "dependency",
      "name": "mongodb/mongodb",
      "ecosystem": "composer",
      "version_constraint": "^1.4",
      "kind": "runtime"
    },
    {
      "id": "file:application/controllers/Api.php",
      "type": "source_file",
      "language_hint": ".php",
      "summary": "Source file `application/controllers/Api.php` (.php)."
    }
  ],
  "edges": [
    { "source": "repo:an-prog/sample_billing_app", "target": "dep:composer:mongodb/mongodb", "relation": "dependsOn" },
    { "source": "repo:an-prog/sample_billing_app", "target": "file:application/controllers/Api.php", "relation": "contains" }
  ]
}

Edge types:

relation	Meaning
contains	repository/directory → file
declares	file → class/function/method
imports	Python module → imported name (proper node, not a dangling string)
dependsOn	repository → dependency set (manifest)
dep_has_package	dependency set → individual package

The Manifest (manifest.json)

A lightweight file index — just the repository node, list of files with their language, and stats. Ideal for dashboards, search indexes, and initial context loads where you don't need the full graph.

{
  "stats": {
    "total_nodes": 3165,
    "total_edges": 3164,
    "file_count": 3152,
    "symbol_count": 0,
    "dependency_count": 10
  }
}

JSONL Chunks (context_chunks.jsonl)

One JSON object per line. Each chunk is a self-contained, retrievable unit:

{"id": "repo:an-prog/sample_billing_app", "type": "repository", "summary": "GitHub repository ...", "meta": {...}}
{"id": "file:application/controllers/Api.php", "type": "source_file", "path": "application/controllers/Api.php", "language": ".php", "summary": "...", "meta": {...}}

This is the primary RAG ingestion format: load each line as a document into your vector database with id as the key. No custom parsing required.

---

Key features

• Single command, any public GitHub repo — HTTPS or SSH URL, optional branch/tag
• Commit-pinned provenance — every graph includes commit_sha, short_sha, branch, and tag so AI answers are reproducible and auditable
• Multi-language file graph — Python (deep: classes/functions/imports), PHP, JavaScript, TypeScript, Go, Java, Kotlin, Rust, C/C++ at file granularity; PHP/JS/TS symbols with optional tree-sitter
• Dependency extraction — automatically parses composer.json, package.json, pyproject.toml, requirements*.txt, go.mod, pom.xml into typed dependsOn edges
• Five output formats — RDF Turtle, JSON-LD, full context graph, lightweight manifest, RAG-ready JSONL chunks
• Karpathy-style wiki generation — add --wiki to generate a structured markdown wiki for architecture memory and AI retrieval
• Wiki lifecycle operations — repo-kg-wiki init | ingest | query | lint for maintaining and validating wiki quality
• Agentic orchestration support — KGBuildAgent can run end-to-end graph + wiki builds with optional LLM enrichment hooks
• README-boosted summaries — the top-level README is embedded in the repository node summary for instant project context
• Configurable scan policy — YAML/JSON config for include/exclude globs, max file size, vendor/node_modules toggles
• Token budget control — --max-chunk-tokens N truncates JSONL chunk summaries to fit LLM context windows
• Python API — build_graph(url, options) for programmatic use in notebooks, CI, or agent tool wrappers
• 100% open source — RDFLib (BSD-3), system git, stdlib only; no paid APIs, no vendor lock-in

---

Installation

Requirements: Python 3.10+, git on PATH.

pip install repo-knowledge-graph

Install with OpenAI-powered agent extras:

pip install "repo-knowledge-graph[agents]"

Or from source:

git clone https://github.com/your-org/repo-knowledge-graph
cd repo-knowledge-graph
pip install -e ".[dev]"

Optional: tree-sitter for PHP/JS/TS symbol extraction

pip install tree-sitter tree-sitter-php tree-sitter-javascript tree-sitter-typescript

Without these, PHP/JS/TS files still appear as file-level nodes in the graph.

Quickstart for testers

The productization step for external evaluation is the benchmark bundle command. It creates one canonical folder with KG artifacts, packed prompt contexts, a JSON result file, and a shareable markdown summary.

pip install repo-knowledge-graph

cat > feature_spec.md <<'EOF'
Add validation for stock location capacity and cover it with tests.
EOF

repo-kg benchmark "https://github.com/owner/repo" \
  --feature-file feature_spec.md \
  --output-dir benchmark_out

What you get:

benchmark_out/
├── benchmark_result.json
├── benchmark_result.schema.json
├── kg/
├── prompts/
└── reports/

Use docs/user_guide.md as the handout for testers. The canonical result schema is packaged at src/repo_knowledge_graph/schemas/benchmark_result.schema.json.

---

Usage

CLI

# Basic: clone, scan, and write all outputs to ./kg_out
repo-kg "https://github.com/owner/repo"

# Custom output directory
repo-kg "https://github.com/owner/repo" -o ./my_output

# Specific branch
repo-kg "https://github.com/owner/repo/tree/develop"

# SSH URL
repo-kg "[e-mail]:owner/repo.git"

# Skip cloning — scan a local repo
repo-kg "https://github.com/owner/repo" --clone-dir /path/to/local/clone

# Include directory nodes + truncate chunks at ~1000 tokens
repo-kg "https://github.com/owner/repo" --include-dirs --max-chunk-tokens 1000

# Generate wiki along with graph artifacts
repo-kg "https://github.com/owner/repo" --wiki

# Customize wiki output behavior
repo-kg "https://github.com/owner/repo" --wiki --wiki-dir-name wiki_docs --wiki-overwrite-existing

# Custom scan policy from a config file
repo-kg "https://github.com/owner/repo" --config scan_config.yaml

# Build a tester-ready benchmark bundle
repo-kg benchmark "https://github.com/owner/repo" --feature-file feature_spec.md -o benchmark_out

Benchmark bundle output

The benchmark entrypoint writes a single canonical layout:

Path	Purpose
benchmark_result.json	Machine-readable comparison result for collection and analysis
benchmark_result.schema.json	JSON schema for validating the result bundle
kg/	Full KG build artifacts for repo structure and debugging
prompts/	Feature spec plus baseline and graph-guided packed contexts
reports/benchmark_report.md	Human-readable summary for testers to share

The result JSON includes the benchmark input, repository metadata, KG stats, role budgets, baseline vs graph-guided estimates, retrieval diagnostics, and an assessment.status field of improved, mixed, or degraded.

Wiki operations CLI

# Initialize wiki from a context graph
repo-kg-wiki init kg_out/context_graph.json kg_out/wiki

# Re-ingest latest graph changes
repo-kg-wiki ingest kg_out/context_graph.json kg_out/wiki

# Ask questions over wiki content
repo-kg-wiki query kg_out/wiki "How does authentication flow?"

# Validate wiki structure/frontmatter
repo-kg-wiki lint kg_out/wiki

Example scan_config.yaml:

include_globs:
  - "src/**"
  - "lib/**"
exclude_globs:
  - "vendor/**"
  - "*.min.js"
max_file_bytes: 524288        # 512 KB
include_vendor: false
include_node_modules: false
tree_sitter_languages:
  - PHP
  - JavaScript
  - TypeScript
max_imports_per_file: 100

Python API

from pathlib import Path
from repo_knowledge_graph.api import build_graph, BuildOptions

result = build_graph(
    "https://github.com/owner/repo",
    options=BuildOptions(
        output_dir=Path("kg_out"),
        parse_deps=True,
        include_directories=True,
        max_chunk_tokens=1000,
      build_wiki=True,
      wiki_dir_name="wiki",
      wiki_overwrite_existing=False,
    ),
)

print(result.stats)
# BuildStats(nodes=3165, edges=3164, python_files=0, treesitter_files=0,
#            other_files=2576, dependency_sets=2, commit_sha='68fcb37...', wiki_pages=143)
print(result.wiki_dir)

# Access graph in memory — no disk required
nodes = result.context_graph["nodes"]
meta  = result.context_graph["meta"]

Agentic orchestration API

from pathlib import Path
from repo_knowledge_graph.agents.orchestrator import KGBuildAgent

agent = KGBuildAgent()
result = agent.build(
  "https://github.com/owner/repo",
  output_dir=Path("kg_out"),
  wiki=True,
)

print(result.plan)
print(result.wiki_dir)
print(result.build.stats)

---

Output file reference

File	Format	Best for
knowledge_graph.ttl	RDF/Turtle	SPARQL queries, triple stores (Jena, Oxigraph, GraphDB), compliance audit
knowledge_graph.jsonld	JSON-LD	Linked data pipelines, semantic web tooling, schema.org consumers
context_graph.json	JSON (schema v2)	Full in-memory graph traversal, agent tools, custom RAG loaders
manifest.json	JSON	Dashboards, search index bootstrapping, quick repo overview
context_chunks.jsonl	JSONL	Vector database ingestion (Pinecone, Weaviate, Chroma, pgvector), streaming RAG
wiki/	Markdown tree	Compounding architecture memory, retrieval substrate for agentic query workflows

---

Architecture

GitHub URL
    │
    ▼
github_url.py  ──── parse owner/repo/ref
    │
    ▼
clone_repo.py  ──── git clone --depth 1
    │
    ▼
git_provenance.py ── HEAD SHA · branch · tag
    │
    ▼
scan.py ─────────── walk files (ScanConfig: globs, max size, vendor policy)
    │           ┌── extract/python_ast.py   (stdlib ast)
    │           ├── extract/treesitter.py   (optional, PHP/JS/TS)
    │           └── extract/dependencies.py (composer/npm/pip/go/maven)
    │
    ├──▶ rdf_builder.py ──── knowledge_graph.ttl + .jsonld
    │
    └──▶ context_graph.py ── context_graph.json
              │
              └──▶ chunked_export.py ── manifest.json + context_chunks.jsonl
                          │
                          └──▶ agents/wiki_builder.py ── wiki/index.md + wiki/entities/* + wiki/concepts/*

Wiki runtime operations:
  repo-kg-wiki (agents/wiki_ops.py): init | ingest | query | lint
  KGBuildAgent (agents/orchestrator.py): end-to-end build orchestration

---

Applicability

This accelerator is valuable for any team that:

• Builds AI coding assistants or Copilots and needs structured repo context beyond raw file retrieval
• Runs RAG on codebases and wants pre-chunked, structured, token-budget-aware documents
• Onboards developers to unfamiliar repos and wants AI-generated, structured overviews instead of "read the docs"
• Does software due diligence or audits and needs a machine-readable inventory of files, languages, and declared dependencies
• Builds graph-native knowledge bases with SPARQL / triplestore pipelines and wants code structure as linked data
• Maintains large polyglot monorepos (PHP, JS, Python, Go, Java) and needs a unified, vendor-agnostic structural view

---

Open-source foundation

Component	Library	License
RDF graph + Turtle/JSON-LD	RDFLib	BSD-3-Clause
Python structure	ast (stdlib)	PSF
Multi-language symbols	tree-sitter (optional)	MIT
TOML parsing (Python 3.11+)	tomllib (stdlib)	PSF
Clone	git CLI	GPLv2 (system tool)
Vocabularies	PROV-O, Schema.org, custom code:	W3C / open

No paid APIs. No vendor lock-in. All outputs are standard open formats.

---

Django A/B experiment (max-pack vs graph-guided)

A reproducible harness compares Scenario 1 (max-pack django/**/*.py up to ~380k tokens) vs Scenario 2 (only django/utils/*.py from manifest.json) for the same feature spec, using gpt-5.4 by default. See experiments/README.md. Install extras: pip install -e ".[experiment]".

---

Testing

• Core graph and scanner tests: tests/test_context_graph.py, tests/test_scan.py, tests/test_deps.py, tests/test_github_url.py
• Focused agent/wiki tests: tests/test_agents.py

Run all tests:

py -m pytest -q

Run focused agent tests:

py -m pytest tests/test_agents.py -q

---

Roadmap

The following capabilities are identified as next priorities (see assessment plan):

• Incremental / cached builds — content-hash per file; rebuild only changed paths
• GitHub API enrichment — topics, description, default branch, language stats via REST (no token needed for public repos)
• SHACL / OWL validation — shapes for RDF graph integrity checks
• Private repo support — GITHUB_TOKEN / SSH key passthrough
• Secret-aware export — .gitignore-respecting mode to prevent accidental credential exposure in AI context

---

License

MIT — see LICENSE.