okf-generator 0.1.53

Generate OKF knowledge bundles from codebases — Claude skill + OpenCode integration

OKF Generator

The knowledge layer for AI coding agents

Scan any codebase into structured, agent-ready knowledge — 18 languages, ~100x fewer tokens than reading whole files, zero LLM required.

Quick Start · Why OKF? · Features · Agents · vs RAG · FAQ

Visual Hook & Demos

okf generate → okf lookup → okf diff → okf visualize → okf mcp → okf pairs

Live Viz Graph  ·  Interactive Dev Container  ·  Render App Demo

---

The Why

AI agents waste tokens re-reading code

Every context reset forces agents to re-read entire files to find one function signature. A 600-line file costs 14,000 tokens — just to find one class. Cloud models with 200K windows mask this cost; local SLMs on a MacBook run out of memory immediately.

Vector databases (RAG) don't solve this — they shred code into arbitrary chunks, losing syntax structure, import pathways, and call relationships.

The OKF Edge: deterministic AST extraction

Instead of chunking text, okf-generator uses tree-sitter AST parsers to create a mathematical map of your codebase. Every function, class, module, and dependency becomes a structured node with typed edges (calls, called-by, imports, depends-on).

The result: an average lookup drops from 45,000 tokens → 1,200 tokens — a ~97.3% reduction.

# Before touching any file, get the exact concept card
okf lookup WorldBankConnector

CLASS: WorldBankConnector
Source      : StockAI/RnD/python/connectors/economic_data.py  line 51
Description : Fetches World Bank development indicators via wbdata API.
Methods     : get_indicator, search
Signature   : class WorldBankConnector
Calls       : [wbdata.get_indicator, pandas.DataFrame]
Called-by   : [DataPipeline.fetch_economic]

No re-reading the file. No guessing. No LLM call required.

---

Core Features

Capability	What it means
18 languages	Native AST extraction for Python, JS/TS, Go, Rust, Java, C++, Swift, Kotlin, Ruby, C/C++/C#, SQL, PHP, Dart, Scala, Julia, YAML — one parser file per language, add a new one in minutes
20 manifest formats	Automatic cross-indexing of package.json, Cargo.toml, Dockerfile, go.mod, requirements.txt, Gemfile, pyproject.toml, and 13 more
Deterministic & offline	Core extraction is 100% offline — zero LLM calls, zero API keys, zero vector infrastructure. Same output every run
Cross-reference linker	Resolves imports → dependency edges and function calls → caller/callee across all languages. Multi-hop reasoning without grep
LSP call-graph enrichment	Optional LSP pass (okf enrich --lsp) uses local language servers for compiler-accurate caller/callee resolution — 4 servers mapped (pyright ✅, gopls, rust-analyzer, typescript-language-server), zero token cost
Interactive visualization	Self-contained D3.js HTML dashboard — force-directed graph, search, filter, dark/light theme, no server required
Token efficiency	~140 tokens per concept lookup vs 14,000+ reading whole files. Local SLMs become viable for enterprise-scale codebases
4 enrichment tiers	Optional LLM layer: base (descriptions), deep (examples + side effects), security (risk audit), full (all + semantic links). Runs at generate time or standalone
Multi-provider routing	Route cheap work to local llama.cpp, security audits to Claude — one config file
Incremental updates	okf update re-scans only changed files — SHA256 manifest tracks mtime + content hashes, edge-diff detects cascade changes, writes only dirty concepts. 8 writes for a 1-file edit in a 68-concept bundle
Bundle diff & impact	okf diff --impact shows exactly what changed between runs — which deps affect which modules
Training data export	okf pairs converts any bundle into JSONL fine-tuning pairs (codegen, QA, doc, summarize, crosslink)

How it works

1. Scan  → tree-sitter AST parsers extract functions, classes, modules (17 langs)
2. Link  → cross-reference linker resolves imports→deps, calls→callees
3. Write → OKF v0.2 bundle: structured markdown, mirrors your source tree
4. Update → incremental: SHA256 manifest detects changes, re-parses only dirty files, re-links, edge-diffs, writes only dirty concepts
5. Use   → lookup, ask, diff, visualize, mcp, dashboard — 19 commands
6. Enrich → optional LSP pass (deterministic call-graph refinement) + optional LLM layer: 4 modes, multi-provider routing

Core extraction is fully deterministic and offline. Enrichment is optional, resumable, works with any AI provider. LSP enrichment requires language server binaries on $PATH.

Full language table with per-parser details: docs/languages-and-manifests.md

Supported languages (18)

Python · JavaScript · TypeScript · Go · Java · Rust · Swift · Kotlin · PHP · Dart · Scala · Julia · Ruby · C · C++ · C# · SQL · YAML

Each language lives in its own file under okf/parsers/. Add a new one in minutes — one file + one registry entry.

Domain Classification

Use --domains crossplane to re-classify YAML concepts using data-driven rules. Built-in rules for Crossplane (XRD, Composition, Claim, ProviderConfig, ManagedResource). Custom domains via --domain-rules ./my-rules.yaml. See docs/domain-classification.md.

Manifest formats (20)

requirements.txt · pyproject.toml · package.json · Cargo.toml · Cargo.lock · yarn.lock · pnpm-lock.yaml · go.mod · go.sum · poetry.lock · composer.json · pom.xml · Gemfile · build.gradle / .kts · Package.swift · Dockerfile / Containerfile · docker-compose.yml · mix.exs · project.clj · package.swift

What a bundle looks like

okf_bundle/
├── SUMMARY.md                    ← agent's bird's-eye view
├── index.md                      ← root navigation
├── _dependencies/                ← deps by ecosystem (pip, npm, cargo…)
└── my_project/
    └── connectors/
        ├── index.md              ← lists all concepts in folder
        ├── economic_data.md      ← Module concept
        └── economic_data/
            ├── WorldBankConnector.md   ← Class (signature, methods, calls)
            ├── get_indicator.md        ← Function (params, returns)
            └── search.md               ← Function

Each file follows the OKF v0.2 dialect (extended from Google's OKF v0.1) with structured YAML frontmatter — agents parse it deterministically, no guessing.

---

Quick Start

Install in 30 seconds

# Via pip
pip install okf-generator

# Or one-liner (macOS / Linux)
curl -fsSL https://raw.githubusercontent.com/UmairBaig8/okf-generator/main/scripts/install.sh | bash

Generate your first bundle

okf generate ./my_project ./okf_bundle

That's it — no API key, no vector DB, no config required.

Incremental updates (fast)

After the first generate, use okf update to re-scan only changed files:

okf update ./my_project ./okf_bundle          # incremental (default)
okf update ./my_project ./okf_bundle --force  # full re-scan
okf update ./my_project ./okf_bundle --watch  # continuous watcher mode

A SHA256 manifest tracks file state. Only changed files are re-parsed; edge-diff detects cascade changes; only dirty concepts are written.

Look up any concept

okf lookup WorldBankConnector

# Filter by type
okf lookup --type Function

# Filter by ecosystem
okf lookup --tag ecosystem:pip

# JSON output for programmatic use
okf lookup --json WorldBankConnector

# Fuzzy / camelCase search
okf lookup repo          # finds UserRepository

Visualize as interactive HTML

okf visualize ./okf_bundle viz.html
# Open viz.html in any browser

Launch the live dashboard

okf dashboard ./okf_bundle --open

Opens a 3-panel FastAPI web UI at http://127.0.0.1:8700 with search, detail inspection, and force-directed graphs.

---

Agent Integration

One command per agent

okf install writes the exact rules, instructions, and commands each agent needs. No manual configuration.

# Install for all detected agents at once
okf install all

# Or pick specific agents
okf install claude      # Claude Code skill (auto-triggers on "index my codebase")
okf install cursor      # Cursor rules (.cursorrules)
okf install copilot     # GitHub Copilot instructions
okf install windsurf    # Windsurf rules
okf install cline       # Cline rules
okf install opencode    # OpenCode /lookup command
okf install mcp         # Register MCP server (OpenCode + Claude Desktop)

MCP server (11 tools)

Start an offline-first MCP server that exposes your knowledge bundle to any MCP client:

okf mcp ./okf_bundle --port 4567

Tools: lookup, get_concept, find_callers, find_callees, list_by_file, list_dependencies, bundle_info, list_by_type, search_by_tag, get_related, get_manifest_source.

Connect from Cursor, Cline, Claude Desktop, or any MCP-compatible IDE.

Quick agent context

Add to your agent instructions:
This project has an OKF knowledge bundle at ./okf_bundle/.
- Use `okf lookup <Name>` for full concept context.
- Use `okf lookup --type <Type>` to filter by type.
- Read `SUMMARY.md` for the full knowledge map.

---

Comparison

okf-generator vs RAG / vector search

Capability	okf-generator	RAG / Vector Search
Retrieval	✓ Precise AST lookup (exact symbols)	~ Approximate (chunk similarity)
Token cost per lookup	✓ ~140 tokens	~ Varies by chunk strategy
Cross-reference edges	✓ Calls / called-by / imports	✗ Not supported
Privacy	✓ 100% offline, no data leaves	✗ Needs embeddings API (data may egress)
Search speed	✓ ~3-4ms (indexed)	~ 200-500ms (embed + search)
Dependency manifest parsing	✓ 20 formats automatically	✗ Not designed for this
CI/CD integration	✓ Built-in GitHub Action + impact diff	✗ Custom pipeline required
Training data export	✓ Built-in JSONL pairs	✗ Not a feature
Context compression	✓ ~97% reduction (45K→1.2K tokens)	~ Varies by chunk strategy
Setup	✓ pip install + 1 command	⚠ Vector DB + embedding pipeline

Full comparison with Sourcegraph, Graphify/CodeSee, and dep-graph tools: docs/comparison.md

---

Enterprise & CI/CD

Built-in GitHub Action

A pre-built workflow (.github/workflows/okf-bundle.yml) auto-generates the bundle on every push/PR to main, caches previous bundles per branch, diffs with --impact, and posts a PR comment showing which dependency changes affect which code:

# .github/workflows/okf-bundle.yml
name: OKF Bundle
on:
  push: { branches: [main] }
  pull_request: { branches: [main] }
jobs:
  generate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install okf-generator
      - run: okf generate . okf_bundle
      - run: okf diff .okf_bundle_prev okf_bundle --impact
      - uses: actions/github-script@v7  # post/update PR comment

Fine-tuning data export

Convert your bundle into JSONL training pairs for fine-tuning private local SLMs:

# 5 pair types: codegen, qa, doc, summarize, crosslink
okf pairs ./okf_bundle ./train.jsonl

Docker image

docker pull ghcr.io/umairbaig8/okf-generator/okf-generator:latest
docker run ghcr.io/umairbaig8/okf-generator/okf-generator --help

Pre-commit hook

Auto-regenerates the bundle when source files change — no stale knowledge graphs:

# .pre-commit-config.yaml
repos:
  - repo: local
    hooks:
      - id: okf-generate
        name: okf generate
        entry: okf generate
        language: system
        pass_filenames: false

CI/CD guide + copy-paste workflow template: docs/user-guide/ci-cd.md

---

CLI Reference

okf --help              Show available commands
okf <command> --help    Show options for a specific command
okf --version           Show version

Command	Usage	Description
generate	okf generate [src] [out] [--enrich]	Scan codebase (auto-detects project root)
update	okf update [src] [out] [--force] [--watch]	Incremental re-scan of changed files
lookup	okf lookup <query>	Instant exact-symbol concept retrieval
ask	okf ask <question>	AI-powered Q&A about your codebase
enrich	okf enrich <bundle> [--mode]	LLM-enhance an existing bundle
lsp	okf lsp [bundle]	LSP call-graph enrichment pass
diff	okf diff <old> <new> [--impact]	Compare two bundles, show dependency impact
visualize	okf visualize <bundle> [out.html]	Generate interactive D3 graph
mcp	okf mcp <bundle> [--port]	Start MCP server with 11 tools
dashboard	okf dashboard <bundle> [--port]	Launch FastAPI live bundle browser
serve	okf serve [dir] [--port]	Browse bundle via local HTTP server
pairs	okf pairs <bundle> [output.jsonl]	Export fine-tuning training pairs
install	okf install [agent]	Write agent integration rules/configs
init	okf init [dir]	Interactive wizard for bundle setup
summarize	okf summarize <bundle>	Regenerate SUMMARY.md for agents
domains	okf domains	List available domain classification rule sets
config	okf config [key=value]	Read/write .okfconfig settings
migrate	okf migrate <bundle>	Migrate v0.1 bundles to v0.2
agent	okf agent [--bundle]	Interactive REPL over the bundle
plugin	okf plugin [list|install|uninstall]	Manage external parser plugins

Full CLI reference: docs/cli-reference.md

---

FAQ

Does this require an API key or internet connection? No. Core extraction (okf generate) is fully offline and deterministic — no LLM call is made unless you explicitly enable --enrich. All 18 language parsers use tree-sitter (or stdlib equivalent) and work completely air-gapped.

How is this different from RAG / vector search? RAG retrieves chunks by semantic similarity, which is approximate and can miss exact symbols. okf lookup is exact: it indexes real functions, classes, modules, and dependencies by name and resolves to the precise concept, with zero embedding/vector infrastructure required. Same result every run.

Does my proprietary code leave my local environment? Never. Core extraction runs 100% offline on your local CPU. No code or metadata is sent to any third-party cloud. Optional LLM enrichment can be enabled manually and works with self-hosted models or private enterprise APIs.

Does this work on monorepos or very large codebases? Yes — scanning is linear in file count. Scope okf generate to a subdirectory if you only need part of the codebase indexed. Multi-bundle support visualizes cross-project edges.

Can I use this without any LLM at all, ever? Yes. okf generate + okf lookup form a complete zero-LLM workflow. LLM enrichment and training-pair synthesis are optional layers on top — you can get full value without ever providing an API key.

Is the bundle safe to commit to git? Yes — that's the intended workflow. Bundles are plain Markdown with YAML frontmatter — they diff cleanly, version alongside the code they describe, and can be reviewed in any code review tool.

---

Contributing

git clone https://github.com/UmairBaig8/okf-generator
cd okf-generator
pip install -e ".[dev]"
pytest tests/ -q

415+ tests passing. See CONTRIBUTING.md for full guidelines.

---

License

MIT — Copyright © 2026 Umair Baig

okf-generator produces OKF v0.2 bundles — an extended dialect of the Open Knowledge Format (OKF) v0.1 by Google. Our v0.2 adds schema version, concept identity, language, status, and typed relationships to every concept file. Not affiliated with Google.