okfsmith

Forge messy documents into OKF v0.2 knowledge bundles.

okfsmith is a Python CLI that ingests PDFs, markdown, wiki dumps, and Notion exports and emits spec-conformant OKF v0.2 bundles: markdown files with YAML frontmatter, plus reserved index.md and log.md. Every concept carries provenance (sources[]), a trust tier, and lifecycle metadata, and every bundle is checked by a built-in §11 validator. Serve it to your agents with one MCP command.

What's new

• okfsmith sync — incremental sync with SHA-256 change detection: only new, changed, renamed, or deleted sources are processed. okfsmith sync ./kb docs/ --no-llm
• Temporal knowledge model — validity windows (valid_from / valid_until), supersession chains, and --as-of queries so stale knowledge stays buried but never deleted. okfsmith search ./kb "refund policy" --as-of 2025-06-01
• MCP expansion — new traverse, provenance, and diff tools, plus evidence budgets (max_chunks, max_tokens, continuation_token) on every tool for bounded agent context. okfsmith mcp ./kb
• okfsmith eval — golden Q&A eval harness scoring the RAG Triad (context relevancy / faithfulness / answer relevancy), with retrieval-vs-generation diagnosis and --fail-under CI gating. okfsmith eval ./kb --init-sample
• Governed MCP write-back — preview_write_concept, write_concept, update_concept, and audit_log tools let agents contribute under code-enforced governance: always unverified tier, human-reviewed content needs explicit downgrade_trust, validator-gated, atomic, append-only audited.

60-second quickstart

pip install okfsmith

okfsmith init ./kb
okfsmith ingest ./kb notes/ report.pdf --no-llm   # offline first pass
okfsmith validate ./kb
okfsmith graph ./kb --format html   # writes ./kb/viz.html — open in a browser

Point any MCP-capable agent at the bundle:

pip install "okfsmith[mcp]"
okfsmith mcp ./kb

Tools exposed: index, list, search, get, neighbors, traverse, provenance, diff (read-only) plus governed write-back (preview_write_concept, write_concept, update_concept, audit_log) — every answer traces back to sources[] in the bundle.

Why okfsmith

Clean markdown already has good OKF tooling. The messy middle — 200-page PDFs with broken reading order, sprawling Notion exports, wiki dumps — is where knowledge usually lives, and that's what okfsmith ingests first:

parse → section → extract → link → emit → validate → serve

• Tiered parsing, free first. Tier 1 is local and keyless: LiteParse for PDFs, stdlib parsing for markdown/text. Office formats (.docx, .pptx, .xlsx) via the optional okfsmith[office] extra; Docling OCR sidecar via okfsmith[ocr] for scanned pages. PyMuPDF is deliberately avoided (AGPL).
• Two extraction modes. --no-llm writes one draft concept per section — fast, offline, deterministic. With an LLM (Ollama by default, or an OpenAI-compatible API via env vars), a draft + critic flow extracts claims with [^source-id] citations feeding sources[].
• OKF v0.2 native. Only type is required in frontmatter. okfsmith also emits sources[]/provenance, generated/verified trust metadata, and lifecycle fields.
• §11 validator. The spec's hard conformance rules (E001–E004) plus advisory lints (dead links, orphans, stubs, legacy v0.1 fields). Broken links are warnings, never errors (spec §6).
• Graph visualization. okfsmith graph --format html renders a self-contained viz.html (works offline): nodes colored by type with a colorblind-safe palette, shaped by trust tier, with backlinks, search, and keyboard access.
• One-command MCP server. okfsmith mcp ./kb over stdio, SSE, or streamable HTTP.
• Skill pack. skills/okfsmith-build/SKILL.md teaches agents the init → ingest → validate → serve loop.

CLI reference

The bundle is always the first positional argument. Expected failures print error [CODE]: with a hint and never a traceback; usage errors exit 2.

Full reference with examples: docs/commands.md.

Interactive chat

okfsmith chat opens a Claude Code / Gemini CLI style REPL over your bundle: ask questions in plain language, get answers with [concept-id] citations. With local Ollama running (or OPENAI_API_KEY set) answers are synthesized and grounded; otherwise the chat stays useful in extractive mode, showing the keyword-matched concepts themselves. It never answers from thin air — no relevant concepts, no invented answer.

$ okfsmith chat ./kb
 ███  █   █ █████  ████ █   █ █████ █████ █   █
█   █ █  █  █     █     ██ ██   █     █   █   █
█   █ ███   ████   ███  █ █ █   █     █   █████
█   █ █  █  █         █ █   █   █     █   █   █
 ███  █   █ █     ████  █   █ █████   █   █   █
okfsmith chat v0.3.0
Bundle: kb (24 concepts) · openai · qwen3:8b

Tips for getting started:
  1. Ask questions about your documents.
  2. Type /help for chat commands.
  3. Type /ingest <path> to add more documents.

kb › how do I authenticate?
✦
Use a Bearer token in the Authorization header [api/auth].

*Sources: [api/auth]*
kb › aur iska source kya hai
✦
The dashboard, under Settings › API Keys [api/auth].

*Sources: [api/auth]*
kb › /exit
Goodbye — your bundle is untouched.

(On a real terminal the logo renders as a yellow→orange→magenta gradient, the prompt bundle name is colored, and answers carry a subtle ✦ marker. Piped output stays plain ASCII — zero escape codes, always.)

Slash commands: /help /ingest /list /read /search /validate /graph /doctor /model /clear /exit. Line history persists at ~/.okfsmith/history; Ctrl-C cancels input, Ctrl-D quits. Flags: --model to pick the model, --no-llm to force extractive mode.

Use any model (API key)

Ollama is the default, but any OpenAI-compatible model works — one key, any provider. OpenRouter is the flagship: a single key routes to hundreds of models (anthropic/claude-sonnet-4-style IDs included):

export OKFSMITH_API_KEY="sk-or-..."        # your OpenRouter key
okfsmith chat ./kb --provider openrouter --model anthropic/claude-sonnet-4
okfsmith ingest ./kb docs/ --provider openrouter --model openai/gpt-4o-mini

Prefer env vars — the flags also work:

export OKFSMITH_API_KEY="..."              # Groq example
export OKFSMITH_PROVIDER=groq
okfsmith chat ./kb --model llama-3.3-70b-versatile

okfsmith chat ./kb --provider deepseek --model deepseek-chat
okfsmith chat ./kb --provider gemini --model gemini-2.0-flash

export AGENTROUTER_API_KEY="..."       # Agent Router gateway example
okfsmith chat ./kb --provider agentrouter --model gpt-4o-mini

Provider presets (15): openrouter · groq · mistral · deepseek · together · fireworks · deepinfra · anyscale · perplexity · xai · gemini · openai · agentrouter · lmstudio · ollama.

Literally anything else — Azure OpenAI, self-hosted vLLM, a llama.cpp server, any compat proxy — works via --api-base:

okfsmith chat ./kb --api-base https://my-proxy/v1 --model my-model

Notes:

• Put the key in OKFSMITH_API_KEY (AGENTROUTER_API_KEY is honored for the agentrouter preset). --api-key also works but lands in your shell history — okfsmith warns you once per session about that.
• Keys are never logged, never shown (banners and okfsmith doctor only say set (hidden)), and never written to disk.
• Anthropic's native API is not OpenAI-compatible, so it can't be called directly — use the openrouter preset (routes to Claude with one key) or point --api-base at an OpenAI-compatible gateway in front of Anthropic.
• okfsmith doctor shows the resolved provider, base URL, model, and key status without probing the network.

Install options

pip install "okfsmith[office]"   # DOCX / PPTX / XLSX
pip install "okfsmith[mcp]"      # MCP server
pip install "okfsmith[ocr]"      # Docling OCR sidecar
pipx install "okfsmith[office,mcp]"

okfsmith doctor verifies your setup.

Docs

📚 Live documentation website: https://bilal-junaid-jiwani.github.io/okfsmith/

Documentation index · Quickstart · Pipeline · Searching · Validation · Temporal model · MCP · Troubleshooting · FAQ

License

Apache-2.0. See LICENSE.