ai-knowledge-filler 1.0.1

AI-powered content production system.

AI Knowledge Filler

AI-powered content production system — generate, enrich, search, validate, and visualize structured Markdown knowledge.

---

The Problem

Every time you use an LLM to generate structured knowledge files, the output drifts — wrong enum values, missing fields, dates in the wrong format, tags as strings instead of arrays. The files look fine until something downstream breaks: a search query returning nothing, a CI check failing, a pipeline corrupting.

The standard fix is post-hoc validation — check after writing, fix manually. That doesn't scale past a few dozen files.

---

How It Works

Prompt → LLM → Validation Engine → Error Normalizer → Retry Controller → Commit Gate → File

The LLM is the only non-deterministic component. Everything else is pure functions.

If output fails schema checks, it never touches disk — the Error Normalizer converts typed error codes into correction instructions and sends them back to the LLM for a retry.

If the same error fires twice on the same field, the pipeline aborts instead of looping — that pattern means your schema has a boundary problem, not the model.

---

Quick Start

pip install ai-knowledge-filler

export ANTHROPIC_API_KEY="sk-ant-..."  # or GOOGLE_API_KEY, OPENAI_API_KEY, GROQ_API_KEY

akf generate "Write a guide on Docker networking"
akf validate ./vault/

Works with Claude, GPT-4, Gemini, Ollama.

Quickstart by Interface

CLI:

akf init
akf generate "Create a guide on API rate limiting"
akf validate --path ./docs

Python API:

from akf import Pipeline

pipeline = Pipeline(output="./output")
result = pipeline.generate("Create a guide on API rate limiting")
print(result.success, result.file_path)

REST API:

akf serve --port 8000
curl -X POST http://127.0.0.1:8000/v1/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt":"Create a guide on API rate limiting"}'

Minimal runnable examples are in examples/cli_quickstart.sh, examples/python_api_quickstart.py, and examples/rest_api_quickstart.sh.

---

External Taxonomy Config

Your ontology lives in akf.yaml — not compiled into the tool:

# akf.yaml
schema_version: "1.0.0"
vault_path: "./vault"

enums:
  type: [concept, guide, reference, checklist, project, roadmap, template, audit]
  level: [beginner, intermediate, advanced]
  status: [draft, active, completed, archived]
  domain:
    - ai-system
    - api-design
    - devops
    - security
    - system-design

Change your taxonomy without touching code or redeploying:

akf init          # generates akf.yaml for your vault
akf validate ./   # validates all files against your config

---

Error Codes

Validation failures produce typed error codes, not free-form messages:

Code	Field	Meaning
E001	type / level / status	Value not in allowed enum set
E002	any	Required field missing
E003	created / updated	Date not ISO 8601
E004	title / tags	Type mismatch (e.g. tags: "security" instead of tags: [security])
E005	frontmatter	General schema violation
E006	domain	Value not in taxonomy
E007	created / updated	created is later than updated
E008	related	Typed relationship label not in relationship_types

The Error Normalizer translates these codes into deterministic correction instructions for the retry:

E006 on field "domain" (received: "backend")
→ "The 'domain' field must be one of: [api-design, backend-engineering, devops, ...]
   You used 'backend' which is not in the taxonomy. Choose the closest match."

---

Retry as Signal

Retry pressure is not a failure metric.

When a domain value triggers elevated retries, the taxonomy has a boundary problem — not the model. The telemetry substrate (append-only JSONL) surfaces which enum values cause friction, so you can refine your ontology based on evidence rather than intuition.

---

Interfaces

CLI:

akf generate "Create a guide on API rate limiting"
akf ask "How do I implement API rate limiting in FastAPI?" --top-k 5
akf ask "How do I implement API rate limiting in FastAPI?" --top-k 5 --no-llm
akf generate "Create Docker security checklist" --model gemini
akf validate ./vault/
akf validate --file outputs/Guide.md
akf serve --port 8000        # REST API
akf serve --mcp              # MCP server (v0.6.x)

Python API:

from akf import Pipeline

pipeline = Pipeline(output="./vault/")
result = pipeline.generate("Create a guide on Docker networking")
results = pipeline.batch_generate(["Guide 1", "Guide 2", "Guide 3"])

REST API:

POST /v1/generate    →  validated file
POST /v1/ask         →  RAG answer (or retrieval-only with no_llm)
POST /v1/validate    →  schema check result
POST /v1/batch       →  multiple files
GET  /v1/models      →  available providers

MCP (v0.6.x, in progress):

akf serve --mcp
# Exposes: akf_generate, akf_validate, akf_enrich, akf_batch

---

RAG Copilot (Phase 1: Indexer)

Phase 1 adds local corpus indexing for semantic search preparation.

Current scope:

• Parse Markdown files from corpus/ with python-frontmatter
• Split content by H2 headers using MarkdownHeaderTextSplitter
• Generate embeddings with sentence-transformers/all-MiniLM-L6-v2
• Store vectors in local Chroma collection akf_corpus

Out of scope (planned for later phases):

• Retriever/query layer
• Dedicated CLI commands
• Claude API integration for Q&A

Install dependencies:

pip install -e .[rag]

Run indexer:

python rag/indexer.py

Optional environment variables:

export RAG_CORPUS_DIR="corpus"
export RAG_CHROMA_PATH="rag/.chroma"
export RAG_COLLECTION_NAME="akf_corpus"
export RAG_EMBEDDING_MODEL="sentence-transformers/all-MiniLM-L6-v2"
export RAG_MARKDOWN_GLOB="*.md"
export RAG_BATCH_SIZE="64"

Expected output format:

Indexed files=<N>, chunks=<M>, collection_count=<K>

Phase 2: Retriever (local semantic search)

After indexing, query the local vector store:

python rag/retriever.py "How do I implement API rate limiting?" --top-k 5

Programmatic usage:

from rag.retriever import retrieve

result = retrieve("How do I implement API rate limiting?", top_k=5)
for hit in result.hits:
  print(hit.distance, hit.metadata.get("source"), hit.metadata.get("section"))

Current scope for Phase 2:

• Retrieval/query layer over Chroma index
• Returns top-k relevant chunks with metadata and distance

Still out of scope:

• Answer synthesis over retrieved chunks
• Dedicated AKF CLI subcommands for RAG
• Claude API integration for final response generation

Phase 3: Copilot synthesis (retrieve + answer)

Generate an answer grounded in retrieved chunks:

python rag/copilot.py "How do I implement API rate limiting in FastAPI?" --top-k 5 --model auto

Programmatic usage:

from rag.copilot import answer_question

result = answer_question(
  "How do I implement API rate limiting in FastAPI?",
  top_k=5,
  model="auto",
)
print(result.answer)
print(result.sources)

Phase 3 scope:

• Retrieval + synthesis flow
• Grounded answer generated from top-k chunks
• Source list returned with the answer

---

What Every Committed File Guarantees

• Required fields present: title, type, domain, level, status, tags, created, updated
• Valid enums: type, level, status from controlled sets
• Domain from your configured taxonomy in akf.yaml
• ISO 8601 dates with created ≤ updated
• tags as array with ≥ 3 items, title as string — no type mismatches

No file reaches disk without passing all checks.

---

Example Output

Input:

Create a guide on API rate limiting

Output (vault/API_Rate_Limiting_Strategy.md):

---
title: "API Rate Limiting Strategy"
type: guide
domain: api-design
level: intermediate
status: active
version: v1.0
tags: [api, rate-limiting, performance, architecture]
related:
  - "[[API Design Principles]]"
  - "[[System Scalability Patterns]]"
created: 2026-03-06
updated: 2026-03-06
---

## Purpose
...

---

Architecture

akf/
  pipeline.py          # Pipeline — generate(), validate(), batch_generate()
  validator.py         # Validation Engine — binary VALID/INVALID, E001–E007
  validation_error.py  # ValidationError dataclass
  error_normalizer.py  # Translates errors → LLM retry instructions
  retry_controller.py  # Convergence protection — aborts on identical error hash
  commit_gate.py       # Atomic write — only VALID files reach disk
  telemetry.py         # Append-only JSONL event stream
  config.py            # Loads akf.yaml or bundled defaults
  server.py            # FastAPI REST API
  mcp_server.py        # MCP server (FastMCP)
  market_pipeline.py   # Three-stage market analysis pipeline
  defaults/
    akf.yaml           # Default taxonomy

cli.py                 # Entry point
llm_providers.py       # Claude / Gemini / GPT-4 / Ollama

---

Model Support

Provider	Key	Notes
Claude	ANTHROPIC_API_KEY	Recommended for complex content
Gemini	GOOGLE_API_KEY	Fast, cost-effective
GPT-4	OPENAI_API_KEY	General purpose
Groq	GROQ_API_KEY	Free tier, fast
Ollama	—	Local, offline, private

---

Tests

pytest --cov=akf --cov-report=term-missing -v

560+ tests, 91% coverage, CI green on Python 3.10 / 3.11 / 3.12.

---

Tooling Policy

• Source of truth for lint/format rules: pyproject.toml
• Primary local quality tools: ruff, black, mypy
• Legacy configs .flake8, .pylintrc, .pydocstyle are removed to avoid conflicting rules
• Codecov policy (stabilization): upload on main pushes only, non-blocking
• Node 24 GitHub Actions compatibility: self-hosted runners must be >=2.327.1

Recommended local checks:

ruff check .
black --check .
mypy cli.py llm_providers.py exceptions.py logger.py akf/ --ignore-missing-imports

---

Installation

# PyPI
pip install ai-knowledge-filler

# With MCP support
pip install ai-knowledge-filler[mcp]

# From source
git clone https://github.com/petro-nazarenko/ai-knowledge-filler.git
cd ai-knowledge-filler
pip install -e .

Installation policy:

• Canonical dependency declaration path: pyproject.toml
• requirements.txt is a thin compatibility entrypoint used to install from the locked constraints in requirements.lock
• CI and release jobs install with pip install -c requirements.lock -e ".[all,dev]"

---

Documentation

• Architecture — module map, data flow, pipeline decisions
• CLI Reference — all commands, flags, exit codes
• User Guide — installation, configuration, troubleshooting
• REST API Threat Model — auth, endpoint exposure, limits, logging/PII
• Contributing — dev setup, quality gates, adding providers

---

License

MIT — free for commercial and personal use.

---

PyPI: https://pypi.org/project/ai-knowledge-filler Version: 1.0.0