khiip 0.2.0

Capture, store, and recall substrate for LLM agents. The layer, not the destination.

Khiip

Open-source substrate for capturing, structuring, and preserving online sources — local-first, no phone-home, plain Markdown you own. For your tools, your agents, your archives.

Khiip (pronounced /kiːp/, like "keep") — short for Khipu, the Inca knot-record system that was the world's first typed-edge knowledge graph.

---

What Khiip is

A capture + storage + recall substrate. You point Khiip at a URL, PDF, X thread, YouTube video, or other online content — Khiip captures it fully, stores it permanently in your own filesystem, and lets you (or your AI agents) recall it later by meaning, structure, or time.

Khiip is the layer, not the destination. Other tools (Obsidian, Logseq, LangChain agents, custom scripts) sit on top and consume what Khiip captures.

Verb usage: to khiip — "I want to khiip this article" / "khiip that thread for later"

Plural: khiips — the captured artifacts in your vault

---

What Khiip does

Capability	What it means
Multi-platform capture	The free AGPL core captures the floor of each source; Khiip Plus (paid) re-derives the deeper rungs from the same kept bytes. X (core: tweet text + author + base engagement (likes/RT/replies) + community notes + polls + reply context + X-Article title; Plus adds full QRT chains, X-Article bodies, embedded media, and the views/bookmarks/quotes engagement set), Reddit (core: post + all top-level comments + crosspost lineage + R-F1 removed-status preservation; credential-free via old.reddit HTML by default, optional OAuth upgrade for rate headroom; Plus adds the full recursive comment tree incl. deep "load more" branches + full-res galleries), Wikipedia (MediaWiki action API → REST summary fallback chain; article content + best-effort references/infobox), generic web articles (core: trafilatura + readability fallback chain + OG/JSON-LD enrichment, body text; Plus adds hero + inline figures + media-woven body), YouTube (core: metadata + flat transcript via yt-dlp + youtube-transcript-api + API v3 fallback chain; Plus adds timestamped transcript + chapter navigation + thumbnail), PDF (markitdown + pdfplumber fallback chain). Instagram, TikTok, Threads, Bluesky on roadmap
Typed structured payloads	All 6 sources emit Pydantic-typed payloads at the extractor boundary: TweetPayload + RedditPayload + WikiPayload + WebPayload + PDFPayload + YouTubePayload. Discriminated union on kind; cross-platform primitives (EngagementCounts / UrlEntity / Media / CommentNode); X-specific XArticle + XArticleBlock (block-structured)
Local-first storage with portable Source-tier	Vault stores rendered Markdown + YAML frontmatter as the canonical Payload tier. Source-tier raw bytes (insurance against upstream rot) live under user-configurable [storage] data_root — defaults to ~/.local/share/khiip/; point at iCloud / Dropbox / external SSD / network mount for cross-machine sync. khiipd validate checks vault ↔ SQLite consistency end-to-end
Bitemporal model	Every capture records both recorded_at (when Khiip fetched) and valid_from (when data was true in the world). Point-in-time queries answer "what did this article say on date X"
P-δ failure handling	Capture-what-we-can: every sub-system reports its own status (extraction / media-fetch / wayback) independently; partial-success captures still land; failed-permanent tombstones retained in recall index. Three-layer visual indicator system: frontmatter canonical + Obsidian-native callouts in vault body + plugin UI badges (v0.5+)
Granular refetch CLI	khiipd refetch <id> [--re-extract|--re-render|--media|--wayback]. Five independently-attemptable dimensions: extraction (network re-fetch → new superseding capture), re-extract (re-derive the payload from preserved bytes — offline, in place), re-render (re-render the body from the typed payload — offline, in place), media (re-walk MediaFetcherRegistry in place), wayback (re-submit Wayback Save Page Now in place)
Per-consumer rendering	Renderer Protocol+Registry — 9 renderers (6 per-source MarkdownRenderers + LegacyMarkdownPassthroughRenderer + JSONRenderer + VaultFrontmatterRenderer). REST ?format= selector returns capture JSON / payload JSON / vault markdown / legacy markdown
Typed knowledge graph	5 first-class edge types (SUPPORTS, CONTRADICTS, SUPERSEDES, ELABORATES, REFERENCES) + RELATES escape for novel relationships. Hybrid model gives you deterministic queries AND novel-relationship coverage
Tiered AI recall (roadmap-aware)	At v0.1.x: bundled MiniLM-L6 ONNX (~80MB; one-time fetch) does the work; cosine top-k over per-source typed-payload embed-text composition. Embedder Protocol is built for swap-in: local LLM (Ollama) + BYOK (OpenAI / Anthropic / Gemini) + BM25 keyword fallback are planned for v0.5+. Today: works fully out of box without any AI; quality is whatever MiniLM gives you
Wayback witness (opt-in)	Off by default — archive.org's anonymous Save-Page-Now is rate-limited and unreliable, so Khiip doesn't promise it. Opt in with [archive] wayback_enabled = true for best-effort anonymous archiving; the archive_urls dict in frontmatter records success / failure / not-attempted, and failures are quiet (no callout). Reliable archiving needs your own archive.org credentials — a BYO-credentials tier is planned
Substrate-tier architecture	Khiip = one canonical substrate + N surfaces (Obsidian plugin / REST + MCP / Chrome ext / mobile / Telegram bot / LLM-agent SDK consumers). Destination-agnostic positioning
Open + portable	AGPL-3.0 daemon + Apache 2.0 SDK (separate repo, when published). No vendor lock-in. Filesystem-canonical. Export in Markdown, YAML, or JSON at any time

---

Strategic positioning

Khiip is the LLM default ingestion layer — the substrate AI agents use to capture, store, and recall data. It's the layer, not the destination: one canonical daemon with multiple surfaces over a single contract (Obsidian plugin, REST, MCP, and more to come). Whether you're a power-user self-hosting through the plugin, an AI builder wiring the REST + MCP surfaces into agent workflows, or piping captures into your existing PKM, you're consuming the same typed substrate underneath.

---

Current status

The current release is v0.2.0 — see CHANGELOG.md for per-release notes (MCP server, dedicated Wikipedia extractor, credential-free Reddit, deep comment trees + gallery fidelity, Wayback opt-in, and more). Pre-public-launch: ToS / Privacy Policy / social handles still in flight; the substrate is feature-complete for v0.1.x.

• ✅ All 6 v0.1 source extractors emit typed payloads (X / Reddit / Web / Wikipedia / YouTube / PDF)
• ✅ Full Renderer Protocol+Registry tier (9 renderer impls)
• ✅ MediaFetcherRegistry (HttpxFetcher + optionally YtDlpFetcher + GalleryDlFetcher)
• ✅ Wayback Machine archive submission (opt-in; off by default — anonymous SPN is rate-limited)
• ✅ Source-tier raw-bytes preservation (option (ii) URI-shape; user-configurable data_root)
• ✅ Typed-payload embed-text composition
• ✅ Visual indicator system layers 1 + 2 (layer 3 plugin badges = v0.5+)
• ✅ REST ?format= selector + POST /api/v1/captures/{id}/refetch endpoint
• ✅ khiipd validate + khiipd refetch CLIs
• ✅ Opt-in video preservation toggle ([media] download_videos)
• ✅ Plugin types.ts mirrors all v0.1 Pydantic models additively
• ✅ MCP server (v0.1.1) — six tools (capture_url, recall, list_captures, get_capture, refetch_capture, daemon_status) over stdio for Claude Desktop / Cursor / any MCP-aware client
• ⏳ Public launch (ToS + USPTO TESS + social handles + Show HN + community-channel announcements)
• ⏳ Plugin UI rendering of new typed-payload + visual indicator + refetch surfaces (v0.5+)
• ⏳ Cloud Video Archive (v0.5+; managed Source-tier + Whisper transcription)

Tests: a hermetic pytest suite, green on every release — see CHANGELOG.md for per-release test deltas. Each version's feature work landed additively on the v0.1.0 substrate (new subpackages / new source channels reusing the existing typed-payload pipeline), not by rewriting it — the foundation-vs-on-top discipline that held 17-for-17 through v0.1.0.

---

Install

On PyPI at public launch. The one-command install below is the launch path; until Khiip is published, use Install from source (just below).

Khiip is a self-contained Python tool — the khiipd daemon + CLI plus a khiip-mcp-server — that installs onto your PATH with no virtualenv to manage (Python 3.10+):

uv tool install khiip      # recommended — https://docs.astral.sh/uv/
# or
pipx install khiip         # https://pipx.pypa.io/

Both expose the khiipd and khiip-mcp-server commands globally. Upgrade later with uv tool upgrade khiip / pipx upgrade khiip.

Install from source — for development or contributing

git clone https://github.com/KhiipAI/khiip.git ~/projects/khiip
cd ~/projects/khiip
uv pip install -e ".[dev]"   # or: pip install -e ".[dev]"

make test runs the hermetic pytest suite; make smoke runs a live recall round-trip against an isolated sandbox.

Quickstart (60-second demo)

Capture an X tweet, recall it by meaning — every command below uses the installed khiipd:

# Start the daemon (first run downloads MiniLM-L6 ONNX ~80MB)
khiipd serve &

# Capture across multiple sources — different extractors, same vault
khiipd capture https://x.com/karpathy/status/2040470801506541998 # X: Karpathy's "LLM knowledge base" idea-file
khiipd capture https://simonwillison.net/2025/Dec/31/the-year-in-llms/ # web: Simon Willison — the year in LLMs
khiipd capture https://en.wikipedia.org/wiki/Quipu # wiki: the Inca knot-record system Khiip is named for
khiipd capture https://www.youtube.com/watch?v=-TAowrw97-4 # YouTube: Tiago Forte — the AI Second Brain

# Recall by meaning, not by keyword
khiipd recall "Andrej Karpathy's LLM knowledge wiki"
khiipd recall "building a second brain with AI"
khiipd recall "Inca knot record-keeping system"

# Check vault ↔ SQLite consistency (read-only)
khiipd validate

# Refetch a capture (extraction / --re-extract / --re-render / --media / --wayback)
khiipd refetch 01JX9...              # default = extraction: network re-fetch -> new superseding capture
khiipd refetch 01JX9... --re-extract # re-derive payload from preserved bytes (offline, in place)
khiipd refetch 01JX9... --re-render  # re-render body from typed payload (offline, in place)
khiipd refetch 01JX9... --media      # re-walk MediaFetcherRegistry in place
khiipd refetch 01JX9... --wayback    # re-submit Wayback in place

What you get:

• A SQLite index at ~/.local/share/khiip/index.db
• Markdown captures with YAML frontmatter at ~/khiip-vault/captures/x/ and ~/khiip-vault/captures/web/
• Semantic recall ranked by cosine similarity over bundled MiniLM-L6 embeddings — zero LLM cost, works offline after the first model fetch

What works today (v0.2.0):

• All 6 v0.1 source extractors (free AGPL core; Khiip Plus re-derives the deeper rungs noted below from the same kept bytes): X (core: tweet + author + base engagement + community notes; Plus: QRT chains + X-Article body + media + views/bookmarks/quotes), Reddit (core: post + all top-level comments + crosspost; credential-free via old.reddit HTML by default; optional OAuth app adds rate headroom; Plus: full recursive comment tree incl. deep "load more" branches + full-res galleries), Wikipedia (MediaWiki action API → REST summary fallback; article content + best-effort references/infobox), generic web (core: trafilatura primary + readability fallback + OG/JSON-LD enrichment, body text; Plus: hero + inline figures), YouTube (core: metadata + flat transcript via yt-dlp → oEmbed+transcript-api → API v3 fallback chain; Plus: timestamped transcript + chapters + thumbnail; operator-opt-in API v3 key widens chain), PDF (markitdown + pdfplumber fallback)
• Typed payloads end-to-end: extractor boundary → vault frontmatter (canonical) → SQLite (derived cache) → recall via per-source embed-text composition
• Media-fetcher chain: HttpxFetcher (photo) → optionally YtDlpFetcher (video; opt-in via [media] download_videos = true) → GalleryDlFetcher (wide-coverage fallback)
• Wayback witness (opt-in): off by default (anonymous SPN is rate-limited/unreliable); enable with [archive] wayback_enabled = true for best-effort archiving recorded in archive_urls
• Source-tier raw bytes: gzipped + preserved under user-configurable data_root (insurance against upstream rot)
• REST surface: POST /api/v1/captures, GET /api/v1/captures/{id}?format=..., POST /api/v1/captures/{id}/refetch?dimension=..., GET /api/v1/recall, GET /health, GET /api/v1/meta
• MCP server (v0.1.1): khiip-mcp-server console script exposes six tools over stdio for Claude Desktop / Cursor integration — see MCP server (Claude Desktop integration) below
• CLI: khiipd serve|capture|recall|validate|refetch|auth|version
• Obsidian plugin (v0.0.1 baseline): capture command + recall sidebar + types.ts mirrors all v0.1 payload models additively. UI rendering of typed-payload fields lands at v0.5+.

What doesn't yet: plugin UI rendering of typed-payload + visual indicator + refetch surfaces (v0.5+); Cloud Video Archive (v0.5+); Whisper transcription (v0.5+).

Heads up: khiipd serve listens on 127.0.0.1:8478 by default and reads/writes ~/.config/khiip/ + ~/.local/share/khiip/ + ~/khiip-vault/. To try Khiip in an isolated sandbox without touching those locations, use make smoke or set KHIIP_HOME=$(mktemp -d) khiipd serve. See the Development section below.

Per-source credentials (agnostic-connector model)

Khiip plumbs between your upstream accounts and your local vault — it never holds upstream credentials of its own. Most sources work with no setup (X via fxtwitter, generic web via trafilatura, Wikipedia, Reddit via old.reddit HTML). A few accept your own platform credentials so requests go through your identity + your rate-limit budget for higher fidelity:

Reddit (optional — capture works credential-free):

Reddit captures work out of the box via the credential-free old.reddit.com HTML channel (live threads + full comment trees, self-hosted on your own IP). Configuring your own Reddit app is an optional upgrade that adds cleaner deep-comment pagination + a 60 req/min headroom and is tried first when present:

1. Visit https://www.reddit.com/prefs/apps → "create another app..." → script type → fill name (e.g. "khiip"), redirect uri http://localhost:8478/ (unused; required by the form) → click "create app"

2. Copy the client ID (short string under the app name) + client secret

3. Either add to ~/.config/khiip/config.toml:

   [extractors.reddit]
   client_id = "your-reddit-client-id"
   client_secret = "your-reddit-client-secret"
   

   …or set env vars (KHIIP_REDDIT_CLIENT_ID + KHIIP_REDDIT_CLIENT_SECRET) before khiipd serve.

If a configured app is missing/expired, capture transparently falls back to the HTML channel rather than failing. The HTML channel is self-host-only by design (a datacenter IP would hit Reddit's WAF + robots.txt).

YouTube (optional; widens the fallback chain from 2 to 3 sources): set [extractors.youtube] api_key = "..." (Google Data API v3 key, free tier) for metadata resilience when yt-dlp + youtube-transcript-api both fail.

---

MCP server (Claude Desktop integration)

v0.1.1 ships a Model Context Protocol server (khiip-mcp-server) that exposes the substrate's six core operations as MCP tools. Claude Desktop, Cursor, or any other MCP-aware client can connect over stdio and capture/recall through Khiip without writing any HTTP code.

Tools at v0.1.1:

Tool	Maps to
capture_url(url, force_new=False)	POST /api/v1/captures
recall(query, limit=10)	GET /api/v1/recall
list_captures(source=None, limit=50, offset=0)	GET /api/v1/captures
get_capture(capture_id, format="json")	GET /api/v1/captures/{id}
refetch_capture(capture_id, dimension="extraction")	POST /api/v1/captures/{id}/refetch
daemon_status()	GET /health + GET /api/v1/meta

Architecture: the MCP server is a thin HTTP proxy that connects to your running Khiip daemon at 127.0.0.1:8478 (configurable via KHIIP_DAEMON_URL env var or [daemon] host/port in ~/.config/khiip/config.toml). It reads the Bearer auth token from ~/.config/khiip/auth.toml (the same file the CLI + Obsidian plugin use). The daemon must be running before MCP tool calls land.

Claude Desktop config (~/Library/Application Support/Claude/claude_desktop_config.json on macOS; analogous paths on other platforms):

{
  "mcpServers": {
    "khiip": {
      "command": "khiip-mcp-server"
    }
  }
}

If khiip-mcp-server isn't on Claude Desktop's PATH (common when you installed via pip install -e inside a venv), point Claude Desktop at the full path:

{
  "mcpServers": {
    "khiip": {
      "command": "/Users/you/projects/khiip/.venv/bin/khiip-mcp-server"
    }
  }
}

Verify the wiring:

# Start the daemon in one terminal
khiipd serve

# In another terminal: send an initialize ping to the MCP server
python -c "
import asyncio
from mcp.client.session import ClientSession
from mcp.client.stdio import stdio_client, StdioServerParameters

async def main():
    params = StdioServerParameters(command='khiip-mcp-server', args=[])
    async with stdio_client(params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()
            tools = await session.list_tools()
            for t in tools.tools:
                print(t.name)

asyncio.run(main())
"

You should see all six tool names listed.

Architectural note: the v0.1.1 MCP server proxies REST. The medium-term plan (recorded in internal design notes) is to refactor business logic into a shared service tier when a hosted/productized surface or a third transport (gRPC / WebSocket / etc.) arrives. The tool surface, signatures, and Claude Desktop config stay stable across that refactor.

---

Development — running tests and smokes

Default paths:

Purpose	Path	Override
Config	~/.config/khiip/	XDG_CONFIG_HOME (Unix convention)
Data (SQLite index)	~/.local/share/khiip/	XDG_DATA_HOME
Vault (Markdown captures)	~/khiip-vault/	configurable per source in config.toml

For dev/test smokes only, set KHIIP_HOME to redirect all three under one root:

make test            # hermetic pytest suite, ~1s
make smoke           # full live round-trip against fxtwitter + fastembed
                     # internally: KHIIP_HOME=$(mktemp -d) khiipd serve ...

Do not set KHIIP_HOME in your shell rc. macOS GUI applications (Obsidian, Claude Desktop, the MCP server when launched by Claude Desktop) do not inherit shell environment variables from launchd-launched processes — the daemon and your plugin would silently disagree on workspace. KHIIP_HOME is a per-invocation knob for tests and demos, not a user setting.

---

Get involved

Khiip is shipping at v0.2.0 (see CHANGELOG.md for the per-release trail). Public launch (ToS / Privacy Policy / social handles / Show HN / community-channel announcements) is the next horizon. Until then:

• ⭐ Star the repo to follow along
• 💬 Watch the repo for releases — the current release is v0.2.0 (see CHANGELOG.md for the per-release trail)
• 📧 Get on the launch list by emailing [e-mail] with subject "launch list"
• 🐛 Issues + PRs — see CONTRIBUTING.md for guidelines
• 🔒 Security disclosures — see SECURITY.md
• 🤝 Code of Conduct — Contributor Covenant 2.1

---

Why "Khiip"?

The Inca khipu (or quipu) was a knotted-cord recording system used from approximately 2500 BCE to the 17th century to encode numerical, narrative, and relational information. Each cord type, knot placement, and color encoded different categories of data. The khipukamayuq was the trained specialist who could read and create the records — essentially the human "agent" interacting with the structured substrate.

Khiip is the digital equivalent: a substrate that captures information from anywhere on the internet and stores it in a structured, typed, queryable form — for AI agents (the modern khipukamayuq) to read and create.

The "ii" spelling distinguishes from the homophonous "keep" while preserving the Khipu etymological root. Pronounced /kiːp/ — same as "keep."

---

License

Daemon, plugin, and substrate code: AGPL-3.0 (see LICENSE)

SDK code (separate repository when published): Apache 2.0

Khiip Plus (the paid enrichment + styled-render tier): not open source — licensed under the Elastic License 2.0. The open-source guarantee covers the AGPL core; Plus is the paid layer that funds it.

The canonical license posture for the daemon is AGPL-3.0. Any code released under AGPL-3.0 stays AGPL-3.0 for those releases — license retraction is not how copyleft works. The project surfaces its commitments through the LICENSE + README rather than a separate charter document.