Skip to main content

Hybrid Vector + Graph system for Obsidian vaults

Project description

๐Ÿง  Clawdiney

CI PyPI Python Version Coverage License: MIT

Expanded Brain for Coding Agents

A hybrid Vector + Graph system that transforms your Obsidian vaults into a living knowledge source for AI coding agents.

Clawdiney Banner


๐Ÿš€ Overview

Clawdiney is a multi-vault knowledge system. It indexes multiple Obsidian vaults โ€” one per project โ€” and provides semantic search + knowledge graph navigation for AI agents.

Core capabilities:

  • Zero-infrastructure storage (v0.2.0): Everything lives in a single local SQLite file (brain.db) โ€” vectors via sqlite-vec, full-text via FTS5, knowledge graph via relational tables. No Docker, no Neo4j, no ChromaDB, no Redis.
  • Hybrid Search: BM25 (exact terms, acronyms, identifiers) + semantic vectors, fused with Reciprocal Rank Fusion; optional cross-encoder reranking (pip install clawdiney[rerank])
  • Multi-Vault Architecture: Each project gets its own vault with isolated data (vault-scoped rows in the same database)
  • CWD Auto-Detection: The system detects which vault to use based on your current working directory โ€” no manual switching
  • Knowledge Graph: Maps relationships between notes via [[WikiLinks]] and shared tags
  • Linking & Fallback: Vaults can link to related vaults (e.g., SDK projects link to their parent project). Searches cascade through the chain: current vault โ†’ linked vaults โ†’ general
  • Native Integration: Connects to MCP-compatible agents (OpenCode, Claude Code, etc.) via SSE or stdio
  • Agent-Written Memory: write_memory turns conversational facts into provenance-marked, searchable notes โ€” resolved against existing entities, deduped, and optionally namespaced per agent (agent_id)
  • Bi-Temporal Facts & Conflict Detection: Graph facts carry validity windows; when an LLM-extracted fact genuinely changes value, both versions are kept and flagged rather than one silently overwriting the other
  • Measured, Not Assumed: clawdiney-eval scores retrieval quality (recall@k/MRR/hit-rate) per mode against a regression baseline โ€” see BENCHMARKS.md

Migrating from v0.1.x (Docker stack)? The Neo4j/ChromaDB/Redis containers are no longer used. Just install v0.2.0, keep Ollama running, and run clawdiney-index once to rebuild the index into brain.db. Your vault files are the source of truth โ€” nothing is lost. Old Docker volumes can be removed with docker compose -f docker/docker-compose.yml down -v.


โš–๏ธ Why Embedded SQLite Instead of a Service Stack

Peers in the AI-agent-memory space (mem0, Zep/Graphiti, Letta/MemGPT) typically run a vector DB + graph DB + cache as separate services. Clawdiney holds vectors (sqlite-vec), full-text (FTS5), and the knowledge graph in one brain.db file:

Clawdiney Typical peer stack
Infrastructure 1 file (brain.db) Vector DB + graph DB (e.g. Neo4j) + cache (e.g. Redis)
Setup pip install, point BRAIN_DB_PATH Provision + configure 2-3 services
Deploy Copy a file Orchestrate a stack (Docker Compose / k8s)
Backup Copy a file Backup each service independently

This is a deliberate trade-off: embedded SQLite means single-writer-at-a-time semantics and no built-in horizontal scaling โ€” right for a single-user or small-team coding-agent memory layer, not a multi-tenant SaaS backend. See BENCHMARKS.md for retrieval-quality numbers (recall@k/MRR/hit-rate by mode) and reranker latency/precision trade-offs, measured with the built-in eval harness (clawdiney-eval).


๐Ÿ“‹ Prerequisites

Before starting, make sure you have installed:

Software Minimum Version Link
Ollama 0.3.x+ ollama.com (embedding model bge-m3)
Python 3.10+ Usually already installed on Unix systems. If not: apt install python3 or brew install python@3.12

Optional:

Extra Install What it adds
Reranker pip install clawdiney[rerank] Cross-encoder reranking (BAAI/bge-reranker-v2-m3, ~2GB RAM). Without it, searches use RRF ordering โ€” still fully functional.

Configuration: BRAIN_DB_PATH env var sets the database location (default: ~/.clawdiney/brain.db). For the project knowledge graph: CARD_LLM_MODEL (Ollama model for project card Purpose/Architecture sections, default qwen3) and ENTITY_RESOLUTION_THRESHOLD (similarity cutoff for merging duplicate entities, default 0.85).

Cross-project intelligence: the project indexer builds a typed knowledge graph (dependencies, shared datastores, patterns) from your codebases. Agents can call get_project_card("name") for a project overview and how_do_projects_relate("a", "b") to see how two projects connect, with evidence.

Supported Systems:

  • โœ… Linux (Ubuntu, Debian, Fedora, Arch, etc.)
  • โœ… macOS (Intel and Apple Silicon)
  • โœ… WSL2 (Windows Subsystem for Linux)
  • โœ… BSD (FreeBSD, OpenBSD - with manual adjustments)

๐Ÿ› ๏ธ Quick Installation

Option A: pip install (recommended for using Clawdiney against your own vault)

pip install clawdiney
# optional: pip install clawdiney[rerank]

Create a .env in the directory you'll run commands from (or export the same variables in your shell) โ€” see .env.example for the full list:

VAULTS_DIR=~/clawdiney-vaults
MCP_DEFAULT_VAULT=general
BRAIN_DB_PATH=~/.clawdiney/brain.db
MODEL_NAME=bge-m3:latest

This installs three console scripts: clawdiney (vault management โ€” see Provisioning Project Vaults below), clawdiney-index (index a vault into brain.db), and clawdiney-mcp (the MCP server entry point, used in MCP Client Configuration). Ollama still needs to be running separately with the bge-m3 model pulled.

ollama pull bge-m3
clawdiney-index

Option B: clone from source (recommended for contributing or running the bootstrapper)

git clone git@github.com:elaranjo/clawdiney.git
cd clawdiney

Configure .env:

cp .env.example .env
nano .env

Run the Bootstrapper:

chmod +x scripts/setup_brain.sh
./scripts/setup_brain.sh

๐Ÿ“‹ What the Bootstrapper Does

The setup_brain.sh script automatically executes:

Step Action
๐Ÿ” Checks if Ollama is installed
๐Ÿ“ Creates .env with default settings (if it doesn't exist)
๐Ÿ Creates Python virtual environment (venv)
๐Ÿ“ฆ Installs Python dependencies (sqlite-vec, ollama, mcp, etc.)
๐Ÿฆ™ Downloads embedding model (bge-m3) via Ollama
๐Ÿง  Indexes your vault(s) into brain.db

๐Ÿ—๏ธ Multi-Vault Architecture

How Vaults Work

Clawdiney discovers vaults by scanning subdirectories in VAULTS_DIR (default: ~/clawdiney-vaults/). Each subdirectory must contain a clawdiney.toml config file.

clawdiney.toml format

Each vault requires a minimal config file:

id = "Payments"
name = "Payments"
description = "Payments service"
linked_vaults = ["general"]
Field Description
id Unique vault identifier (matched against directory names for CWD detection)
name Display name
description Optional description
linked_vaults Vault IDs for fallback search (e.g., a client SDK linked to the service it wraps)

CWD Auto-Detection (Convention > Configuration)

When you call any MCP tool without specifying vault=, Clawdiney inspects your current working directory. It walks the path backwards until it finds a directory name matching a vault id.

Your CWD Detected Vault
~/projetos/Payments/ Payments
~/projetos/MyCompanyApi/src/ MyCompanyApi
~/projetos/Payments-SDK/ Payments-SDK
/any/other/directory general (fallback)

Linking & Fallback Chain

Vaults can link to related vaults for broader search results:

Payments-SDK โ”€โ”€linked_toโ”€โ”€โ–บ [general, Payments]
                                  โ”‚
Auth-SDK     โ”€โ”€linked_toโ”€โ”€โ–บ [general, Auth]
                                  โ”‚
clawdiney    โ”€โ”€linked_toโ”€โ”€โ–บ []   (isolated โ€” no fallback)

When searching, Clawdiney queries: current vault โ†’ linked vaults (in order) โ†’ general. Results from all sources are merged and deduplicated.

Using Your Personal Vault as general

To make your personal Obsidian vault the fallback general vault, create a symlink:

# Create clawdiney.toml inside your personal vault
cat >> /path/to/ObsidianVault/clawdiney.toml << 'EOF'
id = "general"
name = "General"
description = "Personal Obsidian vault - general knowledge"
linked_vaults = []
EOF

# Create symlink
ln -sfn /path/to/ObsidianVault ~/clawdiney-vaults/general

# Reindex
OLLAMA_HOST= ./venv/bin/python3 -m clawdiney.indexer

Provisioning Project Vaults

For teams with multiple projects, use the provisioning script to scan a projects directory and create vaults automatically:

./scripts/provision_project_vaults.sh

This scans ~/projetos/ and creates a vault per project with:

  • Auto-generated clawdiney.toml (SDKs linked to parent, clawdiney isolated)
  • P.A.R.A. folder structure (00_Inbox, 10_Projects, 20_Areas, 30_Resources, 40_Archives, 50_Daily)
  • Project analysis files (README, Architecture, API docs, Domain model)

๐Ÿ”Œ MCP Client Configuration

Clawdiney runs as a local Python process (stdio transport) โ€” no server to start separately, no container. Your MCP client (Claude Code, OpenCode, etc.) launches it on demand.

The command/args differ depending on how you installed Clawdiney: a pip install exposes clawdiney-mcp directly on PATH; a clone + bootstrapper setup runs the module from its venv instead. Pick the block matching Option A or B above.

Claude Code (~/.claude.json, project scope):

{
  "projects": {
    "/home/YOUR_PROJECTS_DIR": {
      "mcpServers": {
        "clawdiney": {
          "command": "clawdiney-mcp",
          "env": {
            "VAULTS_DIR": "/path/to/your/vaults",
            "MCP_DEFAULT_VAULT": "general",
            "MODEL_NAME": "bge-m3:latest",
            "ENABLE_RERANK": "true"
          }
        }
      }
    }
  }
}

Cloned from source instead? Use "command": "/path/to/clawdiney/venv/bin/python3" with "args": ["-m", "clawdiney.mcp_server"].

OpenCode (opencode.json):

{
  "mcp": {
    "clawdiney": {
      "type": "local",
      "command": ["clawdiney-mcp"],
      "enabled": true,
      "environment": {
        "VAULTS_DIR": "/path/to/your/vaults",
        "MCP_DEFAULT_VAULT": "general",
        "MODEL_NAME": "bge-m3:latest",
        "ENABLE_RERANK": "true"
      }
    }
  }
}

Cloned from source instead? Use "command": ["/path/to/clawdiney/venv/bin/python3", "-m", "clawdiney.mcp_server"].

Restart the client session after registering โ€” MCP config is read once at session start.

Remote/network access (SSE transport) is still available by setting MCP_TRANSPORT=sse before launching clawdiney.mcp_server directly, but stdio (the default, no config needed) is what both clients above use and is the supported path.


๐Ÿ” Proactive Context Hook (Optional)

scripts/claude_hook_context.py is a Claude Code UserPromptSubmit hook: it runs search_brain-equivalent retrieval before every prompt and injects the results as context automatically, so relevant SOPs/patterns surface even if the agent doesn't think to search for them.

Install โ€” add to ~/.claude/settings.json under hooks.UserPromptSubmit:

{
  "hooks": [
    {
      "type": "command",
      "command": "/path/to/clawdiney/venv/bin/python3 /path/to/clawdiney/scripts/claude_hook_context.py",
      "timeout": 10,
      "statusMessage": "Querying clawdiney brain..."
    }
  ]
}

Controlling how many sources it injects (default: 3; 0 disables it entirely):

Scope How
Session export CLAWDINEY_HOOK_N_RESULTS=8 before starting Claude Code โ€” applies to every prompt in that shell
Single prompt Add @nN anywhere in the prompt text, e.g. explain the auth flow @n8 โ€” the marker is stripped before the search runs and only affects that one prompt
Disable CLAWDINEY_HOOK_N_RESULTS=0 (session) or @n0 (one prompt)

Precedence: inline @nN marker > CLAWDINEY_HOOK_N_RESULTS > default (3).


๐Ÿš€ Usage

Ensure Ollama Is Running

Clawdiney's only external dependency is Ollama (for embeddings, and card-generation LLM calls). Everything else โ€” vectors, full-text search, and the knowledge graph โ€” lives in a single embedded SQLite file (brain.db, default ~/.clawdiney/brain.db). No services to start or stop.

ollama serve   # if not already running as a daemon
ollama pull bge-m3

Index Your Vault(s)

./venv/bin/python3 -m clawdiney.indexer

Via MCP Client (Recommended)

With MCP configured, the agent has access to read and write tools:

Read Tools (Discovery)

  • search_brain(query, vault?, agent_id?, n_results?) - Hybrid search (BM25 + vector, RRF-fused, optionally reranked) for architectural patterns, SOPs, and design system components. agent_id (optional) also searches that agent's own memory (see write_memory below) alongside shared vault content; results append an "Unresolved conflicts" section when a returned note touches a contradicted fact. n_results (optional, default 3) sets how many results to retrieve โ€” raise it (6-8) for broad/exploratory questions spanning multiple docs, lower it (1-2) for a narrow lookup where you already know which note has the answer; n_results=0 explicitly skips the search instead of returning a real (possibly empty) result set
  • explore_graph(note_name, vault?, agent_id?) - Find entities related to a note or project โ€” notes, WikiLinks, tags, or (for projects) dependencies/patterns โ€” with relation type and evidence. Same agent_id scoping and conflict surfacing as search_brain
  • resolve_note(name) - Resolve ambiguous note names into canonical vault-relative paths
  • get_note_chunks(path) - Inspect indexed chunk headers for a resolved note
  • get_project_card(name) - Full project card (Purpose, Stack, Architecture, Interfaces) โ€” the first call when touching an unfamiliar project
  • how_do_projects_relate(a, b, vault?, agent_id="*") - Graph paths between two projects (shared dependencies, datastores, patterns) with evidence. agent_id="*" (default) applies no filtering; pass a specific id to additionally restrict to that agent's own relations
  • health_check() - Check health of brain.db, Ollama, and the optional reranker, plus per-vault document counts

Write Tools (Knowledge Capture)

  • write_note(path, content, mode) - Create or update a note at any vault location
  • write_memory(fact, source, agent_id="default", vault?) - Persist a natural-language fact (ideally "<Subject> <verb> <value>", e.g. "User prefers embedded SQLite over Docker-based stacks") as agent-written memory. Resolves the subject against existing entities and writes to a provenance-marked 40_Memory/ note โ€” the only write tool triggered by conversational knowledge rather than an explicit save request
  • append_to_daily(content) - Append content to today's daily note (50_Daily/YYYY-MM-DD.md)
  • add_learning(topic, content, area) - Save learnings to appropriate folder (SOPs, Architecture, etc.)
  • delete_note(path) - Delete a note and remove from index

Example Workflow:

"Check in the Brain if there are any SOPs for production deployment." โ†’ Found existing SOP, update it: write_note("30_Resources/SOPs/SOP_Deploy.md", updated_content, mode="overwrite")

"Search the brain for UI Design System patterns." โ†’ No pattern found, create new: add_learning("Button_Component", "# Design System\\n\\nButton patterns...", area="DesignSystem")

"Document today's learnings about the project." โ†’ append_to_daily("## Learnings\\n- Discovered X about the architecture")

"Remember that I prefer embedded SQLite over Docker-based stacks for this kind of tool." โ†’ write_memory("User prefers embedded SQLite over Docker-based stacks", source="conversation")

Full-file reading is intentionally outside the MCP workflow. The agent should use the repository or vault filesystem directly after search_brain has identified the relevant note.

Via Shell (Alternative)

If MCP is not available, use the direct script:

./scripts/ask_brain.sh "production deployment patterns"

Via Python (For developers)

./venv/bin/python3 -m clawdiney.query_engine "your query here"

๐Ÿงฉ Architecture

โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚                AI Agent (Claude Code, OpenCode, etc.)    โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                         โ”‚ MCP Protocol (stdio)
                         โ–ผ
โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚              Clawdiney MCP Server                        โ”‚
โ”‚  โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”  โ”‚
โ”‚  โ”‚      Hybrid Query Engine     โ”‚ โ”‚  Vault Detector  โ”‚  โ”‚
โ”‚  โ”‚  BM25 (FTS5) + Vector (KNN)  โ”‚ โ”‚  (CWD-based)     โ”‚  โ”‚
โ”‚  โ”‚  โ†’ RRF fusion โ†’ rerank       โ”‚ โ”‚  โ†’ vault_id      โ”‚  โ”‚
โ”‚  โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜  โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                    โ–ผ
โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚         brain.db (single embedded SQLite file)           โ”‚
โ”‚  documents ยท chunks ยท chunk_vectors (sqlite-vec)          โ”‚
โ”‚  chunk_fts (FTS5) ยท entities ยท relations (graph)          โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                    โ–ฒ
                    โ”‚ indexed from
โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚              VAULTS_DIR (~/clawdiney-vaults/)            โ”‚
โ”‚                                                         โ”‚
โ”‚  general/ (personal vault)   clawdiney/ (this project)  โ”‚
โ”‚  Each vault has: clawdiney.toml + P.A.R.A. structure    โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

Only external dependency: Ollama (embeddings + card-generation LLM calls). No Docker, no separate vector/graph/cache servers.

๐Ÿ“ Project Structure

clawdiney/
โ”œโ”€โ”€ src/clawdiney/             # Main Python package
โ”‚   โ”œโ”€โ”€ __init__.py            # Package exports
โ”‚   โ”œโ”€โ”€ config.py              # Multi-vault configuration, VAULTS_DIR discovery, BRAIN_DB_PATH
โ”‚   โ”œโ”€โ”€ vault_config.py        # VaultConfig dataclass + clawdiney.toml parser
โ”‚   โ”œโ”€โ”€ storage.py             # Embedded brain.db gateway (sqlite-vec + FTS5 + graph tables)
โ”‚   โ”œโ”€โ”€ indexer.py             # Full indexing into brain.db
โ”‚   โ”œโ”€โ”€ incremental_indexer.py # Incremental sync (content hashes stored in brain.db)
โ”‚   โ”œโ”€โ”€ query_engine.py        # Hybrid search (BM25 + vector, RRF fusion) + vault fallback chain
โ”‚   โ”œโ”€โ”€ reranker.py            # Cross-encoder reranking (optional extra, GPUโ†’CPU fallback)
โ”‚   โ”œโ”€โ”€ embedding_providers.py # EmbeddingProvider protocol (Ollama default, OpenAI optional)
โ”‚   โ”œโ”€โ”€ vault_writer.py        # Thread-safe write operations per vault
โ”‚   โ”œโ”€โ”€ mcp_server.py          # MCP server: CWD auto-detection, search/graph/write tools
โ”‚   โ”œโ”€โ”€ project_indexer.py     # Analyzes codebases โ†’ enriched project cards for Obsidian
โ”‚   โ”œโ”€โ”€ project_index_config.py# Selective file indexing patterns (include/exclude)
โ”‚   โ”œโ”€โ”€ entity_extractor.py    # Project knowledge graph: manifest parsing + LLM extraction
โ”‚   โ”œโ”€โ”€ chunking.py            # Text chunking strategies
โ”‚   โ”œโ”€โ”€ constants.py           # Application constants
โ”‚   โ”œโ”€โ”€ logging_config.py      # Logging setup
โ”‚   โ”œโ”€โ”€ cli.py                 # CLI entry point (vault create/list)
โ”‚   โ””โ”€โ”€ scripts/
โ”‚       โ”œโ”€โ”€ watch_vault.py     # File watcher for real-time vault sync
โ”‚       โ”œโ”€โ”€ sync_vault.py      # Manual sync script (per-vault aware)
โ”‚       โ”œโ”€โ”€ watch_projects.py  # File watcher for codebase โ†’ project card sync
โ”‚       โ””โ”€โ”€ index_projects.py  # CLI: index projects to Obsidian
โ”‚
โ”œโ”€โ”€ tests/                     # Test suite (pytest, no service mocks โ€” real tempfile SQLite)
โ”‚   โ”œโ”€โ”€ test_storage.py
โ”‚   โ”œโ”€โ”€ test_query_engine.py
โ”‚   โ”œโ”€โ”€ test_hybrid_search.py
โ”‚   โ”œโ”€โ”€ test_reranker.py
โ”‚   โ”œโ”€โ”€ test_entity_extractor.py
โ”‚   โ”œโ”€โ”€ test_mcp_server.py
โ”‚   โ””โ”€โ”€ ...
โ”‚
โ”œโ”€โ”€ scripts/                   # Shell scripts
โ”‚   โ”œโ”€โ”€ setup_brain.sh         # Bootstrap setup (venv, deps, pull embedding model, index)
โ”‚   โ”œโ”€โ”€ ask_brain.sh           # Query from command line
โ”‚   โ”œโ”€โ”€ run_tests.sh           # Run test suite
โ”‚   โ”œโ”€โ”€ claude_hook_context.py # Optional Claude Code UserPromptSubmit hook (proactive context)
โ”‚   โ”œโ”€โ”€ provision_project_vaults.sh
โ”‚   โ””โ”€โ”€ ...
โ”‚
โ”œโ”€โ”€ .env.example                # Environment template with multi-vault setup
โ”œโ”€โ”€ pyproject.toml               # Python project configuration (source of truth for dependencies)
โ””โ”€โ”€ README.md                    # This file

๐Ÿ”„ Updating Knowledge

Good news: Clawdiney now has automatic sync! No need to manually re-index.

Auto-Sync (Default)

The MCP server automatically checks for vault changes on startup and syncs any modified files.

Real-Time Watcher (Optional)

For active development, run the file watcher that syncs changes in real-time:

./venv/bin/python3 -m clawdiney.scripts.watch_vault

Manual Sync (On-Demand)

# Check sync status
./venv/bin/python3 -m clawdiney.scripts.sync_vault --status

# Incremental sync (only changed files)
./venv/bin/python3 -m clawdiney.scripts.sync_vault

# Full sync (reindex everything)
./venv/bin/python3 -m clawdiney.scripts.sync_vault --full

The agent has immediate access to new/modified notes after sync completes.


๐Ÿ›ก๏ธ Privacy and Security

  • Multi-Vault Isolation: Each project's data stays in its own directory under VAULTS_DIR. No vault reads another vault's .md files.
  • Symlink Support: You can symlink a vault directory, allowing you to keep your personal vault in one location while Clawdiney references it.
  • Local Data: Everything runs locally on your machine. Nothing is sent to the cloud (except if you use cloud models).
  • Single-file storage: All indexed data lives in one local SQLite file (brain.db), scoped by vault at the row level. Delete the file to wipe everything; copy it to back everything up.

๐Ÿ› Troubleshooting

Start with the MCP health_check() tool โ€” it reports brain.db status (path, document/chunk/entity/relation counts), Ollama connectivity, and whether the reranker loaded, plus per-vault document counts.

MCP client doesn't see the server

  • Check the client config points command/args at <venv>/bin/python3 -m clawdiney.mcp_server and restart the client session (MCP config is only read at session start).
  • Test the server manually: ./venv/bin/python3 -m clawdiney.mcp_server (should print Ollama model validation, then wait on stdio โ€” Ctrl+C to stop).

search_brain returns nothing

  • Run health_check() โ€” if documents: 0, the vault hasn't been indexed yet: ./venv/bin/python3 -m clawdiney.indexer.
  • Check VAULTS_DIR / VAULT_PATH env vars match where your .md files actually live.

"Re-index required" / SchemaMismatchError

  • brain.db was created with a different embedding model or dimension than your current config. Delete the file (rm ~/.clawdiney/brain.db, or your BRAIN_DB_PATH) and re-run the indexer โ€” it rebuilds from your vault files, nothing else is lost.

Reranker doesn't activate

  • Check it's installed: pip install clawdiney[rerank] (adds sentence-transformers, ~2GB with model weights).
  • On small/shared GPUs, the reranker automatically retries on CPU if CUDA runs out of memory โ€” check the logs for "retrying on CPU". If it still fails, set ENABLE_RERANK=false to skip it entirely; search still works via BM25+vector RRF fusion.

Reranker configuration

RERANK_MODEL selects the cross-encoder (default BAAI/bge-reranker-v2-m3, ~568M params). Any sentence-transformers-compatible cross-encoder works โ€” e.g. a smaller/faster one:

RERANK_MODEL=cross-encoder/ms-marco-MiniLM-L-6-v2

Measured with clawdiney-eval --mode hybrid --rerank against the fixture vault (CPU fallback, 8 queries, warm model):

Model Params Wall time (8 queries) recall@5 / MRR / hit_rate
BAAI/bge-reranker-v2-m3 (default) ~568M ~80s (~10s/query) 1.00 / 1.00 / 1.00
cross-encoder/ms-marco-MiniLM-L-6-v2 ~22M ~30s (~4s/query) 1.00 / 1.00 / 1.00
(rerank disabled) โ€” fastest, RRF order only run --no-rerank to measure on your own vault

On this small fixture both models score identically, so the numbers only show the latency delta โ€” they don't prove the smaller model preserves precision on a larger, harder vault. Before adopting a non-default RERANK_MODEL, re-run clawdiney-eval --all-modes against your own vault (or an expanded golden set) and compare against the current baseline (tests/eval/baseline.json) rather than trusting these numbers directly.

Ollama connection error

  • Confirm Ollama is running: ollama list. Start it with ollama serve if not.
  • Confirm the embedding model is pulled: ollama pull bge-m3.

๐Ÿ“š Useful Commands

# Check brain.db health (documents/chunks/entities/relations, Ollama, reranker)
# via any MCP client, or directly:
./venv/bin/python3 -c "from clawdiney.storage import get_storage; print(get_storage().stats())"

# Index all configured vaults
./venv/bin/python3 -m clawdiney.indexer

# Check sync status
./venv/bin/python3 -m clawdiney.scripts.sync_vault --status

# Incremental sync (only changed files)
./venv/bin/python3 -m clawdiney.scripts.sync_vault

# Full sync (reindex everything)
./venv/bin/python3 -m clawdiney.scripts.sync_vault --full

# Real-time file watcher
./venv/bin/python3 -m clawdiney.scripts.watch_vault

# Test search from the command line
./scripts/ask_brain.sh "your query"

โ“ FAQ (Frequently Asked Questions)

"Do I need Obsidian installed?"

No. Obsidian is just an editor. The Brain reads .md files directly, so you only need the Vault files.

"Can I use my personal vault?"

Yes! Use the symlink approach:

# Add clawdiney.toml to your personal vault
cat >> /path/to/your/vault/clawdiney.toml << 'EOF'
id = "general"
name = "General"
description = "Personal Obsidian vault"
linked_vaults = []
EOF

# Symlink it as the general vault
ln -sfn /path/to/your/vault ~/clawdiney-vaults/general

# Reindex
./venv/bin/python3 -m clawdiney.indexer

Your vault becomes the fallback general vault โ€” searched whenever a more specific vault doesn't match.

"How long does indexing take?"

Initial full sync depends on vault size:

  • Small vault (< 100 notes): ~30 seconds
  • Medium vault (100-500 notes): 1-2 minutes
  • Large vault (> 500 notes): 3-5 minutes

Incremental sync (subsequent syncs) only processes changed files and is much faster (typically 1-5 seconds per file).

"Do I need to re-index every time I update an SOP?"

No! Auto-sync handles this automatically:

  • On MCP startup: Checks for changes and syncs automatically
  • With Watcher mode: Syncs in real-time as you edit files
  • Manual: Run python sync_vault.py for on-demand sync

"Does it work on Windows?"

Yes! Through WSL2 (Windows Subsystem for Linux):

  1. Install WSL2: wsl --install (in PowerShell as Admin)
  2. Install Ollama for Windows (or inside WSL2) โ€” see ollama.com
  3. Inside WSL2, follow the normal installation instructions as if it were Linux

"Which Linux distribution is recommended?"

The system has been tested mainly on Ubuntu 22.04+ and Debian 11+, but it should work on any modern distribution with Python 3.10+ and Ollama โ€” no other OS-level requirements.

"What if I use another model instead of Qwen in Ollama?"

It works normally. The Brain is model-agnostic. You use whatever model you prefer in Claude Code. The bge-m3 is just for generating embeddings (vectors), not for answering questions.

"Can the agent write new notes automatically?"

Yes! The MCP server includes write tools:

  • write_note(path, content) - Create or update any note
  • append_to_daily(content) - Add to today's daily note
  • add_learning(topic, content, area) - Save learnings to the right folder

When the agent learns something new during a task, it can save it directly to the vault. The change is automatically indexed and becomes searchable within seconds.

"Where should I save different types of content?"

Use the add_learning() tool which automatically routes to the correct folder:

Area Folder Example
SOPs 30_Resources/SOPs/ Procedures and standards
Architecture 30_Resources/Architecture/ ADRs and design decisions
DesignSystem 30_Resources/DesignSystem/ UI components and patterns
Projects 10_Projects/ Active project documentation
Areas 20_Areas/ Ongoing responsibility areas
Learnings 30_Resources/Learnings/ General insights

๐Ÿค Contributing

To add new tools to MCP:

  1. Edit src/clawdiney/mcp_server.py
  2. Add a new function decorated with @mcp.tool()
  3. Add tests (tests/test_mcp_server.py) and run ./scripts/run_tests.sh before committing.

๐Ÿ“„ License

MIT License


Created with โค๏ธ by the Voices in My Head team

Compatibility: Linux โ€ข macOS โ€ข WSL2 โ€ข Unix-like

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

clawdiney-0.2.4.tar.gz (975.1 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

clawdiney-0.2.4-py3-none-any.whl (95.7 kB view details)

Uploaded Python 3

File details

Details for the file clawdiney-0.2.4.tar.gz.

File metadata

  • Download URL: clawdiney-0.2.4.tar.gz
  • Upload date:
  • Size: 975.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for clawdiney-0.2.4.tar.gz
Algorithm Hash digest
SHA256 7eb39d1e834d68b03d6d5634a2a520b06983e4e35ea2205ac52f02559e55d4f2
MD5 cfec94d6f4e6d0b321cf2e29b9c77cea
BLAKE2b-256 e4a3855f245375075c2688710bfb885df06cdce92f85c8473a467419893d62db

See more details on using hashes here.

Provenance

The following attestation bundles were made for clawdiney-0.2.4.tar.gz:

Publisher: publish.yml on elaranjo/clawdiney

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file clawdiney-0.2.4-py3-none-any.whl.

File metadata

  • Download URL: clawdiney-0.2.4-py3-none-any.whl
  • Upload date:
  • Size: 95.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for clawdiney-0.2.4-py3-none-any.whl
Algorithm Hash digest
SHA256 33aed89e05a69e6c737166acbf8fae6bac403a6f19d39a9feaa76f9fc87d2f36
MD5 27dcf7252eadb7769a4d20c6cddfeaf0
BLAKE2b-256 fd197ff4676b918ba59cda027cbb471126b1a0cc3e6eb48db20c153160f7cb7d

See more details on using hashes here.

Provenance

The following attestation bundles were made for clawdiney-0.2.4-py3-none-any.whl:

Publisher: publish.yml on elaranjo/clawdiney

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page