memctl 0.1.0

A Unix-native memory control plane for LLM orchestration

memctl

A Unix-native memory control plane for LLM orchestration.

One file, one truth. Ingest files, recall with FTS5, pipe into any LLM.

pip install memctl
memctl init
memctl push "project architecture" --source src/ | llm "Summarize the architecture"
echo "The architecture uses event sourcing" | memctl pull --tags arch

---

Why memctl?

LLMs forget everything between turns. memctl gives them persistent, structured, policy-governed memory backed by a single SQLite file.

• Zero dependencies — stdlib only. No numpy, no torch, no compiled extensions.
• One file — Everything in memory.db (SQLite + FTS5 + WAL).
• Unix composable — push writes to stdout, pull reads from stdin. Pipe freely.
• Policy-governed — 30 detection patterns block secrets, injection, and instructional content before storage.
• Content-addressed — SHA-256 dedup ensures idempotent ingestion.
• Forward-compatible — Identical schema to RAGIX. Upgrade seamlessly.

---

Installation

pip install memctl

For Office/ODF document ingestion (.docx, .odt, .pptx, .odp, .xlsx, .ods):

pip install memctl[docs]

For MCP server support (Claude Code / Claude Desktop):

pip install memctl[mcp]

For everything:

pip install memctl[all]

Requirements: Python 3.10+ (3.12 recommended). No compiled dependencies for core. PDF extraction requires pdftotext from poppler-utils (sudo apt install poppler-utils or brew install poppler).

---

Quickstart

1. Initialize a memory workspace

memctl init
# Creates .memory/memory.db, .memory/config.yaml, .memory/.gitignore

Set the environment variable for convenience:

eval $(memctl init)
# Sets MEMCTL_DB=.memory/memory.db

2. Ingest files and recall

# Ingest source files + recall matching items → injection block on stdout
memctl push "authentication flow" --source src/auth/

# Ingest Office documents (requires memctl[docs])
memctl push "project status" --source reports/*.docx slides/*.pptx

# Ingest PDFs (requires pdftotext)
memctl push "specifications" --source specs/*.pdf

# Recall only (no ingestion)
memctl push "database schema"

3. Store LLM output

# Pipe LLM output into memory
echo "We chose JWT for stateless auth" | memctl pull --tags auth,decision --title "Auth decision"

# Or pipe from any LLM CLI
memctl push "API design" | llm "Analyze this" | memctl pull --tags api

4. Search

# Human-readable
memctl search "authentication"

# JSON for scripts
memctl search "database" --json -k 5

5. Inspect and manage

memctl show MEM-abc123def456     # Show item details
memctl stats                     # Store metrics
memctl stats --json              # Machine-readable stats
memctl consolidate               # Merge similar STM items
memctl consolidate --dry-run     # Preview without writing

---

CLI Reference

memctl <command> [options]

Commands

Command	Description
init [PATH]	Initialize a memory workspace (default: .memory)
push QUERY [--source ...]	Ingest files + recall matching items to stdout
pull [--tags T] [--title T]	Read stdin, store as memory items
search QUERY [-k N]	FTS5 full-text search
show ID	Display a single memory item
stats	Store statistics
consolidate [--dry-run]	Deterministic merge of similar STM items
serve	Start MCP server (requires memctl[mcp])

Global Flags

Flag	Description
--db PATH	SQLite database path
--json	Machine-readable JSON output
-q, --quiet	Suppress stderr progress messages
-v, --verbose	Enable debug logging

Command Details

memctl init

memctl init [PATH] [--force] [--fts-tokenizer fr|en|raw]

Creates the workspace directory, SQLite database with schema, config.yaml, and .gitignore. Prints export MEMCTL_DB="..." to stdout for eval.

Idempotent: running twice on the same path exits 0 without error.

memctl push

memctl push QUERY [--source FILE ...] [--budget N] [--tier TIER] [--tags T] [--scope S]

Two-phase command:

1. Ingest (optional): processes --source files with SHA-256 dedup and paragraph chunking.
2. Recall: FTS5 search for QUERY, format matching items as an injection block on stdout.

stdout contains only the injection block (format_version=1). Progress goes to stderr.

memctl pull

echo "..." | memctl pull [--tags T] [--title T] [--scope S]

Reads text from stdin and stores it as memory items. Attempts structured proposal extraction first; falls back to single-note storage. All content passes through the policy engine before storage.

memctl search

memctl search QUERY [--tier TIER] [--type TYPE] [-k N] [--json]

FTS5 full-text search. Returns human-readable output by default, or JSON with --json.

memctl consolidate

memctl consolidate [--scope S] [--dry-run] [--json]

Deterministic consolidation: clusters STM items by type + tag overlap (Jaccard), merges each cluster (longest content wins), promotes to MTM. High-usage MTM items promote to LTM. No LLM calls.

---

Environment Variables

Variable	Default	Description
MEMCTL_DB	.memory/memory.db	Path to SQLite database
MEMCTL_BUDGET	2200	Token budget for injection blocks
MEMCTL_FTS	fr	FTS tokenizer preset (fr/en/raw)
MEMCTL_TIER	stm	Default write tier
MEMCTL_SESSION	(unset)	Session ID for audit provenance

Precedence: CLI --flag > MEMCTL_* env var > compiled default. Always.

---

Exit Codes

Code	Meaning
0	Success (including idempotent no-op)
1	Operational error (bad args, empty input, policy rejection)
2	Internal failure (unexpected exception, I/O error)

---

Shell Integration

Add to .bashrc, .zshrc, or your project's env.sh:

export MEMCTL_DB=.memory/memory.db

# Shortcuts
meminit()  { memctl init "${1:-.memory}"; }
memq()     { memctl push "$1"; }                        # recall only
memp()     { memctl push "$1" ${2:+--source "$2"}; }    # push with optional source
mempull()  { memctl pull --tags "${1:-}" ${2:+--title "$2"}; }

Pipe Recipes

# Ingest docs + recall + feed to LLM + store output
memctl push "API design" --source docs/ | llm "Summarize" | memctl pull --tags api

# Search and pipe to jq
memctl search "auth" --json | jq '.[].title'

# Batch ingest a directory
memctl push "project overview" --source src/ tests/ docs/ -q

# Export all items as JSONL
memctl search "" --json | jq -c '.[]'

---

MCP Server

memctl exposes 7 MCP tools for integration with Claude Code, Claude Desktop, VS Code, and any MCP-compatible client.

Start the Server

memctl serve --db .memory/memory.db
# or
python -m memctl.mcp.server --db .memory/memory.db

Claude Code Integration

Add to .claude/settings.json:

{
  "mcpServers": {
    "memctl": {
      "command": "memctl",
      "args": ["serve", "--db", ".memory/memory.db"]
    }
  }
}

MCP Tools

Tool	Description
memory_recall	Token-budgeted context injection (primary tool)
memory_search	Interactive FTS5 discovery
memory_propose	Store findings with policy governance
memory_write	Direct write (privileged/dev operations)
memory_read	Read items by ID
memory_stats	Store metrics
memory_consolidate	Trigger deterministic merge

Tool names use the memory_* prefix for drop-in compatibility with RAGIX.

---

How It Works

Architecture

memctl/
├── types.py           Data model (MemoryItem, MemoryProposal, MemoryEvent, MemoryLink)
├── store.py           SQLite + FTS5 + WAL backend (9 tables + schema_meta)
├── extract.py         Text extraction (text files + binary format dispatch)
├── ingest.py          Paragraph chunking, SHA-256 dedup, source resolution
├── policy.py          Write governance (30 patterns: secrets, injection, instructional)
├── config.py          Dataclass configuration
├── cli.py             8 CLI commands
├── consolidate.py     Deterministic merge (Jaccard clustering, no LLM)
├── proposer.py        LLM output parsing (delimiter + regex)
└── mcp/
    ├── tools.py       7 MCP tools (memory_* prefix)
    ├── formatting.py  Injection block format (format_version=1)
    └── server.py      FastMCP server entry point

14 source files. ~4,800 lines. Zero compiled dependencies for core.

Memory Tiers

Tier	Purpose	Lifecycle
STM (Short-Term)	Recent observations, unverified facts	Created by pull. Consolidated or expired.
MTM (Medium-Term)	Verified, consolidated knowledge	Created by consolidate. Promoted by usage.
LTM (Long-Term)	Stable decisions, definitions, constraints	Promoted from MTM by usage count or type.

Policy Engine

Every write path passes through the policy engine. No exceptions.

Hard blocks (rejected):

• 10 secret detection patterns (API keys, tokens, passwords, private keys, JWTs)
• 8 injection patterns (prompt override, system prompt fragments)
• 8 instructional block patterns (tool invocation syntax, role fragments)
• Oversized content (>2000 chars for non-pointer types)

Soft blocks (quarantined to STM with expiry):

• 4 instructional quarantine patterns (imperative self-instructions)
• Missing provenance or justification
• Quarantined items stored with injectable=False

FTS5 Tokenizer Presets

Preset	Tokenizer	Use Case
fr	unicode61 remove_diacritics 2	French-safe default (accent normalization)
en	porter unicode61 remove_diacritics 2	English with Porter stemming
raw	unicode61	No diacritics removal, no stemming

Expert override: memctl init --fts-tokenizer "porter unicode61 remove_diacritics 2"

Supported Formats

Category	Extensions	Requirement
Text / Markup	.md .txt .rst .csv .tsv .html .xml .json .yaml .toml	None (stdlib)
Source Code	.py .js .ts .jsx .tsx .java .go .rs .c .cpp .sh .sql .css …	None (stdlib)
Office Documents	.docx .odt	pip install memctl[docs]
Presentations	.pptx .odp	pip install memctl[docs]
Spreadsheets	.xlsx .ods	pip install memctl[docs]
PDF	.pdf	pdftotext (poppler-utils)

All formats are extracted to plain text before chunking and ingestion. Binary format libraries are lazy-imported — a missing library produces a clear ImportError with install instructions.

Content Addressing

Every ingested file is hashed (SHA-256). Re-ingesting the same file is a no-op. Every memory item stores a content_hash for deduplication.

Consolidation

Deterministic, no-LLM merge pipeline:

1. Collect non-archived STM items
2. Cluster by type + tag overlap (Jaccard similarity)
3. Merge each cluster: longest content wins; tie-break by earliest created_at, then lexicographic ID
4. Write merged items at MTM tier + supersedes links
5. Archive originals (archived=True)
6. Promote high-usage MTM items to LTM

---

Database Schema

Single SQLite file with WAL mode. 9 tables + 1 FTS5 virtual table:

Table	Purpose
memory_items	Core memory items (22 columns)
memory_revisions	Immutable revision history
memory_events	Audit log (every read/write/consolidate)
memory_links	Directional relationships (supersedes, supports, etc.)
memory_embeddings	Reserved for RAGIX (empty in memctl)
corpus_hashes	SHA-256 file dedup registry
corpus_metadata	Corpus-level metadata
schema_meta	Schema version, creation info
memory_palace_locations	Reserved for RAGIX
memory_items_fts	FTS5 virtual table for full-text search

Schema version is tracked in schema_meta. Current: SCHEMA_VERSION=1.

---

Migration to RAGIX

memctl is extracted from RAGIX and maintains schema-identical databases. To upgrade:

pip install ragix[all]
# Point at the same database — all items carry over
ragix memory stats --db .memory/memory.db

Feature	memctl	RAGIX
SQLite schema	Identical	Identical
Injection format	format_version=1	format_version=1
MCP tool names	memory_*	memory_*
FTS5 recall	Yes	Yes (+ hybrid embeddings)
Embeddings	No	Yes (FAISS + Ollama)
LLM-assisted merge	No	Yes
Graph-RAG	No	Yes
Reporting	No	Yes

---

Python API

from memctl import MemoryStore, MemoryItem, MemoryPolicy

# Open or create a store
store = MemoryStore(db_path=".memory/memory.db")

# Write an item
item = MemoryItem(
    title="Architecture decision",
    content="We chose event sourcing for state management",
    tier="stm",
    type="decision",
    tags=["architecture", "event-sourcing"],
)
store.write_item(item, reason="manual")

# Search
results = store.search_fulltext("event sourcing", limit=10)
for r in results:
    print(f"[{r.tier}] {r.title}: {r.content[:80]}")

# Policy check
policy = MemoryPolicy()
from memctl.types import MemoryProposal
proposal = MemoryProposal(
    title="Config", content="Some content",
    why_store="Important finding",
    provenance_hint={"source_kind": "doc", "source_id": "design.md"},
)
verdict = policy.evaluate_proposal(proposal)
print(verdict.action)  # "accept", "quarantine", or "reject"

store.close()

---

Testing

pip install memctl[dev]
pytest tests/ -v

210 tests covering types, store, policy, ingest, text extraction, forward compatibility, contracts, CLI (subprocess), and pipe composition.

---

License

MIT License. See LICENSE for details.

---

Author: Olivier Vitrac, PhD, HDR | [e-mail] | Adservio Innovation Lab