dagestan 1.0.0

Temporal graph memory layer for large language models

Dagestan

Temporal Graph Memory Layer for LLMs

Dagestan stores LLM memory as a typed temporal knowledge graph instead of flat vector embeddings. It tracks entities, concepts, events, preferences, and goals — with time-aware confidence decay, contradiction detection, and relationship-based retrieval.

Why

Current LLM memory solutions (vector DBs) treat memory as a pile of embeddings retrieved by similarity. They don't understand:

• Time — old information and new information are treated the same
• Relationships — there's no structure between memories
• Contradictions — conflicting facts coexist silently
• Decay — nothing fades; nothing is curated

Dagestan addresses these gaps with a graph-based approach where memory has structure, relationships, and temporal awareness.

Status

v0.1 — Foundation release. Working memory layer with:

• Typed temporal graph (Entity, Concept, Event, Preference, Goal nodes)
• LLM-based knowledge extraction from conversations
• Contradiction detection between conflicting preferences/goals
• Exponential temporal decay on confidence scores
• Gap detection (incomplete entity profiles)
• Bridge node detection (cross-cluster connections)
• Centrality-based importance scoring
• Query-driven graph retrieval (keyword + structure, no embeddings)
• JSON persistence
• CLI for graph inspection
• 63 passing unit tests

Install

# Core (no LLM dependencies)
pip install -e .

# With OpenAI
pip install -e ".[openai]"

# With Anthropic
pip install -e ".[anthropic]"

# Development
pip install -e ".[dev]"

Quick Start

from dagestan import Dagestan

# Initialize with an LLM provider for extraction
mem = Dagestan(provider="openai", db_path="./memory.json")

# Ingest conversation text
mem.ingest("User mentioned they love Python and want to build a startup focused on graph databases.")

# Retrieve relevant context
context = mem.retrieve("What does the user care about?")
print(context)

# Run curation (applies decay, finds contradictions)
report = mem.curate()
print(f"Contradictions found: {report.contradictions_found}")

# Get structured context for next conversation
strategy = mem.strategy()
print(strategy)

Without an LLM (manual graph building)

from dagestan import Dagestan, Node, Edge, NodeType, EdgeType

mem = Dagestan(db_path="./memory.json", auto_save=True)

# Add nodes directly
user = mem.add_node(Node(type=NodeType.ENTITY, label="User"))
pref = mem.add_node(Node(type=NodeType.PREFERENCE, label="Prefers tea"))
goal = mem.add_node(Node(type=NodeType.GOAL, label="Build Dagestan"))

mem.add_edge(Edge(source_id=user.id, target_id=pref.id, type=EdgeType.HAS_PREFERENCE))
mem.add_edge(Edge(source_id=user.id, target_id=goal.id, type=EdgeType.WANTS))

# Query the graph
results = mem.retrieve("user preferences")
print(results)

CLI

# Show graph summary
dagestan info --db ./memory.json

# List all nodes
dagestan nodes --db ./memory.json

# Filter by type
dagestan nodes --type preference --db ./memory.json

# Query the graph
dagestan retrieve "user goals" --db ./memory.json

# Run curation
dagestan curate --db ./memory.json

# Export full graph
dagestan export --db ./memory.json

Architecture

Conversation → Extraction (LLM) → Temporal Graph → Operations → Retrieval
                                       ↓
                                   Curation
                                 (decay, contradictions,
                                  gaps, bridges)

Node Types: Entity, Concept, Event, Preference, Goal

Edge Types: relates_to, caused, contradicts, happened_before, has_preference, wants

Temporal Metadata: Every node carries created_at, last_reinforced, confidence_score, and decay_rate. Confidence degrades exponentially unless reinforced by new conversation.

Graph Operations

These run on the graph structure — no LLM needed for computation.

Operation	What it does
Contradiction Detection	Finds conflicting preferences/goals for the same entity
Temporal Decay	Reduces confidence based on time since last reinforcement
Gap Detection	Identifies entities with incomplete knowledge profiles
Bridge Detection	Finds nodes connecting otherwise disconnected clusters
Centrality Scoring	Ranks nodes by connection count + recency

Project Structure

dagestan/
├── __init__.py              # Main Dagestan class (public API)
├── cli.py                   # Command-line interface
├── graph/
│   ├── schema.py            # Node, Edge, NodeType, EdgeType
│   ├── temporal_graph.py    # Core graph with CRUD + snapshots
│   └── operations.py        # Contradiction, decay, gap, bridge, centrality
├── extraction/
│   ├── extractor.py         # Conversation → graph via LLM
│   └── prompts.py           # Extraction prompt templates
├── curation/
│   ├── curator.py           # Curation pipeline orchestrator
│   └── strategy.py          # Context strategy generation
├── retrieval/
│   └── retriever.py         # Query-driven graph traversal
├── storage/
│   └── store.py             # JSON persistence (SQLite planned for v0.2)
├── integrations/            # Planned: drop-in OpenAI/Anthropic wrappers
viz/
├── __init__.py
├── __main__.py              # Entry point (python -m viz)
├── server.py                # Stdlib HTTP server + REST API + SSE
├── watcher.py               # File-change polling (no deps)
├── export.py                # LaTeX/TikZ, DOT, CSV export
├── generate_demo.py         # Demo graph generator
└── static/
    ├── index.html           # Main UI page
    ├── style.css            # Dark-theme styling
    └── app.js               # vis-network graph rendering

Visualization

Dagestan ships with a zero-dependency interactive graph visualizer for debugging and exploring your knowledge graphs in the browser.

# Launch with auto-detected graph file
python -m viz.server

# Or point at a specific file and port
python -m viz.server --file demo_memory.json --port 8765

Then open http://localhost:8765.

Features

Feature	Description
Live Graph Rendering	Interactive force-directed layout via vis-network
Auto-Refresh / SSE	File watcher detects changes and pushes updates via Server-Sent Events
Diff Highlighting	New nodes glow green with a ✦ marker; removed counts shown in dashboard
Node & Edge Inspector	Click any element to view full metadata, confidence, decay rate
Neighborhood Explorer	Double-click a node to zoom into its local neighborhood
Stats Dashboard	Node/edge counts, type distributions, confidence histograms
Temporal Decay View	Color-coded confidence levels (green → yellow → red)
Filter by Type	Toggle node types on/off
Search	Find nodes by label (keyboard shortcut: /)
LaTeX / DOT / CSV Export	Export graphs for research papers (TikZ, Graphviz DOT, CSV)
File Switcher	Browse and switch between graph JSON files in the project
Keyboard Shortcuts	R refresh, F fit view, / search, ? help, Esc clear

Generate a Demo Graph

python -m viz.generate_demo --output viz_demo_graph.json --nodes 25
python -m viz.server --file viz_demo_graph.json

See viz/README.md for full details.

Roadmap

Version	Focus	Status
v0.1	Core graph, extraction, basic operations, JSON storage	Done
v0.2	SQLite backend, all 5 operations fully tuned	Planned
v0.3	Query-driven traversal improvements, context compression	Planned
v1.0	Benchmarks, paper, honest evaluation vs baselines	Planned
v2.0	Graph operations as reasoning substrate	Research

Evaluation (LOCOMO Benchmarks)

Dagestan includes an evaluation harness for benchmarking against the LOCOMO benchmark.

Evaluation code/config lives under evals/:

• evals/run_locomo_eval.py — main evaluation runner (QA, summarization, coherence)
• evals/run_ablations.py — ablation study runner
• evals/configs/*.yaml — model/data settings (e.g., smoke_test.yaml, locomo_gemini_flash.yaml)

Quick commands (from repo root):

# Install eval-only dependencies (dataset download / caching)
pip install datasets huggingface_hub pyyaml

# Download/cache the LOCOMO dataset locally
make download-locomo

# Smoke run (fast dry-run)
make smoke

# Full benchmark run
make locomo

# Single tasks
make qa
make summarization
make coherence

# Ablations
make ablations
make ablations-baselines

Notes:

• The runner writes outputs under evals/results/<run_id>/.
• The make smoke target uses the provider: stub setting from evals/configs/smoke_test.yaml. If your current Dagestan backend does not support provider="stub" for extraction/ingestion, run with a real provider (e.g., openai / anthropic) by editing the config.

Performance claims should be made only after you complete the benchmark runs.

Constraints

• Zero GPU required — runs on CPU only
• Minimal dependencies — core is stdlib only, LLM client is optional
• LLM-agnostic — works with any provider via simple callable interface
• Storage-agnostic — JSON now, SQLite/Neo4j planned
• Honest evaluation — we measure where it fails, not just where it wins

License

MIT