cli-memory-os 0.1.0

CLI-based Personal Knowledge Operating System

Memory‑OS 🧠

Memory‑OS is a local Personal Knowledge Operating System that syncs, indexes, and retrieves information across your GitHub repositories, emails, and Notion workspaces. It runs a unified interactive CLI, exposing hybrid keyword + semantic search and natural language QA powered by RAG, local embeddings, and a knowledge graph.

---

🏗️ Architecture

Memory-OS is built as a modular architecture consisting of ingestion, databases, scoring ranking engines, and a terminal user loop:

graph TD
    %% Ingestion
    subgraph Ingestion [1. Ingestion Layer]
        GH[GitHub Repos & Docs]
        GM[Gmail Inbox Messages]
        NT[Notion Page Contents]
        CP[Composio Integration Platform]
        GH --> CP
        GM --> CP
        NT --> CP
    end

    %% Storage & Indexing
    subgraph Storage [2. Storage & Indexing Layer]
        DB[(SQLite: workspace.db)]
        QD[(Qdrant: Qdrant Server)]
        N4J[(Neo4j Graph Database)]
        SQL_G[(SQLite Graph Fallback)]
        
        CP -->|Insert Metadata & Docs| DB
        DB -->|Text Chunks| CH[Chunking Core]
        CH -->|Local Embeddings| EM[SentenceTransformer]
        EM -->|Vectors| QD
        
        DB -->|Graph Construction| N4J
        DB -->|Graph Construction| SQL_G
    end

    %% Retrieval & RAG
    subgraph Retrieval [3. Retrieval & RAG Layer]
        HS[Hybrid Search Router]
        QD -->|Cosine Similarity| HS
        DB -->|Keyword Matching| HS
        N4J -->|Graph Lookups| HS
        SQL_G -->|Graph Fallback Lookups| HS
        
        HR[Hybrid Ranking Scoring]
        HS --> HR
        
        RAG[RAG Context Builder]
        HR -->|Merged Context| RAG
        
        LLM[Groq LLM Pipeline]
        RAG -->|Prompt Assembly| LLM
    end

    %% User Interaction
    subgraph User [4. Interface Layer]
        CLI[main.py: CLI Command Loop]
        CLI -->|Sync/Rebuild| Ingestion
        CLI -->|Search/Ask Queries| Retrieval
        LLM -->|Formatted Answer| CLI
    end

---

🗄️ Database Technology Rationale

Memory-OS adopts a multi-model storage engine strategy, selecting each technology to excel at its designated retrieve-and-rank role:

Database	Selection Rationale
SQLite (workspace.db)	Chosen for lightweight structured storage. It holds raw documents, chunk segments, email metadata, and repository statistics, providing ACID compliance and ultra-fast exact keyword searches.
Qdrant	Chosen as a high-performance vector database optimized for storing and executing cosine similarity search queries on $384$-dimensional dense vector embeddings generated by Sentence-Transformers.
Neo4j / Fallback SQLite	Neo4j is utilized as a native graph database to map complex developer relationships (e.g. Repository-[USES]->Technology or Email-[SENT_BY]->User). If Neo4j is unreachable, it seamlessly falls back to a relational SQLite graph schema, preserving search functionality offline.

---

🗄️ Workspace Structure

Memory-OS manages directories and configuration under the user's home folder (~/.memory-os/):

~/.memory-os/
├── config.toml                     # Global TOML settings configuration
├── active_profile                  # Stores name of the currently active profile
└── workspaces/
    ├── default/                    # Default workspace profile directory
    │   ├── workspace.db            # SQLite relational database
    │   ├── qdrant/                 # Qdrant local bind-mounted storage folder
    │   ├── neo4j/                  # Neo4j local bind-mounted storage folder
    │   ├── logs/                   # Log folder containing memory_os.log
    │   └── cache/                  # Chunker cache folder
    └── personal/                   # Personal workspace profile directory

---

📦 Setup & Installation

Ensure you have Python >= 3.12 and Docker + Docker Compose installed.

1. Install via pip

You can install Memory-OS directly as a package:

pip install .

This registers the CLI entry point executable memory-os on your path.

2. Run the Initialization Wizard

Kick off the interactive wizard to verify system dependencies, configure API keys, spin up containers, and pre-warm model weights:

memory-os init

---

🚀 CLI Commands Reference

Memory-OS exposes a comprehensive CLI for administration:

Core Daemon Lifecycle

• memory-os start: Spins up Neo4j and Qdrant database services in the background using Docker Compose.
• memory-os stop: Stops the database services while retaining data directories intact.

Operations & Ingestion

• memory-os sync [--source SOURCE] [--rebuild]: Triggers incremental data imports from registered sources (GitHub, Gmail, Notion). Add --rebuild for full vector/graph resets.
• memory-os ask <question>: Runs natural language queries against the RAG retrieval pipeline.
• memory-os graph <repo>: Visualizes the relationships of an indexed repository in the terminal knowledge graph.

Diagnostics & Monitoring

• memory-os doctor: Analyzes connection validation health across all endpoints and prints actionable troubleshooting tips on failure.
• memory-os monitor: Displays aggregated data latencies (indexing speed, search rates, LLM times) by parsing system log traces.
• memory-os benchmark: Performs query speed runs on keyword, semantic, hybrid searches, and RAG pipelines.
• memory-os logs [--tail N]: Tails the running logs of Memory-OS (rotates at 5MB, up to 3 backups).

Configuration Management

• memory-os config show: Displays the active key-value configuration block.
• memory-os config get <key>: Fetches a nested key value (e.g. groq.model).
• memory-os config set <key> <value>: Sets a nested key value with type validation checks.
• memory-os config reset: Prompts and reverts all configurations to factory defaults.

Workspace Profile Profiles

• memory-os workspace list: Lists all profiles (* marks active).
• memory-os workspace create <name>: Allocates a new workspace folder tree.
• memory-os workspace switch <name>: Switches the active profile context.
• memory-os workspace delete <name>: Wipes profile folder directories.
• memory-os workspace info: Displays detailed record metrics (nodes, vectors, sizes) for the active profile.

Portability (Export / Import)

• memory-os export <backup-zip>: Compresses database schemas, configurations, and vector indices into a versioned zip package.
• memory-os import <backup-zip>: Overwrites the active workspace profile using files from an export package after validating versions and model compatibility.

---

🔌 Plugin System Architecture

Memory-OS features a structured connector registry. Every connector implements BaseConnector (connectors/base.py) and is registered using the @register decorator (connectors/registry.py):

from connectors.base import BaseConnector
from connectors.registry import register

@register
class SlackConnector(BaseConnector):
    name = "Slack"
    slug = "slack"

    def authenticate(self) -> bool:
        # Check OAuth or API status
        return True

    def sync(self) -> dict:
        # Fetch channels and messages
        return {"synced": 42}

    def health(self) -> tuple[bool, str]:
        return True, "Connected"

To list registered plugins:

memory-os plugins

---

🛠️ Troubleshooting

• Database Offline / Port conflicts: If Neo4j (ports 7474/7687) or Qdrant (port 6333) fails to start, modify port configurations:

  memory-os config set neo4j.port_http 7475
  memory-os config set qdrant.port 6334
  memory-os start
  

• Failing Diagnostics: Run memory-os doctor to inspect status. It provides detailed actionable tips to address common environment issues.

---

📜 License

This project is licensed under the MIT License.