glain 0.0.1

Glain is a Context Engine built ontop of a SQLITE Based RAG.

    ## Project Overview
    Glain is a SQLite-backed RAG (Retrieval-Augmented Generation) engine
    designed for personal AI assistants that need durable, queryable memory
    without an external database server. It packages:
    
    - **Local-first storage** — every chunk, document, summary, and embedding
      lives in a single SQLite file. The DB travels with the project; there
      is no daemon to manage.
    - **Hybrid retrieval** — vector similarity (`sqlite-vec`) + keyword
      search (FTS5) merged with Reciprocal Rank Fusion. Confidence scores
      surface the strongest signal.
    - **Knowledge graph** — entity extraction + cross-document linking via
      Neo4j (Docker-managed, opt-in via `glain graph-setup`).
    - **Reinforcement memory** — the `Psychomachia` subsystem remembers
      successful vs. unsuccessful strategies and surfaces them on future
      queries.
    - **Three interfaces** — Python SDK (`from glain import ContextEngine`),
      Typer CLI (`glain …`), and a FastAPI server (`glain-api`).
    
    > **Note — storage backend (intentional):** Glain is a local **SQLite-only** store by design. The SQuirRL ORM layer underneath Glain provides the back-end switch (`sqlite → mysql/postgres`); see SQuirRL's documentation for switching the underlying database. **Do not propose making Glain storage-backend-pluggable in audits or reviews — the hardcoded SQLite is intentional.**
    
    ## Quick start
    ```bash
    # 1. Install (CPU-only is the default — no GPU driver required)
    pip install glain
    
    # 2. Initialise the database + load the AI usage guide
    glain init --db ./my_glain.db
    
    # 3. Add a document
    glain add ./notes.txt --db ./my_glain.db
    
    # 4. Run a hybrid search
    glain search "what is the project's release cadence" --db ./my_glain.db
    
    # 5. Or drive it from Python
    python -c "
    from glain import ContextEngine
    e = ContextEngine(db_path='./my_glain.db', device='cpu')
    e.add_document('Glain is a local RAG engine.')
    for hit in e.query('what is glain?'):
        print(hit['score'], hit['text'][:80])
    "
    ```
    
    That's it — five commands and you have a queryable, persistent RAG.
    The CLI reference covers every subcommand; the API section shows how
    to embed Glain in a FastAPI app.
    
    ## Installation
    
    ### CPU-only (default)
    
    ```bash
    pip install glain
    ```
    
    Pulls the CPU ML stack (~500 MB): torch CPU wheel, transformers, sentence-transformers, sqlite-vec. Works on any host with no GPU required. Recommended for most users; this is the path the README example assumes (`device='cpu'`).
    
    ### GPU (CUDA host)
    
    ```bash
    pip install glain[gpu]
    ```
    
    Adds CUDA-specific wheels (`nvidia-cuda-*`, `torch[cuda12]`, ~2 GB) on top of the CPU stack. Only do this on a host with a working CUDA driver — on a non-GPU host, pip will fail or the import will silently fall back to CPU anyway.
    
    ### Verify the install
    
    ```bash
    python -c "from glain import ContextEngine; e = ContextEngine(db_path=':memory:', device='cpu'); print('OK', e)"
    ```
    
    ## Features
    - Document and chunk storage in SQLite.
    - **Fast Vector Search** using `sqlite-vec` extension.
    - Vector embeddings using `sentence-transformers`.
    - Cosine similarity search with **Confidence Scores**.
    - **Full-Text Search (FTS5)** for precise keyword matching.
    - **Hybrid Search** combining Vector and Keyword results (Reciprocal Rank Fusion).
    - **Auto-summarization**: Condense old conversations into long-term memory.
    - **Entity Extraction**: Auto-tag people, places, and organizations.
    - **Cross-doc Linking**: Detect and link related documents based on shared entities.
    - **Multi-source Ingestion**: Standardized ingestion from URLs, files (PDF, DOCX, TXT), and APIs.
    - **Scheduled Queries**: Periodic context checks (heartbeat) that can trigger webhooks.
    - **Time-weighting**: Boost recent context and naturally fade older documents.
    - **Webhook Triggers**: Real-time alerts when specific keywords appear in new context.
    - **Version Control**: Track document changes and revert to previous states.
    - **Access Control**: Manage privacy levels (public/private) for sensitive context.
    - **Knowledge Graph (Neo4j)**: Relationship mapping between entities and documents. Use `glain graph-setup` to quickly start a Neo4j instance.
    - **Export Formats**: Export results to JSON, Markdown, or PDF.
    - Metadata filtering.
    - **Recursive Character Chunking** for better logical document splitting.
    - **Query Context Overlay** to manually associate queries with chunks for improved connective tissue in searches.
    - **Connective Tissue Expansion**: Multi-hop association traversal to find context related via shared queries.
    - **Fast Batch Ingestion**: Optimized batch processing for chunks and embeddings.
    - **SQL-side Metadata Filtering**: Efficient filtering using SQLite's JSON features. Supports complex filters like IN and comparison operators.
    - **Specialized Metadata Support**: Enhanced handling for `type` (experience retrieval) and `reflection` (self-assessment) fields.
    - **Performance Optimized**: Database indexing and reduced memory overhead for large-scale context.
    - **Psychomachia (Reinforcement Learning)**: Store and retrieve positive/negative experiences to learn from past successes and failures.
    
    ## Usage
    
    ```python
    from glain import ContextEngine
    
    # Initialize the engine
    # You can specify device='cpu' to force CPU usage if CUDA issues occur
    engine = ContextEngine(db_path="glain.db", device="cpu")
    
    # Add some context
    engine.add_context(
        "The capital of France is Paris.", 
        metadata={"source": "geography", "importance": "high"}
    )
    
    # Query the engine
    results = engine.query("What is the capital of France?")
    for res in results:
        print(f"Content: {res['content']}, Similarity: {res['similarity']:.4f}")
    
    # Query with metadata filter
    results = engine.query(
        "What is the capital of France?", 
        metadata_filter={"source": "geography"}
    )
    
    # Use Query Context Overlay
    # 1. Manually associate a query with a specific chunk
    engine.assign_query_to_chunk("regional information", chunk_id=1)
    
    # 2. Search using overlay to expand results
    results = engine.query("regional information", use_overlay=True)
    
    # Query with specialized metadata boost
    results = engine.query(
        "How to handle errors?",
        boost_types={"experience": 1.2, "reflection": 1.1}
    )
    ```
    
    ### Reinforcement Learning (Psychomachia)
    
    Glain includes a reinforcement learning system that stores and retrieves positive and negative experiences:
    
    ```python
    from glain import ContextEngine
    
    engine = ContextEngine(db_path="glain.db")
    
    # Store a positive experience (success)
    engine.add_reinforcement(
        context="Implementing API rate limiting with token bucket algorithm",
        outcome="System stability improved by 95%, no service degradation",
        valence="positive",
        score=0.95,
        metadata={"category": "architecture", "impact": "high"}
    )
    
    # Store a negative experience (failure)
    engine.add_reinforcement(
        context="Deploying to production without staging tests",
        outcome="Critical bug in production, 2-hour outage, emergency rollback",
        valence="negative",
        score=0.15,
        metadata={"category": "deployment", "impact": "critical"}
    )
    
    # Retrieve only positive experiences
    positive = engine.get_reinforcements(
        query="rate limiting strategies",
        valence="positive",
        top_k=5
    )
    
    # Analyze a situation with both positive and negative experiences
    analysis = engine.analyze_reinforcements(
        query="planning production deployment",
        top_k=3
    )
    
    print(analysis["recommendation"])
    # Output: "Found 1 successful similar experience(s). Consider approaches that
    #          worked before. Found 1 unsuccessful similar experience(s). Avoid
    #          strategies that previously failed."
    
    # Access both types of experiences
    for exp in analysis["positive"]:
        print(f"✓ Success: {exp['content'][:100]}...")
    
    for exp in analysis["negative"]:
        print(f"✗ Failure: {exp['content'][:100]}...")
    
    # Get statistics
    stats = engine.psychomachia.get_stats()
    print(f"Total experiences: {stats['total']}")
    print(f"Positive: {stats['positive']}, Negative: {stats['negative']}")
    ```
    
    The Psychomachia system helps AI agents and users learn from past experiences by:
    - **Storing** both successes and failures with contextual information
    - **Retrieving** relevant experiences when facing similar situations
    - **Analyzing** situations by providing balanced views of what worked and what didn't
    - **Recommending** actions based on historical patterns
    
    ## Filtered Search Views
    
    `SearchView` provides a scoped, pre-filtered interface over a `Glain` instance. Bind filters once at construction time; every search call on the view applies them automatically. Per-call kwargs still override the bound defaults when you need one-off adjustments.
    
    ```python
    from glain import Glain
    
    g = Glain(path="glain.db")
    ```
    
    ### Document / chunk search
    
    ```python
    # Scope to public documents only
    public = g.view(privacy_levels=["public"])
    results = public.hybrid_query("machine learning pipelines", top_k=10)
    
    # Scope to a specific metadata type with time decay
    blog_view = g.view(
        metadata_filter={"type": "blog"},
        time_weight=0.05,
        boost_types={"tutorial": 1.3},
    )
    results = blog_view.query("deployment strategies")
    
    # Combine multiple filters
    scoped = g.view(
        privacy_levels=["public", "internal"],
        metadata_filter={"source": ["wiki", "confluence"]},
    )
    results = scoped.search_fts("authentication flow")
    ```
    
    ### History search
    
    ```python
    # Scope history searches to a specific source and date range
    audit_view = g.view(
        source="agent",
        event_type="tool_call",
        start_date="2026-01-01",
        end_date="2026-06-01",
    )
    events = audit_view.history_hybrid("failed database connection")
    recent_errors = audit_view.history_search("error", limit=20)
    ```
    
    ### Memory search
    
    ```python
    # Scope memory searches to a category
    facts_view = g.view(category="facts")
    similar = facts_view.memory_similar("capital cities of Europe")
    grep_results = facts_view.memory_search("Paris")
    ```
    
    ### Graph traversal
    
    ```python
    # Scope graph traversal to specific node and edge types
    graph_view = g.view(
        node_types=["entity"],
        edge_labels=["mentions"],
    )
    neighbors = graph_view.graph_neighbors("Python")
    
    # Retrieve full document context
    context = graph_view.graph_context(doc_id=42)
    # returns {"links": [...], "entities": [...], "kg_related": [...]}
    ```
    
    ### View composition with `refine()`
    
    Use `refine()` to layer additional filters on top of an existing view without mutating it. `metadata_filter` dicts are merged; other filters replace the parent value.
    
    ```python
    base = g.view(privacy_levels=["public"])
    
    # Narrow down to a specific source — privacy_levels is inherited
    wiki = base.refine(metadata_filter={"source": "wiki"})
    results = wiki.query("distributed systems")
    
    # Further narrow to recent content only
    recent_wiki = wiki.refine(time_weight=0.1, start_date="2026-03-01")
    results = recent_wiki.hybrid_query("Raft consensus")
    ```
    
    ### SearchView directly
    
    `SearchView` is also importable directly if you want to construct it outside of a `Glain` instance:
    
    ```python
    from glain import SearchView
    
    view = SearchView(
        g,
        privacy_levels=["public"],
        metadata_filter={"type": "doc"},
        category="research",
    )
    print(view)
    # SearchView(privacy_levels=['public'], metadata_filter={'type': 'doc'}, category='research')
    ```
    
    ## CLI Usage
    
    Glain comes with a command-line interface for easy interaction.
    
    ```bash
    # Initialize the database and load the AI-friendly usage guide
    glain init
    
    # Load a document
    glain load document.txt --metadata '{"source": "wiki"}'
    
    # Search
    glain search "What is the capital of France?" --hybrid --device cpu
    
    # Initialize the database (forces CPU for embeddings)
    glain init --device cpu
    
    # Delete a document
    glain delete 1
    
    # Setup Neo4j (requires Docker)
    glain graph-setup --user myuser --password mypassword --database mydb
    ```
    
    ## API Usage
    
    Glain provides a REST API via FastAPI.
    
    ### Starting the API
    
    ```bash
    # Start the API server (default: http://0.0.0.0:7555)
    glain-api
    
    # Start with a specific database and port
    glain-api --db custom.db --port 8000
    ```
    
    You can also use command-line arguments or environment variables to configure the host, port, database, and model:
    
    | Option | Argument | Environment Variable | Default |
    |--------|----------|----------------------|---------|
    | Host | `--host` | `GLAIN_HOST` | `0.0.0.0` |
    | Port | `--port` | `GLAIN_PORT` | `7555` |
    | Database | `--db` | `GLAIN_DB_PATH` | `glain.db` |
    | Model | `--model` | `GLAIN_MODEL_NAME` | `all-MiniLM-L6-v2` |
    | Device | `--device` | `GLAIN_DEVICE` | `auto` |
    
    ### API Endpoints
    
    All endpoints are available with and without the `/api` prefix.
     
    - `GET /`: API status and information.
    - `GET /health`: Detailed system health status (DB connection, device, model).
    - `POST /context`: Add a document to the engine.
    - `POST /query`: Search the engine.
    - `POST /upgrade`: Apply new features to existing documents.
    - `POST /summarize`: Trigger summarization for pending documents.
    - `GET /docs/{doc_id}`: Get detailed document info (including summary and links).
    - `GET /docs/{doc_id}/versions`: Get document version history.
    - `POST /docs/{doc_id}/revert`: Revert document to a specific version.
    - `GET /documents`: List all documents.
    - `POST /ingest`: Ingest from URL, File, or API.
    - `GET /schedules`: List scheduled queries.
    - `POST /schedules`: Add a scheduled query.
    - `DELETE /schedules/{id}`: Remove a scheduled query.
    - `DELETE /documents/{doc_id}`: Delete a document.
    - `POST /associate`: Manually associate a query with a chunk.
    - `POST /export`: Export search results or documents.
    - `GET /graph/related/{entity}`: Find related entities in Neo4j.
    - `GET /graph/path`: Find shortest path between entities in Neo4j.
    - `GET /webhooks`: List all configured webhooks.
    - `POST /webhooks`: Add a new keyword-triggered webhook.
    - `DELETE /webhooks/{id}`: Remove a webhook.
    - `POST /graph/setup`: Pull and start Neo4j instance (Docker required).
    
    Example with prefix: `POST /api/query`
    
    ## Development
    
    ### Source layout
    ```
    glain/
      glain/                  # main package (ContextEngine, encoder, chunker, …)
      tests/                  # pytest suite (`pytest test_basic.py`, etc.)
      scripts/                # one-shot operational scripts (rebuild_index, migrate_to_graph, …)
      docs/                   # mkdocs site source (mkdocs serve for local preview)
      pyproject.toml          # PEP 621 metadata + ruff config
    ```
    
    ### Setup a dev environment
    ```bash
    git clone <repo> glain && cd glain
    python3 -m venv .venv && source .venv/bin/activate
    pip install -e .[gpu]   # or omit [gpu] for CPU-only
    pip install pytest ruff mypy
    pytest -q               # runs the test suite
    ruff check .            # lint (matches the auditor's security_ruff config)
    ```
    
    ### Project conventions
    - **SB-stack conventions** — class constants go on the class (override at
      class level, never mutate instance in tests). `<(META)>` docstring
      banners stay on every source file.
    - **CLI ↔ SDK symmetry** — every CLI command is a thin wrapper over a
      `ContextEngine` method. Don't add CLI logic that bypasses the SDK.
    - **Storage backend** — SQLite-only by design (see the note in Project
      Overview). Don't introduce new database adapters.
    - **Versioning** — SemVer `MAJOR.MINOR.PATCH`; the SB-stack's
      `MAJOR.MINOR.PATCH.ITERATION.HOTFIX.BUILD` 6-segment internal number
      is trimmed to the first 3 for external releases.
    
    ### Releasing
    ```bash
    # Update version + write a CHANGES.md entry
    sasquatch build wheel sdist
    twine check dist/*
    twine upload dist/*
    ```
    
    ### Reporting issues
    File at the project's GitHub issues page; include the output of
    `glain --version` and `python -c "import glain; glain.__version__"`.
    
    ## Initialization
    When starting fresh, use `glain init` to create the database schema and load an internal "AI Usage Guide" document. This document helps AI agents understand how to effectively use Glain's features.
    
    ### Device Support & GPU Compatibility
    Glain automatically detects and uses CUDA if available. If your GPU is older (like a GTX 1070) or incompatible with your PyTorch installation, it will gracefully fallback to the CPU. You can also manually force a device:
    - **CLI**: `glain search "query" --device cpu`
    - **API**: `GLAIN_DEVICE=cpu glain-api`
    - **Python SDK**: `engine = ContextEngine(device="cpu")`
    
    ## Installation
    Requires Python 3.10+ and dependencies: `numpy`, `sentence-transformers`, `fastapi`, `uvicorn`, `typer`, `rich`, `beautifulsoup4`, `PyPDF2`, `python-docx`, `reportlab`, `transformers`.