ecdallm 0.2.6

Retrieval-Augmented Generation - (RAG)

ecdallm

ecdallm is a lightweight Retrieval-Augmented Generation (RAG) application that lets you chat with your own documents using either a local LLM or an external OpenAI-compatible provider.

It combines:

• FastAPI web interface
• Local embedding pipeline (FastEmbed)
• Persistent vector storage (ChromaDB)
• Document ingestion (PDF, TXT, DOCX)
• CLI launcher
• Local LLM support (e.g., LM Studio)
• External LLM support (OpenAI-compatible APIs)

The goal is to provide a simple, reproducible environment for document-grounded LLM interaction with flexible model connectivity.

---

Overview

ecdallm allows you to:

1. Upload documents
2. Index them into a vector database
3. Run semantic retrieval
4. Query an LLM with grounded context

All embeddings and vector storage run locally.

The chat model can run:

• locally (LM Studio, Ollama, etc.)
• externally (OpenRouter or OpenAI-compatible APIs)

This makes the system suitable for:

• research environments
• private document analysis
• offline experimentation
• RAG prototyping
• hybrid local/cloud workflows

---

Installation

Install from PyPI:

pip install ecdallm

---

Running the application

Start the CLI:

ecdallm

The CLI will:

• find a free port (starting from 8000)
• start the FastAPI server
• open the browser automatically

Example output:

ecdallm running at http://127.0.0.1:8000/
INFO: Uvicorn running on http://127.0.0.1:8000

---

LLM Configuration

When the application starts, click Continue and choose:

• Local LLM
• External LLM

Configuration is stored in the browser session.

Embeddings always run locally using FastEmbed with:

nomic-embed-text-v1.5

---

Using a local LLM

ecdallm expects an OpenAI-compatible endpoint.

For example, with LM Studio:

1. Start LM Studio server
2. Load a chat model
3. Enable the local API server

Typical endpoint:

http://localhost:1234/v1

Default configuration:

Base URL: http://localhost:1234/v1
API Key: lm-studio

The backend automatically detects the available chat model via:

GET /models

---

Using an external LLM

ecdallm can connect to any OpenAI-compatible API provider.

Examples include:

• OpenRouter
• OpenAI-compatible gateways
• Self-hosted inference APIs

Example configuration (OpenRouter):

Base URL: https://openrouter.ai/api/v1
Model: openrouter/aurora-alpha
API Key: sk-or-...

Steps:

1. Create an account with the provider
2. Generate an API key
3. Choose a chat model
4. Enter the configuration in the web interface

When validating, ecdallm:

• checks connectivity
• performs a test chat completion
• stores configuration in session storage

Your API key is sent only to your backend for validation and is not used directly in the browser.

Embeddings remain local.

---

Supported document types

• PDF
• TXT
• DOCX

---

Workflow

1. Upload documents

Use the Upload page to add files.

2. Index documents

Files are automatically indexed into ChromaDB using FastEmbed.

3. Chat with documents

Open the Chat page and ask questions.

The assistant will:

• retrieve relevant chunks
• build a grounded prompt
• query the configured LLM
• return a concise answer

---

Project structure

ecdallm/
├── cli.py
└── app/
    ├── main.py
    ├── rag.py
    ├── paths.py
    ├── search_engine.py
    ├── vector.py
    ├── templates/
    ├── static/
    ├── uploads/
    └── rag_store/

---

Notes

Embeddings and retrieval always run locally.

The chat model can be:

• local (LM Studio, Ollama, etc.)
• external (OpenAI-compatible providers)

This keeps the system flexible while maintaining local document processing.

---

Erasmus Data Collaboratory

Developed by the Erasmus Data Collaboratory (ECDA).

• Zaman Ziabakhshganji --- creator and maintainer
• Farshad Radman --- co-author and contributor
• Jos van Dongen --- co-author and contributor

---

License

MIT License