donkit-ragops 0.4.15

CLI agent for building RAG pipelines

RAGOps Agent

Optimal RAG in hours, not months.

A smart, LLM-powered CLI agent that automates the entire lifecycle of Retrieval-Augmented Generation (RAG) pipelines — from creation and experimentation to deployment. Forget spending months tweaking chunking strategies, embeddings, and vector DBs by hand. Just describe what you need, and let the agent run 100+ parallel experiments to discover what actually works for your data — fast, accurate, and infra-agnostic.

Built by Donkit AI — Automated Context Engineering.

Who is this for?

• AI Engineers building assistants and agents
• Teams in need of accuracy-sensitive and multiagentic RAG where errors compound across steps
• Organizations aiming to reduce time-to-value for production AI deployments

Key Features

• Parallel Experimentation Engine — Explores 100s of pipeline variations (chunking, vector DBs, prompts, rerankers, etc.) to find what performs best — in hours, not months.
• Docker Compose orchestration — Automated deployment of RAG infrastructure (vector DB, RAG service)
• Built-in Evaluation & Scoring — Automatically generates evaluation dataset (if needed), runs Q&A tests and scores pipeline accuracy on your real data.
• Multiple LLM providers — Supports Vertex AI (Recommended), OpenAI, Anthropic Claude, Azure OpenAI, Ollama, OpenRouter

Main Capabilities

• Interactive REPL — Start an interactive session with readline history and autocompletion
• Web UI — Browser-based interface at http://localhost:8067 (donkit-ragops-web, auto-opens browser)
• Docker Compose orchestration — Automated deployment of RAG infrastructure (vector DB, RAG service)
• Integrated MCP servers — Built-in support for full RAG build pipeline (planning, reading, chunking, vector loading, querying, evaluation)
• Checklist-driven workflow — Each RAG project is structured as a checklist — with clear stages, approvals, and progress tracking
• Session-scoped checklists — Only current session checklists appear in the UI
• SaaS mode — Connect to Donkit cloud for experiments
• Enterprise mode — deploy to VPC or on-premises with no vendor lock-in (reach out to us via https://donkit.ai)

Quick Install

The fastest way to install Donkit RAGOps. The installer automatically handles Python and dependencies.

macOS / Linux:

curl -sSL https://raw.githubusercontent.com/donkit-ai/ragops/main/scripts/install.sh | bash

Windows (PowerShell):

irm https://raw.githubusercontent.com/donkit-ai/ragops/main/scripts/install.ps1 | iex

After installation:

donkit-ragops        # Start CLI agent
donkit-ragops-web    # Start Web UI (browser opens automatically at http://localhost:8067)

---

Installation (Alternative Methods)

Option A: Using pipx (Recommended)

# Install pipx if you don't have it
pip install pipx
pipx ensurepath

# Install donkit-ragops
pipx install donkit-ragops

Option B: Using pip

pip install donkit-ragops

Option C: Using Poetry (for development)

# Create a new project directory
mkdir ~/ragops-workspace
cd ~/ragops-workspace

# Initialize Poetry project
poetry init --no-interaction --python="^3.12"

# Add donkit-ragops
poetry add donkit-ragops

# Activate the virtual environment
poetry shell

After activation, you can run the agent with:

donkit-ragops

Or run directly without activating the shell:

poetry run donkit-ragops

Quick Start

Prerequisites

• Python 3.12+ installed
• Docker Desktop installed and running (required for vector database)
  • Windows users: Docker Desktop with WSL2 backend is fully supported
• API key for your chosen LLM provider (Vertex AI, OpenAI, or Anthropic)

Step 1: Install the package

pip install donkit-ragops

Step 2: Run the agent (first time)

donkit-ragops

On first run, an interactive setup wizard will guide you through configuration:

1. Choose your LLM provider (Vertex AI, OpenAI, Anthropic, or Ollama)
2. Enter API key or credentials path
3. Optional: Configure log level
4. Configuration is saved to .env file automatically

That's it! No manual .env creation needed - the wizard handles everything.

Alternative: Manual configuration

If you prefer to configure manually or reconfigure later:

# Run setup wizard again
donkit-ragops --setup

Or create a .env file manually in your working directory:

# Vertex AI (Google Cloud)
RAGOPS_LLM_PROVIDER=vertex
RAGOPS_VERTEX_CREDENTIALS=/path/to/service-account-key.json

# OpenAI
RAGOPS_LLM_PROVIDER=openai
RAGOPS_OPENAI_API_KEY=sk-...
RAGOPS_LLM_MODEL=gpt-4o-mini # Specify the OpenAI model to use
# RAGOPS_OPENAI_BASE_URL=https://api.openai.com/v1

# Anthropic Claude
RAGOPS_LLM_PROVIDER=anthropic
RAGOPS_ANTHROPIC_API_KEY=sk-ant-...

# Ollama (local)
RAGOPS_LLM_PROVIDER=ollama
RAGOPS_OLLAMA_BASE_URL=http://localhost:11434

Step 3: Start using the agent

Tell the agent what you want to build:

you> Create a RAG pipeline for my documents in /Users/myname/Documents/work_docs

The agent will automatically:

• ✅ Create a projects/<project_id>/ directory
• ✅ Plan RAG configuration
• ✅ Process and chunk your documents
• ✅ Start Qdrant vector database (via Docker)
• ✅ Load data into the vector store
• ✅ Deploy RAG query service

What gets created

./
├── .env                          # Your configuration (auto-created by wizard)
└── projects/
    └── my-project-abc123/        # Auto-created by agent
        ├── compose/              # Docker Compose files
        │   ├── docker-compose.yml
        │   └── .env
        ├── chunks/               # Processed document chunks
        └── rag_config.json       # RAG configuration

Usage

Note: The command ragops-agent is also available as an alias for backward compatibility.

The agent starts in interactive REPL mode by default. Use subcommands like ping for specific actions.

Interactive Mode (REPL)

# Start interactive session
donkit-ragops

# With specific provider
donkit-ragops -p vertexai

# With custom model
donkit-ragops -p openai -m gpt-4

# Start in SaaS/enterprise mode (requires login first)
donkit-ragops --enterprise

REPL Commands

Inside the interactive session, use these commands:

• /help, /h, /? — Show available commands
• /exit, /quit, /q — Exit the agent
• /clear — Clear conversation history and screen
• /provider — Switch LLM provider interactively
• /model — Switch LLM model interactively

Command-line Options

• -p, --provider — Override LLM provider from settings
• -m, --model — Specify model name
• -s, --system — Custom system prompt
• --local — Force local mode (default)
• --saas — Force SaaS mode (requires login)
• --enterprise — Force enterprise mode (requires login)
• --setup — Run setup wizard to reconfigure
• --show-checklist/--no-checklist — Toggle checklist panel (default: shown)

Subcommands

# Health check
donkit-ragops ping

# Saas/Enterprise mode authentication
donkit-ragops login --token YOUR_TOKEN  # Login to Donkit cloud
donkit-ragops logout                    # Remove stored token
donkit-ragops status                    # Show mode and auth status

Environment Variables

LLM Provider Configuration

• RAGOPS_LLM_PROVIDER — LLM provider name (e.g., openai, vertex, azure_openai, ollama, openrouter)
• RAGOPS_LLM_MODEL — Specify model name (e.g., gpt-4o-mini for OpenAI, gemini-2.5-flash for Vertex)

OpenAI / OpenRouter / Ollama

• RAGOPS_OPENAI_API_KEY — OpenAI API key (also used for OpenRouter and Ollama)
• RAGOPS_OPENAI_BASE_URL — OpenAI base URL (default: https://api.openai.com/v1)
  • OpenRouter: https://openrouter.ai/api/v1
  • Ollama: http://localhost:11434/v1
• RAGOPS_OPENAI_EMBEDDINGS_MODEL — Embedding model name (default: text-embedding-3-small)

Azure OpenAI

• RAGOPS_AZURE_OPENAI_API_KEY — Azure OpenAI API key
• RAGOPS_AZURE_OPENAI_ENDPOINT — Azure OpenAI endpoint URL
• RAGOPS_AZURE_OPENAI_API_VERSION — Azure API version (default: 2024-02-15-preview)
• RAGOPS_AZURE_OPENAI_DEPLOYMENT — Azure deployment name for chat model
• RAGOPS_AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT — Azure deployment name for embeddings model

Vertex AI (Google Cloud)

• RAGOPS_VERTEX_CREDENTIALS — Path to Vertex AI service account JSON
• RAGOPS_VERTEX_PROJECT — Google Cloud project ID (optional, extracted from credentials if not set)
• RAGOPS_VERTEX_LOCATION — Vertex AI location (default: us-central1)

Anthropic

• RAGOPS_ANTHROPIC_API_KEY — Anthropic API key

Logging

• RAGOPS_LOG_LEVEL — Logging level (default: INFO)
• RAGOPS_MCP_COMMANDS — Comma-separated list of MCP commands

Agent Workflow

The agent follows a structured workflow:

1. Language Detection — Detects user's language from first message
2. Project Creation — Creates project directory structure
3. Checklist Creation — Generates task checklist in user's language
4. Step-by-Step Execution:
   • Asks for permission before each step
   • Marks item as in_progress
   • Executes the task using appropriate MCP tool
   • Reports results
   • Marks item as completed
5. Deployment — Sets up Docker Compose infrastructure
6. Data Loading — Loads documents into vector store

Web UI

RAGOps includes a browser-based interface for easier interaction:

# Start Web UI server (browser opens automatically)
donkit-ragops-web

# Start Web UI without opening browser
donkit-ragops-web --no-browser

# Development mode with hot reload
donkit-ragops-web --dev

The browser will automatically open at http://localhost:8067. The Web UI provides:

• Visual project management
• File upload and attachment
• Real-time agent responses
• Checklist visualization
• Settings configuration

SaaS Mode

SaaS mode is a fully managed SaaS platform. All backend infrastructure — databases, vector stores, RAG services, and experiment runners — is hosted by Donkit. You get the same CLI interface, but with powerful cloud features.

Setup

# 1. Login with your API token
donkit-ragops login --token YOUR_API_TOKEN

# 2. Start in SaaS mode
donkit-ragops --saas

# 3. Check status
donkit-ragops status

# 4. Logout when done
donkit-ragops logout

What's Included

• Managed infrastructure — No Docker, no local setup. Everything runs in Donkit cloud
• Automated experiments — Run 100+ RAG architecture iterations to find optimal configuration
• Experiment tracking — Compare chunking strategies, embeddings, retrievers side-by-side
• Evaluation pipelines — Batch evaluation with precision/recall/accuracy metrics
• File attachments — Attach files using @/path/to/file syntax in chat
• Persistent history — Conversation and project history preserved across sessions
• MCP over HTTP — All MCP tools executed server-side

Enterprise Mode

Enterprise mode runs fully inside your infrastructure — no data ever leaves your network. All components — from vector databases to experiment runners — are deployed within your VPC, Kubernetes cluster, or even a single secured server. You get the same CLI and web UI, but with full control over data, compute, and compliance. No vendor lock-in, no hidden dependencies — just RAG automation, on your terms.

What's Included

• Self-hosted infrastructure — Run the full Donkit stack in your VPC, Kubernetes cluster, or air-gapped server
• Automated experiments — Execute 100+ RAG variations locally to identify the best-performing pipeline
• Experiment tracking — Monitor and compare pipeline variants (chunking, retrieval, reranking) within your environment
• Evaluation pipelines — Run secure, on-prem evaluation with precision, recall, and answer relevancy metrics
• Local file attachments — Add documents from using @/path/to/file in chat or or connect your data sources via APIs
• Session-based state — Preserve project and conversation history within your private deployment
• MCP over IPC — All orchestration runs inside your infrastructure; no external HTTP calls required

Modes of work comparison

Feature	Local Mode	SaaS Mode	Enterprise Mode
Infrastructure	Self-hosted (Docker)	Managed by Donkit	Managed by customer
Vector stores	Local Qdrant/Milvus/Chroma	Cloud-hosted	Managed by customer
Experiments	Manual	Automated iterations	Automated iterations
Evaluation	Basic	Full pipeline with metrics	Full pipeline with metrics
Data persistence	Local files	Cloud database	Full data residency control

MCP Servers

RAGOps Agent CE includes built-in MCP servers:

ragops-rag-planner

Plans RAG pipeline configuration based on requirements.

Tools:

• plan_rag_config — Generate RAG configuration from requirements

ragops-read-engine

Processes and converts documents from various formats.

Tools:

• process_documents — Convert PDF, DOCX, PPTX, XLSX, images to text/JSON/markdown/TOON

ragops-chunker

Chunks documents for vector storage.

Tools:

• chunk_documents — Split documents into chunks with configurable strategies
• list_chunked_files — List processed chunk files

ragops-vectorstore-loader

Loads chunks into vector databases.

Tools:

• vectorstore_load — Load documents into Qdrant, Chroma, or Milvus
• delete_from_vectorstore — Remove documents from vector store

ragops-compose-manager

Manages Docker Compose infrastructure.

Tools:

• init_project_compose — Initialize Docker Compose for project
• compose_up — Start services
• compose_down — Stop services
• compose_status — Check service status
• compose_logs — View service logs

ragops-rag-query

Executes RAG queries against deployed services.

Tools:

• search_documents — Search for relevant documents in vector database
• get_rag_prompt — Get formatted RAG prompt with retrieved context

rag-evaluation

Evaluates RAG pipeline performance with batch processing.

Tools:

• evaluate_batch — Run batch evaluation from CSV/JSON, compute Precision/Recall/Accuracy

donkit-ragops-mcp

Unified MCP server that combines all servers above into a single endpoint.

# Run unified server
donkit-ragops-mcp

Claude Desktop configuration:

{
  "mcpServers": {
    "donkit-ragops-mcp": {
      "command": "donkit-ragops-mcp"
    }
  }
}

All tools are available with prefixes:

• chunker_* — Document chunking
• compose_* — Docker Compose orchestration
• evaluation_* — RAG evaluation
• planner_* — RAG configuration planning
• query_* — RAG query execution
• reader_* — Document reading/parsing
• vectorstore_* — Vector store operations

Note: Checklist management is now handled by built-in agent tools, not MCP.

Examples

Basic RAG Pipeline

donkit-ragops

you> Create a RAG pipeline for customer support docs in ./docs folder

The agent will:

1. Create project structure
2. Plan RAG configuration
3. Chunk documents from ./docs
4. Set up Qdrant + RAG service
5. Load data into vector store

Custom Configuration

donkit-ragops -p vertexai -m gemini-2.5-pro

you> Build RAG for legal documents with 1000 token chunks and reranking

Multiple Projects

Each project gets its own:

• Project directory (projects/<project_id>)
• Docker Compose setup
• Vector store collection
• Configuration

Development

Prerequisites

• Python 3.12+
• Poetry for dependency management
• Docker Desktop (for testing vector stores and RAG services)

Setup

# Clone the repository
git clone https://github.com/donkit-ai/ragops.git
cd ragops/ragops-agent-cli

# Install dependencies
poetry install

# Activate virtual environment
poetry shell

Project Structure

ragops-agent-cli/
├── src/donkit_ragops/
│   ├── agent/              # LLM agent core and local tools
│   │   ├── agent.py        # Main LLMAgent class
│   │   ├── prompts.py      # System prompts for different providers
│   │   └── local_tools/    # Built-in agent tools
│   ├── llm/                # LLM provider integrations
│   │   └── providers/      # OpenAI, Vertex, Anthropic, etc.
│   ├── mcp/                # Model Context Protocol
│   │   ├── client.py       # MCP client implementation
│   │   └── servers/        # Built-in MCP servers
│   ├── repl/               # REPL implementation
│   │   ├── base.py         # Base REPL context
│   │   ├── local_repl.py   # Local mode REPL
│   │   └── enterprise_repl.py  # SaaS/Enterprise mode REPL
│   ├── web/                # Web UI (FastAPI + WebSocket)
│   │   ├── app.py          # FastAPI application
│   │   └── routes/         # API endpoints
│   ├── enterprise/         # SaaS/Enterprise mode components
│   ├── cli.py              # CLI entry point (Typer)
│   └── config.py           # Configuration management
├── tests/                  # Test suite (170+ tests)
└── pyproject.toml          # Poetry project configuration

Running the CLI Locally

# Run CLI
poetry run donkit-ragops

# Run with specific provider
poetry run donkit-ragops -p openai -m gpt-4o

# Run Web UI
poetry run donkit-ragops-web

# Run unified MCP server
poetry run donkit-ragops-mcp

Running Tests

# Run all tests
poetry run pytest

# Run with coverage
poetry run pytest --cov=donkit_ragops

# Run specific test file
poetry run pytest tests/test_agent.py

# Run specific test
poetry run pytest tests/test_agent.py::test_function_name -v

Code Quality

# Format code (REQUIRED before commit)
poetry run ruff format .

# Lint and auto-fix (REQUIRED before commit)
poetry run ruff check . --fix

# Check without fixing
poetry run ruff check .

Version Management

IMPORTANT: Version must be incremented in pyproject.toml for every PR:

# Check current version
grep "^version" pyproject.toml

# Increment version in pyproject.toml before committing
# patch: 0.4.5 → 0.4.6 (bug fixes)
# minor: 0.4.5 → 0.5.0 (new features)
# major: 0.4.5 → 1.0.0 (breaking changes)

Adding a New MCP Server

Step 1. Create server file in src/donkit_ragops/mcp/servers/:

from fastmcp import FastMCP
from pydantic import BaseModel, Field

server = FastMCP("my-server")

class MyToolArgs(BaseModel):
    param: str = Field(description="Parameter description")

@server.tool(name="my_tool", description="What the tool does")
async def my_tool(args: MyToolArgs) -> str:
    # Implementation
    return "result"

def main() -> None:
    server.run(transport="stdio")

Step 2. Add entry point in pyproject.toml:

[tool.poetry.scripts]
ragops-my-server = "donkit_ragops.mcp.servers.my_server:main"

Step 3. Mount in unified server (donkit_ragops_mcp.py):

from .my_server import server as my_server
unified_server.mount(my_server, prefix="my")

Adding a New LLM Provider

1. Create provider in src/donkit_ragops/llm/providers/
2. Register in provider_factory.py
3. Add configuration to config.py
4. Update supported_models.py

Debugging

# Enable debug logging
RAGOPS_LOG_LEVEL=DEBUG poetry run donkit-ragops

# Debug MCP servers
RAGOPS_LOG_LEVEL=DEBUG poetry run donkit-ragops-mcp

Docker Compose Services

The agent can deploy these services:

Qdrant (Vector Database)

services:
  qdrant:
    image: qdrant/qdrant:latest
    ports:
      - "6333:6333"
      - "6334:6334"

RAG Service

services:
  rag-service:
    image: donkit/rag-service:latest
    ports:
      - "8000:8000"
    environment:
      - DATABASE_URI=http://qdrant:6333
      - CONFIG=<base64-encoded-config>

Architecture

┌─────────────────┐
│  RAGOps Agent   │
│     (CLI)       │
└────────┬────────┘
         │
         ├── MCP Servers ───────────────┐
         │   ├── ragops-rag-planner     │
         │   ├── ragops-chunker         │
         │   ├── ragops-vectorstore     │
         │   └── ragops-compose         │
         │                              │
         └── LLM Providers ─────────────┤
             ├── Vertex AI              │
             ├── OpenAI                 │
             ├── Anthropic              │
             └── Ollama                 │
                                        │
                                        ▼
                            ┌──────────────────┐
                            │ Docker Compose   │
                            ├──────────────────┤
                            │ • Qdrant         │
                            │ • RAG Service    │
                            └──────────────────┘

Troubleshooting

Windows + Docker Desktop with WSL2

The agent fully supports Windows with Docker Desktop running in WSL2 mode. Path conversion and Docker communication are handled automatically.

Requirements:

• Docker Desktop for Windows with WSL2 backend enabled
• Python 3.12+ installed on Windows (not inside WSL2)
• Run the agent from Windows PowerShell or Command Prompt

How it works:

• The agent detects WSL2 Docker automatically
• Windows paths like C:\Users\... are converted to /mnt/c/Users/... for Docker
• No manual configuration needed

Troubleshooting:

# 1. Verify Docker is accessible from Windows
docker info

# 2. Check Docker reports Linux (indicates WSL2)
docker info --format "{{.OperatingSystem}}"
# Should output: Docker Desktop (or similar with "linux")

# 3. If Docker commands fail, ensure Docker Desktop is running

MCP Server Connection Issues

If MCP servers fail to start:

# Check MCP server logs
RAGOPS_LOG_LEVEL=DEBUG donkit-ragops

Vector Store Connection

Ensure Docker services are running:

cd projects/<project_id>
docker-compose ps
docker-compose logs qdrant

Credentials Issues

Verify your credentials:

# Vertex AI
gcloud auth application-default print-access-token

# OpenAI
echo $RAGOPS_OPENAI_API_KEY

License

This project is licensed under the MIT License - see the LICENSE file for details.

Related Projects

• donkit-chunker — Document chunking library
• donkit-vectorstore-loader — Vector store loading utilities
• donkit-read-engine — Document parsing engine

---

Built with ❤️ by Donkit AI