LLM API Server

A reusable Flask server providing an OpenAI-compatible API for local LLM backends (Ollama, LM Studio) with tool calling support.

Requirements: Python 3.11+

API Stability

This project follows Semantic Versioning.

Current status: Pre-1.0 development (version 0.x). The API is stable but may have occasional breaking changes as we refine the design.

From version 1.0 onwards:

• Stable API surface:

  • LLMServer class instantiation and run() method
  • ServerConfig class and from_env() factory
  • Built-in tools: BUILTIN_TOOLS, get_current_datetime, calculate, create_web_search_tool, create_doc_search_tool
  • RAG module: DocSearchIndex, RAGConfig, search() method
  • Evaluation framework: Evaluator, TestCase, HTMLReporter, JSONReporter
  • REST endpoints: /health, /v1/models, /v1/chat/completions

• Compatibility guarantees:

  • Minor versions (1.x) maintain backward compatibility
  • Breaking changes only in major versions with migration guides
  • Deprecated features announced at least one minor version before removal

• Support matrix:

  • Python: 3.11, 3.12
  • Backends: Ollama, LM Studio
  • Optional extras: webui, websearch, rag, eval, dev

Features

• OpenAI-compatible API - Drop-in replacement for OpenAI's /v1/chat/completions endpoint
• Multiple backends - Supports Ollama and LM Studio
• Tool calling - Full support for function calling with LangChain tools
• Streaming responses - Real-time token streaming
• RAG module - Local documentation search with hybrid retrieval, semantic chunking, and re-ranking
• Evaluation framework - Test and validate LLM responses with beautiful HTML reports
  • Full markdown-formatted responses (no truncation)
  • Collapsible long responses with syntax highlighting
  • Professional styling for code, tables, and lists
• Built-in tools - Date/time, calculator, and web search (via Ollama API)
• WebUI integration - Optional Open Web UI frontend
• Smart caching - System prompt auto-reload on file changes
• Debug logging - Comprehensive tool execution logging
• Graceful tool loop completion - When hitting iteration/timeout limits, generates a synthesized response

Installation

Using uv (recommended)

# Clone and install
git clone https://github.com/assareh/llm-tools-server.git
cd llm-tools-server
uv sync

# With optional dependencies
uv sync --extra webui      # For Open Web UI support
uv sync --extra websearch  # For web search tool
uv sync --extra rag        # For RAG document search module
uv sync --extra eval       # For HTML reports with markdown formatting
uv sync --extra dev        # For development tools
uv sync --all-extras       # Install everything

Using pip

pip install llm-tools-server

Or install from source:

git clone https://github.com/assareh/llm-tools-server.git
cd llm-tools-server
pip install -e .

Optional dependencies

# For Open Web UI support
pip install llm-tools-server[webui]

# For web search tool
pip install llm-tools-server[websearch]

# For RAG document search module
pip install llm-tools-server[rag]

# For HTML reports with markdown formatting
pip install llm-tools-server[eval]

# For development
pip install llm-tools-server[dev]

Quick Start

from llm_tools_server import LLMServer, ServerConfig
from langchain_core.tools import tool

# Define your tools
@tool
def get_weather(location: str) -> str:
    """Get weather for a location."""
    return f"Weather in {location}: Sunny, 72°F"

ALL_TOOLS = [get_weather]

# Configure server
config = ServerConfig.from_env("MYAPP_")  # Reads MYAPP_BACKEND, MYAPP_PORT, etc.
config.BACKEND_TYPE = "lmstudio"
config.BACKEND_MODEL = "openai/gpt-oss-20b"
config.MODEL_NAME = "myapp/assistant"
config.SYSTEM_PROMPT_PATH = "system_prompt.md"

# Create and run server
server = LLMServer(
    name="MyApp",
    model_name=config.MODEL_NAME,
    tools=ALL_TOOLS,
    config=config,
    default_system_prompt="You are a helpful assistant."
)

if __name__ == "__main__":
    server.run(port=8000)

Built-in Tools

LLM API Server includes common tools that you can use out of the box:

Using Built-in Tools

from llm_tools_server import LLMServer, BUILTIN_TOOLS, ServerConfig
from llm_tools_server import get_current_datetime, calculate, create_web_search_tool
from langchain_core.tools import tool

config = ServerConfig.from_env("MYAPP_")

# Option 1: Use all built-in tools
server = LLMServer(
    name="MyApp",
    model_name="myapp/assistant",
    tools=BUILTIN_TOOLS,
    config=config
)

# Option 2: Import specific tools
server = LLMServer(
    name="MyApp",
    model_name="myapp/assistant",
    tools=[get_current_datetime, calculate],
    config=config
)

# Option 3: Add web search tool (requires websearch extra)
web_search = create_web_search_tool(config)  # Uses config.OLLAMA_API_KEY
server = LLMServer(
    name="MyApp",
    model_name="myapp/assistant",
    tools=BUILTIN_TOOLS + [web_search],
    config=config
)

# Option 4: Combine built-in tools with custom tools
@tool
def get_weather(location: str) -> str:
    """Get weather for a location."""
    return f"Weather in {location}: Sunny, 72°F"

server = LLMServer(
    name="MyApp",
    model_name="myapp/assistant",
    tools=BUILTIN_TOOLS + [get_weather],  # Combine both
    config=config
)

Available Built-in Tools

• get_current_datetime() - Returns the current date and time in local timezone (e.g., "Wednesday, November 26, 2025 at 2:30 PM PST")
• calculate(expression: str) - Safely evaluates mathematical expressions
  • Supports: +, -, *, /, //, %, ** (power)
  • Example: calculate("2 + 3 * 4") → "14"
• create_web_search_tool(config) - Web search using Ollama API
  • Requires optional websearch dependency: uv sync --extra websearch
  • Requires OLLAMA_API_KEY to be configured
  • Parameters: query, max_results (default 5), site (optional filter)
• create_doc_search_tool(index, name, description) - Search local RAG index
  • Requires optional rag dependency: uv sync --extra rag
  • Parameters: query, top_k (default 5)

Sample System Prompt Content

When using built-in tools, include relevant instructions in your system prompt. Here are templates you can customize:

Basic Tools (datetime, calculate):

## Available Tools

You have access to the following tools:

- **get_current_datetime**: Get the current date and time. Use this when users ask about the current time, today's date, or time-sensitive questions.

- **calculate**: Evaluate mathematical expressions. Use for any arithmetic calculations. Supports +, -, *, /, // (floor division), % (modulo), ** (power), and parentheses.
  Examples: "2 + 3 * 4" → 14, "2 ** 8" → 256, "(10 + 5) / 3" → 5

With Web Search:

## Available Tools

- **web_search**: Search the web for current information. Use for:
  - Recent events or news
  - Current documentation or tutorials
  - Stack Overflow answers
  - Any information that may have changed since your training

  Parameters:
  - query: Search terms
  - max_results: Number of results (default 5)
  - site: Optional domain filter (e.g., "stackoverflow.com")

With Document Search (RAG):

## Available Tools

- **doc_search**: Search the indexed documentation. ALWAYS use this tool first when users ask questions about [YOUR PRODUCT/DOCS]. This searches locally indexed documentation and returns relevant excerpts with source URLs.

  Parameters:
  - query: What to search for
  - top_k: Number of results (default 5)

When answering questions:
1. First search the documentation using doc_search
2. If the documentation doesn't have the answer, say so clearly
3. Always cite your sources with the URLs provided

Complete Example (All Tools):

## Available Tools

You have access to these tools to help answer questions:

### get_current_datetime
Get the current date and time in local timezone.

### calculate
Safely evaluate mathematical expressions. Supports +, -, *, /, //, %, ** and parentheses.

### doc_search
Search [YOUR PRODUCT] documentation. Use this FIRST for any questions about [YOUR PRODUCT].

### web_search
Search the web for general information. Use for topics not covered in the documentation.

## Guidelines

1. For [YOUR PRODUCT] questions: Always search documentation first with doc_search
2. For calculations: Use the calculate tool instead of computing manually
3. For current events or external topics: Use web_search
4. Always cite sources when using search tools

RAG Module (Document Search)

LLM API Server includes a RAG (Retrieval-Augmented Generation) module for building local documentation search systems with hybrid retrieval and cross-encoder re-ranking.

Installation

# Install with RAG dependencies
uv sync --extra rag

Architecture Overview

The RAG module uses a four-component architecture:

1. DocumentCrawler (crawler.py) - URL discovery with sitemap/recursive/manual modes
2. Chunker (chunker.py) - Semantic HTML parsing with parent-child relationships
3. DocSearchIndex (indexer.py) - Main orchestrator for indexing and search
4. RAGConfig (config.py) - Comprehensive configuration with sensible defaults

Key Features

Intelligent Crawling:

• Three modes with smart fallback: Sitemap → Recursive → Manual
• Robots.txt compliance with intelligent parsing
• URL normalization (removes query params, anchors, trailing slashes)
• Include/exclude patterns with regex support
• Parallel fetching with configurable rate limiting

Semantic HTML Chunking:

• Token-aware processing using tiktoken (matches GPT tokenization)
• Preserves atomic content - code blocks and tables kept intact
• Parent-child hierarchy - focused retrieval with full context recovery
• Heading-based sectioning - respects document structure (h1-h6)
• Boilerplate removal - CSS selector-based filtering of nav, footer, etc.

Hybrid Search:

• BM25 keyword search - captures exact term matches
• Semantic vector search - captures conceptual similarity
• Reciprocal Rank Fusion (RRF) - combines rankings from both retrievers
• Configurable weights - tune keyword vs semantic emphasis in RRF formula

Cross-Encoder Re-ranking:

• MS MARCO cross-encoder for accurate final ranking
• Min-max score normalization for consistent ranking
• Configurable candidate pool (rerank_top_k)

Production-Ready Features:

• Incremental indexing - resume interrupted crawls, add new docs efficiently
• SHA256 checksums - index integrity verification
• Content deduplication - by hash before indexing
• GPU auto-detection - MPS (Apple Silicon), CUDA, or CPU fallback
• Rich metadata - heading path, section ID, document type, code identifiers

Local-First Design:

• FAISS for vector storage (no cloud dependencies)
• HuggingFace embeddings - configurable from fast (MiniLM) to accurate (BGE-large)
• Everything runs locally - no API keys required for core functionality

Contextual Retrieval (Optional):

• LLM-generated context - Anthropic's approach for ~40-50% better retrieval
• Background processing - index usable immediately while contexts generate
• Resumable - progress saved every 50 chunks, can interrupt and resume
• Uses server backend - same LM Studio/Ollama configured for requests

Quick Start

from llm_tools_server.rag import DocSearchIndex, RAGConfig

# Configure RAG
config = RAGConfig(
    base_url="https://docs.example.com",
    cache_dir="./doc_index",
    # Optional: manual URLs (additive or exclusive)
    manual_urls=["https://docs.example.com/important-page"],
    manual_urls_only=False,  # False = add to crawled URLs, True = only index these
)

# Build index (first time)
index = DocSearchIndex(config)
index.crawl_and_index()

# Search
results = index.search("How do I configure authentication?", top_k=5)

for result in results:
    print(f"Score: {result['score']:.3f}")
    print(f"URL: {result['url']}")
    print(f"Section: {result['heading_path']}")
    print(f"Text: {result['text'][:200]}...")
    if 'parent_text' in result:
        print(f"Parent context: {result['parent_text'][:200]}...")
    print()

Configuration Options

config = RAGConfig(
    base_url="https://docs.example.com",
    cache_dir="./doc_index",

    # Crawling settings
    manual_urls=["https://..."],           # Optional list of specific URLs
    manual_urls_only=False,                # True = only index manual URLs
    max_crawl_depth=3,                     # Maximum recursion depth
    rate_limit_delay=0.1,                  # Seconds between requests
    max_workers=5,                         # Parallel fetching threads
    max_pages=None,                        # Limit total pages (None = unlimited)
    request_timeout=10.0,                  # HTTP request timeout in seconds
    max_url_retries=3,                     # Skip URLs after N consecutive failures
    url_include_patterns=["docs/.*"],      # Regex patterns to include
    url_exclude_patterns=[".*/api/.*"],    # Regex patterns to exclude

    # Chunking settings
    child_chunk_size=350,                  # Tokens per child chunk
    child_chunk_overlap=50,                # Overlap tokens between child chunks
    parent_chunk_size=900,                 # Tokens per parent chunk
    parent_chunk_overlap=100,              # Overlap tokens between parent chunks

    # Search settings
    hybrid_bm25_weight=0.3,                # BM25 keyword search weight
    hybrid_semantic_weight=0.7,            # Semantic vector search weight
    search_top_k=5,                        # Default results to return
    rerank_enabled=True,                   # Enable cross-encoder re-ranking
    rerank_top_k=80,                       # Candidates for re-ranking

    # Model settings (embedding model: speed vs quality tradeoff)
    # Options:
    #   - "sentence-transformers/all-MiniLM-L6-v2": Fast (22M params), good quality (default)
    #   - "BAAI/bge-base-en-v1.5": Medium (110M params), better quality
    #   - "BAAI/bge-large-en-v1.5": Slow (335M params), best quality
    # Note: Changing embedding model requires full index rebuild
    embedding_model="sentence-transformers/all-MiniLM-L6-v2",
    rerank_model="cross-encoder/ms-marco-MiniLM-L-12-v2",  # Cross-encoder for re-ranking

    # Contextual retrieval settings (optional, requires server_config)
    contextual_retrieval_enabled=False,    # Enable LLM-generated context for chunks
    contextual_retrieval_background=False, # Run in background thread (index usable immediately)
    contextual_max_workers=4,              # Parallel context generation workers

    # Index settings
    update_check_interval_hours=168,       # Check for updates (7 days)
    page_cache_ttl_hours=168,              # TTL for cached pages without lastmod (7 days, 0 = never)

    # Progress display
    show_progress=True,                    # Show tqdm progress bars
)

Crawling Modes

The RAG module automatically selects the best crawling strategy:

1. Default behavior: Tries to discover sitemap.xml → uses if found
2. Fallback: If no sitemap → recursive crawling from base_url
3. Manual mode: manual_urls_only=True → only index specified URLs

Incremental Updates

# Check if update needed
if index.needs_update():
    print("Index is stale, rebuilding...")
    index.crawl_and_index()
else:
    print("Index is up-to-date")
    index.load_index()

# Force rebuild (clears all state, re-crawls everything)
index.crawl_and_index(force_rebuild=True)

# Force refresh (refetches all cached pages, bypasses TTL)
index.crawl_and_index(force_refresh=True)

Cache Invalidation

Pages are automatically refreshed based on:

1. Sitemap lastmod - Pages with lastmod in sitemap are refetched when the date changes
2. TTL expiration - Pages without lastmod expire after page_cache_ttl_hours (default: 7 days)
3. Manual refresh - Use force_refresh=True to bypass all caching

Advanced: Hybrid Search Weights

Adjust the balance between keyword and semantic search using Reciprocal Rank Fusion (RRF). The weights scale each retriever's contribution to the RRF score: score += weight / (rank + 60).

# More keyword-focused (good for technical docs with specific terms)
config = RAGConfig(
    base_url="https://docs.example.com",
    hybrid_bm25_weight=0.5,      # Equal weight to BM25 ranks
    hybrid_semantic_weight=0.5,  # Equal weight to semantic ranks
)

# More semantic-focused (good for concept-based queries)
config = RAGConfig(
    base_url="https://docs.example.com",
    hybrid_bm25_weight=0.2,      # Lower influence from BM25 ranks
    hybrid_semantic_weight=0.8,  # Higher influence from semantic ranks
)

Advanced: Contextual Retrieval

Contextual retrieval prepends LLM-generated context to each chunk before embedding, improving retrieval accuracy by ~40-50% (Anthropic's approach).

from llm_tools_server import ServerConfig
from llm_tools_server.rag import DocSearchIndex, RAGConfig

# Get server config for backend settings
server_config = ServerConfig.from_env()

# Enable contextual retrieval
rag_config = RAGConfig(
    base_url="https://docs.example.com",
    contextual_retrieval_enabled=True,      # Enable context generation
    contextual_retrieval_background=True,   # Run in background (recommended)
)

# Pass server_config to use same backend for context generation
index = DocSearchIndex(rag_config, server_config)
index.crawl_and_index()
index.load_index()  # Starts background contextualization if enabled

Or run contextual retrieval explicitly (can be interrupted and resumed):

index.crawl_and_index()              # Build index first (fast)
index.add_contextual_retrieval()     # Run separately, saves progress every 50 chunks

Reference: https://www.anthropic.com/news/contextual-retrieval

Advanced: Embedding Models

Change embedding model without re-crawling:

config = RAGConfig(
    base_url="https://docs.example.com",
    embedding_model="BAAI/bge-large-en-v1.5",  # Upgrade to better model
)
index = DocSearchIndex(config)
index.crawl_and_index()  # Detects model change, rebuilds embeddings only

Or explicitly rebuild embeddings:

index.rebuild_embeddings()  # Uses saved chunks, skips crawling

Configuration

Using ServerConfig

from llm_tools_server import ServerConfig

# Create from environment variables
config = ServerConfig.from_env("MYAPP_")

# Or configure directly
config = ServerConfig()
config.BACKEND_TYPE = "ollama"  # or "lmstudio"
config.BACKEND_MODEL = "openai/gpt-oss-20b"
config.OLLAMA_ENDPOINT = "http://localhost:11434"
config.DEFAULT_PORT = 8000
config.DEFAULT_TEMPERATURE = 0.0

Environment Variables

With prefix MYAPP_:

• MYAPP_BACKEND - Backend type (ollama, lmstudio)
• MYAPP_BACKEND_MODEL - Model identifier
• MYAPP_PORT - Server port (default: 8000)
• MYAPP_TEMPERATURE - Default temperature (default: 0.0)
• MYAPP_SYSTEM_PROMPT_PATH - Path to system prompt file
• MYAPP_DEBUG_TOOLS - Enable tool debug logging (true/false)
• MYAPP_MAX_TOOL_ITERATIONS - Maximum tool loop iterations (default: 5)
• MYAPP_TOOL_LOOP_TIMEOUT - Maximum seconds for tool loop (default: 120, 0 = no timeout)
• OLLAMA_ENDPOINT - Ollama API endpoint
• OLLAMA_API_KEY - Ollama API key for web search (optional)
• LMSTUDIO_ENDPOINT - LM Studio API endpoint

API Endpoints

GET /health

Health check endpoint.

curl http://localhost:8000/health

GET /v1/models

List available models.

curl http://localhost:8000/v1/models

POST /v1/chat/completions

OpenAI-compatible chat completions endpoint.

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "myapp/assistant",
    "messages": [{"role": "user", "content": "What is the weather in Paris?"}],
    "stream": false
  }'

Per-Request Model Override: You can specify a different backend model per request by setting the model field to a model name different from the server's configured model_name. This passes through to LM Studio/Ollama, allowing dynamic model selection:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5:7b",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Evaluation Framework

LLM API Server includes a comprehensive evaluation framework for testing your LLM applications.

Quick Example

from llm_tools_server.eval import Evaluator, TestCase, HTMLReporter

# Define test cases
tests = [
    TestCase(
        question="What is 2+2?",
        description="Basic arithmetic test",
        expected_keywords=["4", "four"],
        min_response_length=10
    ),
    TestCase(
        question="What are the best travel credit cards?",
        description="Domain knowledge test",
        expected_keywords=["chase", "sapphire", "travel"],
        min_response_length=100
    )
]

# Run evaluation
evaluator = Evaluator(api_url="http://localhost:8000")
results = evaluator.run_tests(tests)

# Generate HTML report
reporter = HTMLReporter()
reporter.generate(results, "evaluation_report.html")

# Print summary
summary = evaluator.get_summary(results)
print(f"Success Rate: {summary['success_rate']:.1f}%")

Features

• Flexible validation - Expected keywords, response length, custom validators
• Beautiful HTML reports - Markdown-formatted responses with syntax highlighting
  • Full responses (no truncation)
  • Collapsible long responses with expand/collapse buttons
  • Code blocks with syntax highlighting
  • Tables, lists, and blockquote formatting
  • Requires optional eval extra: uv sync --extra eval
• Multiple report formats - HTML, JSON, and console output
• Performance tracking - Response times and success rates
• Custom validators - Write domain-specific validation logic
• CI/CD ready - JSON reports and exit codes for automation

Documentation

See llm_tools_server/eval/README.md for complete documentation and examples.

Run the example script:

python example_evaluation.py

Advanced Usage

Custom Initialization Hook

def init_database():
    print("Initializing database...")
    # Your initialization code here

server = LLMServer(
    name="MyApp",
    model_name="myapp/assistant",
    tools=ALL_TOOLS,
    config=config,
    init_hook=init_database  # Called before server starts
)

Custom Logger Names

server = LLMServer(
    name="MyApp",
    model_name="myapp/assistant",
    tools=ALL_TOOLS,
    config=config,
    logger_names=["myapp.tools", "myapp.backend", "tools"]
)

System Prompt Auto-Reload

Create a system_prompt.md file:

You are MyApp, an AI assistant specialized in...

The server automatically reloads this file when it changes (based on modification time).

Development

Using uv (recommended)

# Clone and install with dev dependencies
git clone https://github.com/assareh/llm-tools-server.git
cd llm-tools-server
uv sync --extra dev

# Run tests
uv run pytest

# Format and lint
./lint.sh

# Or manually
uv run black llm_tools_server/
uv run ruff check --fix llm_tools_server/

Using pip

# Clone the repository
git clone https://github.com/assareh/llm-tools-server.git
cd llm-tools-server

# Install in development mode
pip install -e ".[dev]"

# Run tests
pytest

# Format code
black llm_tools_server/

# Lint
ruff check --fix llm_tools_server/

License

MIT License - see LICENSE file for details.

Acknowledgments

Built with:

• Flask
• LangChain
• Open Web UI

Support

If you find this project helpful, consider supporting its development:

Your support helps maintain and improve this project!