llm-aggregator 0.1.7

A model aggregator service for multiple LLM backends.

LLM Aggregator

LLM Aggregator keeps a live list of every model exposed by your local OpenAI-compatible servers.

---

Features

• Polls models from configured LLM provider servers (/v1/models).
• Enriches model information with a helper LLM.
• Optionally hands model information from external websites to helper LLM.
• Ships with a minimal UI showing providers, models, and host RAM.
• The builtin UI can easily be replaced.

---

Web Interface

The builtin UI shows a single table plus a small RAM widget, so you immediately see what is running:

Model	Base URL	Types	Family	Context	Quant	Params	Summary
llama3.1:8b	http://10.7.2.100:11434/v1	llm	Llama 3.1	8K	Q4_K_M	8B	General chat tuned for balance
qwen2.5:14b	http://10.7.2.100:8080/v1	llm,embed	Qwen 2.5	32K	Q5_0	14B	Multilingual reasoning focused

Columns:

• Model – identifier reported by the provider.
• Base URL – where the model is served.
• Types – capabilities (LLM, VLM, embedder, etc.).
• Family – base architecture inferred by the helper LLM.
• Context – approximate context window in tokens.
• Quant – quantization hinted by the model name or docs.
• Params – estimated parameter count.
• Summary – one-line description generated by the helper LLM.

---

Installation

Prerequisites

• Python 3.10 or higher
• LLM servers (Ollama, llama.cpp, nexa, etc.) with OpenAI-compatible APIs

Install from PyPI

pip install llm-aggregator

---

Usage

Set the LLM_AGGREGATOR_CONFIG environment variable to point at your config.yaml and the service will load it on startup.

Starting the Service

export LLM_AGGREGATOR_CONFIG=/path/to/config.yaml
llm-aggregator

Or run directly:

export LLM_AGGREGATOR_CONFIG=/path/to/config.yaml
python -m llm_aggregator

By default, the web interface will be available at http://localhost:8888.

---

Configuration

All runtime behavior is controlled through the YAML file pointed to by the LLM_AGGREGATOR_CONFIG environment variable. Use config.yaml as a reference template.

UI modes

Use static_enabled and custom_static_path to set one of three modes:

• static_enabled: true (default) serves the built-in UI.
• static_enabled: true and custom_static_path: /path/to/dir serves your files instead of the built-in UI.
• static_enabled: false serves no UI at all. Provide your own UI using the REST endpoints.

Configuration Options

• host / port – Where the FastAPI server and static frontend bind.
• log_level – Logging verbosity (DEBUG, INFO, WARNING, ERROR, CRITICAL). Defaults to INFO if omitted.
• log_format – Optional logging format string. When omitted the service leaves existing logging configuration untouched.
• logger_overrides – Map of logger names to override their logging level (e.g., httpx: WARNING).
• brain – Settings for the enrichment LLM:
  • base_url – HTTP endpoint of the enrichment provider.
  • id – Model identifier passed to the provider.
  • api_key – Optional API-Key.
  • max_batch_size – Number of models to enrich at once (defaults to 1).
• providers – Map of provider name to an OpenAI-compatible backend to query:
  • base_url – Public URL returned via the REST API.
  • internal_base_url – Optional internal URL used for server-to-server calls; defaults to base_url when omitted.
  • api_key – Optional API-Key for that provider.
  • files_size_gatherer – Optional block to report on-disk model size:
    • path – Script or executable invoked as <path> <base_path> <full_model_name>.
    • base_path – Filesystem root passed to the script.
    • timeout_seconds – Optional per-provider timeout (default: 15s).
• model_info_sources – Optional external websites where model information is fetched from for enrichment. Each entry requires a human-readable name (shown to the LLM) and a url_template that contains {model_id}.
• time – Background scheduling knobs (all in seconds):
  • fetch_models_interval
  • fetch_models_timeout
  • enrich_models_timeout
  • enrich_idle_sleep
  • website_markdown_cache_ttl – TTL for cached markdown scraped from external sources.
• ui – Optional static UI:
  • static_enabled – true: static web frontend is served at /index.html and assets at /static.
  • custom_static_path – Optional directory to replace the bundled UI; must contain a readable index.html and asset files.
• brain_prompts – LLM instructions kept separate so the block can live at the end of the YAML:
  • system – System message injected ahead of every enrichment request.
  • user – Main user instruction describing the enrichment JSON contract.
  • model_info_prefix_template – Optional prefix template applied to fetched markdown snippets; receives {model_id} and {provider_label} placeholders.

---

REST API

• GET /v1/models – OpenAI ListModelsResponse plus a meta object on each data item with the enriched metadata. Example:

  {
    "object": "list",
    "data": [
      {
        "id": "llama3.1:8b",
        "object": "model",
        "created": 1,
        "owned_by": "ollama",
        "meta": {
          "base_url": "http://127.0.0.1:11434/v1",
          "types": ["llm"],
          "model_family": "Llama 3.1",
          "context_size": "8K",
          "quant": "Q4_K_M",
          "param": "8B",
          "size": 481406976,
          "summary": "General chat tuned for balance"
        }
      }
    ]
  }
  

• GET /api/stats – Returns an array of recent RAM usage percentages sampled for the Chart.js widget in the UI

  [57.5,57.6,57.6]
  

• POST /api/clear – Empty request; clears model cache and restarts model information collection.