radlab-llm-router-plugins 0.1.4

Overview

The LLM‑Router project ships with a modular plugin system that lets you plug‑in anonymizers (also called maskers) and guardrails into request‑processing pipelines.
Each plugin implements a tiny, well‑defined interface (apply) and can be composed in an ordered list to form a * pipeline*. Pipelines are instantiated by the MaskerPipeline and GuardrailPipeline classes and are driven automatically by the endpoint logic in endpoint_i.py.

---

1. Anonymizers (Maskers)

1.1 What they do

• Goal – Remove or replace personally‑identifiable information (PII) from a payload before it reaches the LLM or an external service.
• Typical strategy – Run a pipeline of maskers that locate spans corresponding to IDs, emails, IPs, etc., and replace each span with a placeholder such as {{MASKED_ITEM}}.

1.2 Built‑in anonymizer plugins

Plugin	Description	Technical notes
FastMaskerPlugin (fast_masker_plugin.py)	Thin wrapper around the FastMasker utility class. Receives a JSON‑compatible payload and returns the same payload with all detected PII masked.	Implements PluginInterface. The heavy lifting is delegated to FastMasker.mask_payload(payload). No extra I/O; the FastMasker instance is created once in __init__.

1.3 How a masker is used

1. The endpoint (e.g. EndpointI._do_masking_if_needed) checks the global flag FORCE_MASKING.
2. If enabled, it creates a MaskerPipeline with the list of masker plugin identifiers (e.g. ["fast_masker"]).
3. The pipeline calls each plugin’s apply method sequentially, feeding the output of one as the input of the next.
4. The final payload – now stripped of PII – proceeds to the rest of the request flow (guardrails, model dispatch, etc.).

---

2. Guardrails

2.1 What they do

• Goal – Verify that a request (or its response) complies with policy rules (e.g. no hateful, illegal, or unsafe content).
• Typical strategy – Split the payload into manageable text chunks, run a pipeline of guardrails, aggregate per‑chunk scores, and decide whether the overall request is safe.

2.2 Built‑in guardrail plugins

Plugin	Description	Technical notes
NASKGuardPlugin (nask_guard_plugin.py)	HTTP‑based guardrail that forwards the payload to the external NASK guardrail service (/nask_guard endpoint) and returns a boolean safe flag together with the raw response.	Inherits from HttpPluginInterface. The apply method calls _request(payload) (provided by the base class) and extracts results["safe"]. Errors are caught and logged; on failure the plugin returns (False, {}).
SojkaGuardPlugin (sojka_guard_plugin.py)	HTTP‑based guardrail that forwards the payload to the Sójka guardrail service (/sojka_guard endpoint) and returns a safety flag.	Mirrors the design of NASKGuardPlugin. The endpoint_url is built from the LLM_ROUTER_GUARDRAIL_SOJKA_GUARD_HOST environment variable. On success it returns (True, response), otherwise (False, {}).
(Implicit) GuardrailProcessor (processor.py)	Core logic used by the internal NASK guardrail Flask route (nask_guardrail). Tokenises the payload, creates overlapping chunks, runs a Hugging‑Face text‑classification pipeline, and produces a detailed safety report.	Handles model loading (AutoTokenizer, pipeline("text‑classification")), chunking (_chunk_text), and scoring thresholds (MIN_SCORE_FOR_SAFE, MIN_SCORE_FOR_NOT_SAFE). Returns a dict: {"safe": <bool>, "detailed": [...]}.

2.3 How a guardrail is used

1. The endpoint calls _is_request_guardrail_safe(payload) (or the analogous response guardrail).
2. If FORCE_GUARDRAIL_REQUEST is true, a GuardrailPipeline is built from the configured plugin IDs (e.g. ["nask_guard", "sojka_guard"]).
3. The pipeline iterates over each guardrail plugin; each apply returns (is_safe, message).
4. The first plugin that reports is_safe=False short‑circuits the pipeline and the request is rejected with a 400/500 error payload.

---

2.5 ML-Based PII Classification

For cases where regex patterns alone are insufficient (e.g. context-dependent PII detection), the project integrates with the anonymizer-model repository:

Repository: radlab-dev-group/anonymizer-model

Feature	Description
Approach	NER (Named Entity Recognition) model based on Hugging Face AutoModelForTokenClassification
What it does	Identifies PII entities in Polish text with context-aware detection (not just pattern matching)
Training pipeline	Configurable via JSON, logs to W&B, exports best model (F1 macro) to final_model/
REST API	Flask service supporting multiple model versions with optional dynamic quantization
CLI	pii-classifier convert / generalise / report for data prep and analysis
Inference	Sub-token merging, punctuation cleaning, gap preservation for human-readable entity spans
Web tester	HTML/JS interface for real-time PII detection testing

How it complements regex maskers

Approach	Strength	Best for
Regex maskers (FastMasker)	Deterministic, zero false negatives for known formats, fast	Structured IDs (KRS, NIP, PESEL, NRB, REGON, VIN, etc.)
PII classifier (anonymizer-model)	Context-aware, handles unknown formats, generalizes across domains	Free-text entities, names, addresses, context-dependent PII

Quick start

# Clone and install
git clone https://github.com/radlab-dev-group/anonymizer-model.git
cd anonymizer-model
pip install .

# Run the API (serves multiple model versions)
python3 -m pii_classification.api.app

# API endpoints
#   GET  /models   — list available models + default
#   POST /predict   — { "text": "...", "model": "optional_name" }

---

2.6 Polish Identification Regex Patterns

The FastMaskerPlugin ships with a comprehensive set of rules for detecting Polish business and personal identifiers. These rules use regex matching + checksum/form validation to minimize false positives.

Available Polish rules

Rule	Placeholder	Format	Validation
KrsRule	{{KRS}}	1234567890 or 123-456-78-90 / 123 456 78 90	10 digits (format-only)
NrbRule	{{NRB}}	PL58105012981000009062923173 (26 digits)	26 digits
NipRule	{{NIP}}	1234567890 or 123-456-78-90	Checksum with weights [6,5,7,2,3,4,5,6,7] mod 11
PeselRule	{{PESEL}}	11 digits	Checksum with weights [1,3,7,9,1,3,7,9,1,3] mod 10
RegonRule	{{REGON}}	9 or 14 digits	Checksum (different weights per form)
BankAccountRule	{{BANK_ACCOUNT}}	PL58 1050 1298 1000 0090 6292 3173	Polish IBAN (28 chars), supports masked XX groups

How they work

Each rule follows the same pattern:

1. Regex match — detects candidate strings (e.g. 10-digit sequences for KRS)
2. Checksum validation — validates the candidate against the official algorithm
3. Placeholder replacement — valid matches are replaced with {{PLACEHOLDER}}; invalid ones are left untouched
4. Optional anonymizer — if an anonymizer_fn is provided, it's called with (original, tag_type) and its result ( wrapped in {}) is used instead

Adding rules to the pipeline

To enable Polish rules in your masking pipeline, add their plugin IDs to MASKING_STRATEGY_PIPELINE:

MASKING_STRATEGY_PIPELINE = ["fast_masker"]

All Polish rules are included in the FastMasker plugin by default. For a full list of available rules, see the fast_masker README.

---

2.7 Semantic Routing (Model Selection)

Four routing plugins are available for model selection. The two semantic plugins (simple_semantic_routing, semantic_biencoder_routing) activate when payload["model"] == "auto"; the Codex plugin (agentic_routing_codex) activates on "auto_codex", routing the OpenAI-Responses-style requests emitted by the Codex CLI agent by request class and work mode. The Claude Code plugin (agentic_routing_claude_code) decides by model name alone, mapping the model IDs Claude Code asks for onto the models this router serves.

Sitting behind them, the agentic knowledge layer (2.7.5) is not a classifier: it injects repository knowledge into an already-routed request and can escalate its model when the agent is looping with no known fix.

2.7.1 Simple Semantic Routing (Heuristic)

The Simple Semantic Routing plugin (simple_semantic_routing, utils/routing/simple_semantic/) activates on payload["model"] == "auto" and rewrites it in three dependency-free stages: intent scoring over the last user message — weighted keywords (case-insensitive substrings), phrases with a :weight suffix and regex patterns (+3.0 each) — giving code / math / creative / general, or none when nothing scores; complexity from a word-count estimate (int(words * 1.25)) against the simple / medium thresholds; and model selection from an ordered pool where index 0 is the cheapest and index n-1 the strongest. intent_adjustment only ever escalates the intents that declare one; an intent with an empty adjustment is demoted a single pool step. Only payload["model"] is written, so the decision lives in the log: SimpleSemanticRouting: intent=code, complexity=simple (10 tokens) -> qwen3.6:35b.

Configuration ships in simple_semantic.json. Overrides: LLM_ROUTER_ROUTING_COMPLEXITY_THRESHOLDS (simple|medium), LLM_ROUTER_ROUTING_MODELS (the pool), LLM_ROUTER_ROUTING_DEFAULT_MODEL and LLM_ROUTER_ROUTING_INTENT_<name> — that last one replaces an intent and clears its patterns and weights, and there is no ..._CONFIG variable for this plugin.

Full documentation — stage-by-stage mechanics, measured intent × complexity → model matrices for two- and three-model pools, configuration reference, tuning, gotchas and a runnable verification snippet: Simple Semantic Routing README.

2.7.2 Bi-Encoder Semantic Routing (Model Selection)

The Bi-Encoder routing plugin (semantic_biencoder_routing, utils/routing/semantic_biencoder/) also answers payload["model"] == "auto", but decides by embedding similarity. Each routing target is indexed at startup as "Target: {name}. {description}" plus its examples, split into overlapping token chunks, embedded by a sentence-transformers BiEncoder (google/embeddinggemma-300m by default), L2-normalized and stored in a faiss.IndexFlatIP; a docstore maps doc IDs back to target names. At request time the last user message is embedded (text longer than the model's max_seq_length is windowed and averaged, capped at MAX_QUERY_WINDOWS = 4), the top_k hits are averaged per target, and the winning target's model_name is selected when its cosine reaches similarity_threshold; the payload gains routing.target_name and routing.similarity. The index is persisted as index.faiss + docstore.pkl and reloaded on the next start, rebuilt automatically when the embedding model outputs a different dimension. Needs the [ml] / [ml-gpu] extras — and it fails hard rather than passing traffic through.

Configuration ships in semantic_biencoder.json. Overrides: LLM_ROUTER_ROUTING_SEMANTIC_BIENCODER_CONFIG (inline JSON or a path), _MODEL, _TARGETS, _CHUNK_SIZE, _CHUNK_OVERLAP, _PERSIST_DIR; similarity_threshold and top_k are config-only. The shipped similarity_threshold: 0.0 reroutes every request — measured on the shipped targets, real matches score 0.70–0.82 while unrelated or nonsense text still scores ≈0.35–0.37, so 0.5–0.6 is the usual production band.

Full documentation — startup and query mechanics, measured similarities per target, configuration reference, failure modes, tuning, gotchas (persisted-index staleness included) and a verification recipe: Bi-Encoder Routing README.

2.7.3 Codex CLI routing (auto_codex)

The Codex Routing plugin (agentic_routing_codex, utils/routing/agentic_routing/codex/) routes the OpenAI-Responses-style requests emitted by the Codex CLI coding agent. It activates only when payload["model"] equals its configured trigger — "auto_codex" by default — so it never intercepts the "auto" traffic owned by the two semantic plugins. It rewrites payload["model"] and adds payload["agent_mode"] plus a payload["routing"] block, and nothing else; routing fails open, so a payload it cannot classify is passed through untouched.

A work mode is resolved by a deterministic cascade — an explicit agent_mode override, the request class (compaction, aux_title), the <collaboration_mode> Plan block injected by the CLI, PL+EN keyword scoring, then an optional embedding cosine-similarity lookup — and falls back to implement:

Mode	Model	Routed by
plan	qwen/Qwen3.8-Flash-Next	<collaboration_mode> Plan Mode block
implement	qwen/Qwen3.8-Flash-Next	Fallback for a plain main turn
test	qwen/Qwen3.8-27B	Keywords in the classified user text
git_review	qwen/Qwen3.8-27B	Keywords in the classified user text
review	qwen/Qwen3.8-Flash-Next	Keywords in the classified user text
debug	qwen/Qwen3.8-Flash-Next	Keywords in the classified user text
aux_title	qwen/Qwen3.8-27B	Request class — system thread or title shape
compaction	qwen/Qwen3.8-27B	Request class — context compaction

Modes ship in agentic_routing_codex.json and are overridable with LLM_ROUTER_ROUTING_SEMANTIC_AGENTIC_CODEX_* environment variables (…_CONFIG, …_TRIGGER, …_MODEL_<MODE>, …_MODELS, …_MODES, …_FALLBACK_MODE, …_HEURISTIC_ENABLED, …_HEURISTIC_MIN_SCORE, …_CLASSIFY_MAX_CHARS, …_MODEL, …_SEMANTIC_ENABLED, …_SIMILARITY_THRESHOLD, …_TOP_K, …_CHUNK_SIZE, …_CHUNK_OVERLAP, …_PERSIST_DIR, …_MODE_<name>_KEYWORDS). MODES filters the mode list but does not move the fallback: if the whitelist excludes the configured fallback_mode (implement), set FALLBACK_MODE as well — otherwise configuration validation fails at startup.

Full documentation — the cascade and scoring model, installation, loading the plugin into the router, model selection and its pitfalls, tuning, a verification recipe and a troubleshooting table — lives in the Codex CLI Routing README.

Calibration and evaluation guides live in the same llm_router_plugins/utils/routing/agentic_routing/codex/ directory: Polski — CODEX_EVAL_HOWTO_PL.md and English — CODEX_EVAL_HOWTO_EN.md. They explain configuration roles, heuristic scores and embedding similarity, threshold/margin tuning, calibration/holdout datasets, the evaluation CLI and report metrics, and deployment/index rebuilding. They also document the automated scripts/codex-tune-eval.sh workflow; quality is evaluated by work mode, independently of model assignments.

2.7.4 Claude Code Model Swap (claude-* → configured models)

The Claude Code Model Swap plugin (agentic_routing_claude_code, utils/routing/agentic_routing/claude_code/) rewrites the model Claude Code asks for into the model the operator serves. Claude Code resolves its fable / opus / sonnet / haiku aliases client-side and sends a versioned ID, so pointing it at your own backend means overriding one environment variable per tier (ANTHROPIC_DEFAULT_FABLE_MODEL, ANTHROPIC_DEFAULT_OPUS_MODEL, ANTHROPIC_DEFAULT_SONNET_MODEL, ANTHROPIC_DEFAULT_HAIKU_MODEL, CLAUDE_CODE_SUBAGENT_MODEL) on every host. This plugin moves that mapping into one config file: a tier names the model IDs it answers and the model to serve them with, and a matching request gets the target written into payload["model"] (and payload["model_name"], when the payload carries it) plus a payload["routing"] block.

Tier	Model IDs Claude Code sends	Setting it replaces
fable	claude-fable-5-1, claude-fable-5, best	ANTHROPIC_DEFAULT_FABLE_MODEL
opus	claude-opus-5-5 … claude-opus-4-6	ANTHROPIC_DEFAULT_OPUS_MODEL
plan	opusplan	the model: "opusplan" phase
sonnet	claude-sonnet-5, claude-sonnet-4-6, -4-5	ANTHROPIC_DEFAULT_SONNET_MODEL
haiku	claude-haiku-4-5, claude-3-5-haiku-*	ANTHROPIC_DEFAULT_HAIKU_MODEL

Matching is content-free and deterministic: the requested name is normalized (window suffix [1m], provider version :0, @YYYYMMDD revision, us.anthropic. / anthropic/ prefix, models/ path and -YYYYMMDD release date stripped) and resolved through four layers, most specific first — literal, normalized exact, longest wildcard prefix (claude-opus-*), then family token (claude-3-5-haiku-20241022 → Haiku). Configuration ships in agentic_routing_claude_code.json and is overridable with LLM_ROUTER_ROUTING_SEMANTIC_AGENTIC_CLAUDE_CODE_* (…_CONFIG, …_ENABLED, …_MATCH_FAMILIES, …_FIELDS, …_MODEL_<TIER>, …_MODELS, …_MODES, …_MODE_<tier>_MODELS). Routing fails open and silent: a model no tier claims comes back as the same object it arrived as, with no log line, while an inconsistent config (duplicate tier names, an embedded or bare wildcard, one exact name claimed twice) raises at startup. No ML dependencies.

Full documentation — the normalization and match layers, installation, router and CLI wiring, tier configuration, linting, gotchas, a verification recipe and a troubleshooting table — lives in the Claude Code Model Swap README.

2.7.5 Agentic knowledge layer (agentic_knowledge)

The Agentic knowledge layer (agentic_knowledge, utils/knowledge/agentic_knowledge/) stands behind the Codex routing plugin and enriches a request with what the repository has already learned, without the agent cooperating. It reads the routing decision (routing, agent_mode, model), searches the Semantic Search Engine for verified fixes, dead ends and the pre-compaction archive of the current scope, and injects the result as a separate, removable context block. It also learns from the traffic: the miner turns a test_failure → patch → test pass episode into a recipe, a patch → test pass episode that no failure opened into a playbook — the correct steps of a task are knowledge too, and without this kind a turn that went right from the first try wrote nothing at all — and a repeated failure into a deadend, publishes them to a Redis Stream, and an indexing worker masks them with FastMasker before indexing; the window a compaction request is about to summarize is archived whole. Every mined episode also carries the work phase its evidence names (test, implement, …), and a retrieval source can be restricted to phases in its policy, so a mode can read only what was learned while doing its kind of work. The retrieval result is a routing signal: an agent looping on one error signature with no known fix is moved to a configured, stronger model (knowledge-or-escalate, off by default).

The repository's own documentation joins the same knowledge base: a scan reads the *.md files below the directory the request declared, splits them at their headings, and publishes them as doc documents in a collection of their own (<prefix>_<scope>_docs), searched on the turn's subject rather than on its failure signature. Each work mode reads the kinds it wants — implement the readme, guides and contributing, plan those plus the changelog and notes, review the guides, contributing and security — and a guide that documents a file the turn is editing outranks one that does not. On top of retrieval the session carries a standing map of that documentation — which files exist and what they cover, structural only (paths, kinds, headings), stable per repository and budgeted before anything is retrieved — so an agent stops spending commands rediscovering what a project has written down, and a mined fix or dead end points at the page that documents the module it touched (src/export.py → docs/export.md), so a recipe arrives with the rule it has to keep. A file is republished when its content changes and the document it superseded is retired; a scan that ran out of its ceilings publishes no retirement at all. llm-router-agentic-kb-docs does the same scan on demand (--dry-run to see what would be published), which is also how a repository whose router runs on another machine gets documentation.

The engine may be reached by a pre-issued token or by a username/password login (SSE_API_USERNAME/SSE_API_PASSWORD; the client logs in on its first call).

The worker is a separate process the operator starts — scripts/plugins/agentic_knowledge/run-agentic-kb-worker.sh in the llm-router checkout takes its environment from the router's own run script. It creates a repository's collection on that repository's first event; a project that has produced nothing yet simply has an empty knowledge base.

A mode routing did not resolve falls back to the "*" policy rather than to no search at all, and the read budget (read.timeout_seconds) is a sum of engine calls — one per collection a mode reads — not a per-call timeout, which is set apart by SSE_API_TIMEOUT.

Connection settings (SSE and Redis) come only from LLM_ROUTER_KNOWLEDGE_AGENTIC_*; the policy (adapters, per-mode sources, weights and budgets, escalation models, retention) is agentic_knowledge.json. No SSE host means no reads, no Redis means no writes and a per-process cache instead of the shared one, and every failure is fail-open — the request passes through untouched. Codex is served fully (reads and writes); Claude Code is served in a basic, read-only form: scope, latest user text, injection into system.

Full documentation — the read and write paths, per-mode policies, the generation cache, knowledge-or-escalate, configuration and environment tables, the worker, privacy and retention, stateless mode, offline evaluation and limitations — lives in the Agentic knowledge layer README.

---

3. Pipelines

Both masker and guardrail pipelines share the same design pattern:

Class	Purpose
MaskerPipeline (pipeline.py – masker version)	Executes a list of masker plugins in order, transforming the payload step‑by‑step.
GuardrailPipeline (pipeline.py – guardrail version)	Executes guardrail plugins sequentially, stopping on the first failure.

3.1 Registration

• Plugins are registered lazily via MaskerRegistry.register(name, logger) or GuardrailRegistry.register(name, logger).
• The registry maps a string identifier (e.g. "fast_masker") to a concrete plugin class, allowing pipelines to resolve the classes at runtime.

3.2 Configuration

All plugin identifiers are stored in environment variables or constants such as:

MASKING_STRATEGY_PIPELINE = ["fast_masker"]
GUARDRAIL_STRATEGY_PIPELINE_REQUEST = ["nask_guard", "sojka_guard"]

These lists are consumed by the endpoint initialization (EndpointI._prepare_masker_pipeline, EndpointI._prepare_guardrails_pipeline).

---

4. Adding a New Plugin

1. Create a subclass of either PluginInterface (for maskers) or HttpPluginInterface / a custom guardrail base.
2. Define a name class attribute – this is the identifier used in pipeline configuration.
3. Implement apply(self, payload: Dict) -> Dict (masker) **or apply(self, payload: Dict) -> Tuple[bool, Dict] ** (guardrail).
4. Register the plugin – either automatically via the registry’s register call in the pipeline constructor, or manually by calling MaskerRegistry.register(name=MyPlugin.name, logger=logger).

Example stub for a new masker:

# my_custom_masker.py
from llm_router_plugins.maskers.plugin_interface import PluginInterface
import logging
from typing import Dict, Optional


class MyCustomMasker(PluginInterface):
    name = "my_custom_masker"

    def __init__(self, logger: Optional[logging.Logger] = None):
        super().__init__(logger=logger)
        # Load any heavy resources here (e.g., a spaCy model)

    def apply(self, payload: Dict) -> Dict:
        # Perform your masking logic and return the modified payload
        return payload

After placing the file in llm_router_plugins/maskers/plugins/, enable it by adding "my_custom_masker" to MASKING_STRATEGY_PIPELINE.

---

5. Retrieval‑Augmented Generation (RAG) Support

The project now includes a LangChain‑based RAG plugin that enables semantic search over user‑provided documents. The implementation lives in llm_router_plugins/utils/rag/langchain_plugin.py and is driven by the helper CLI scripts located in scripts/.

5.1 What the plugin does

Feature	Description
Indexing	Reads a directory of text‑like files (.txt, .md, .html, .js, …), splits them into token‑based windows, embeds each chunk with a configurable transformer model, and stores the vectors in a FAISS (or compatible) vector store.
Searching	Given a user query, retrieves the most similar chunks and injects them into the payload (e.g., appends to the last user message) so that downstream LLM calls can use the retrieved context.
Configuration	All parameters (collection name, embedder model, device, chunk size, overlap, persistence directory) are driven by environment variables prefixed with LLM_ROUTER_. See the table below for the full list.
CLI helpers	Two ready‑to‑use scripts: scripts/llm-router-rag-langchain-index.sh (indexes a repository) and scripts/llm-router-rag-langchain-search.sh (runs a search or starts an interactive REPL).

5.2 Environment variables

Variable	Default	Meaning
LLM_ROUTER_LANGCHAIN_RAG_COLLECTION	must be set	Name of the FAISS collection (e.g. sample_collection).
LLM_ROUTER_LANGCHAIN_RAG_EMBEDDER	/mnt/data2/llms/models/community/google/embeddinggemma-300m	Path or Hugging‑Face identifier of the embedding model.
LLM_ROUTER_LANGCHAIN_RAG_DEVICE	cuda:2	Torch device (cpu, cuda:0, …).
LLM_ROUTER_LANGCHAIN_RAG_CHUNK_SIZE	1024	Number of tokens per chunk.
LLM_ROUTER_LANGCHAIN_RAG_CHUNK_OVERLAP	100	Number of overlapping tokens between consecutive chunks.
LLM_ROUTER_LANGCHAIN_RAG_PERSIST_DIR	./workdir/plugins/utils/rag/langchain/${LLM_ROUTER_LANGCHAIN_RAG_COLLECTION}	Directory where the FAISS index and docstore are persisted.

Example export block (add to your shell profile or a .env file)

export LLM_ROUTER_LANGCHAIN_RAG_COLLECTION="${LLM_ROUTER_LANGCHAIN_RAG_COLLECTION:-sample_collection}"
export LLM_ROUTER_LANGCHAIN_RAG_EMBEDDER="${LLM_ROUTER_LANGCHAIN_RAG_EMBEDDER:-/mnt/data2/llms/models/community/google/embeddinggemma-300m}"
export LLM_ROUTER_LANGCHAIN_RAG_DEVICE="${LLM_ROUTER_LANGCHAIN_RAG_DEVICE:-cuda:2}"
export LLM_ROUTER_LANGCHAIN_RAG_CHUNK_SIZE="${LLM_ROUTER_LANGCHAIN_RAG_CHUNK_SIZE:-1024}"
export LLM_ROUTER_LANGCHAIN_RAG_CHUNK_OVERLAP="${LLM_ROUTER_LANGCHAIN_RAG_CHUNK_OVERLAP:-100}"
export LLM_ROUTER_LANGCHAIN_RAG_PERSIST_DIR="${LLM_ROUTER_LANGCHAIN_RAG_PERSIST_DIR:-./workdir/plugins/utils/rag/langchain/${LLM_ROUTER_LANGCHAIN_RAG_COLLECTION}}"

5.3 Using the CLI scripts

Index a repository (example for the documentation site):

scripts/llm-router-rag-langchain-index.sh
# Internally runs:
# llm-router-rag-langchain index --path "../.github/pages/llmrouter.cloud/" --ext .html .js .md

Search (interactive REPL):

scripts/llm-router-rag-langchain-search.sh
# Internally runs:
# llm-router-rag-langchain search
# (you will be prompted for a query, type “exit” to quit)

One‑shot search:

llm-router-rag-langchain search --query "What is Retrieval‑Augmented Generation?" --top_n 5

The CLI returns the raw matching chunks together with similarity scores. The LangchainRAGPlugin automatically formats the retrieved text and appends it to the user’s last message, prefixed with:

If the context below will help answer the above question, use it.
Context separated with double enter