smallevals - Small Language Models Evaluation Suite for RAG Systems

A lightweight evaluation framework powered by tiny ( really tiny ) 0.6B models — runs 100% locally on CPU/GPU/MPS, attach any vector DB connection and run, fast and free.

Evaluation tools requiring LLM-as-a-judge or external, that costs/doesn't scale easily. evaluates in seconds in GPU, in minutes in any CPU !

Evaluate Retrieval

Evaluation of RAG system includes retrieval and RAG stage, attacks to test retrieval and RAG answers(in the near future)!

Models

Current Focus: Retrieval evaluation (QAG-0.6B), after being sure the model generates correct answers and better questions for RAG(it does, but still room for improvement), the model will be the first model of pipeline leading to (RAG) generation evaluation models (CRC-0.6B, GJ-0.6B, ASM-0.5B) which are the future work.

How does it work?

Question Generator model, reads your chunk, assumes the chunk is the one that answers the question, and tries to match it back via Vector DB query.

This allows directly to test your retrieval pipelines tied to your RAG systems. Whatever the complexity of your RAG system, you'll be sure if your vector queries works fine.

Why this is a need?

Other frameworks requiring APIs are costly, hard-to-scale, although they are better(for now).

Installation

pip install smallevals

Quick Start

Evaluate Retrieval Quality (Python)

Connect to your favourite Vector DB (Milvus, Elastic, PGVector, Chroma, Pinecone, FAISS, Weawiate), attach your favourite embeddings, generate questions, and visualise results!

Under the hood, generates question per chunk, and tries to retrieve it as a single-first relevant docs, calculate scores.

from smallevals import evaluate_retrievals, SmallEvalsVDBConnection

vdb = SmallEvalsVDBConnection(
    connection=chroma_client,
    collection="my_collection",
    embedding=embedding
)

# Run evaluation
result = evaluate_retrievals(connection=vdb, top_k=10, n_chunks=200) # Generate question for 200 chunks, and test to retrieve them!

And evaluate results!

smallevals dash --host 0.0.0.0 --port 8050 --debug

Generate QA from Documents (CLI)

smallevals generate_qa --docs-dir ./documents --num-questions 100

### QAG-0.6B

The model was trained on TriviaQA, SQuAD 2.0, Hand-curated synthetic data generated using Qwen-70B , generating a question from the chunk/doc.

Given the passage below, extract ONE question/answer pair grounded strictly in a single atomic fact.

PASSAGE:
"Eiffel tower is built at 1989"

Return ONLY a JSON object.

{
  "question": "When was the Eiffel Tower completed?",
  "answer": "1889"
}

Known issues:

• Model is trained on text/wiki data, bias towards well structured text.
• Dataset contains question that ask generic questions, dataset will be more carefully crafted in v3.

### Other Models:

Other models to be trained to eliminate the need of external LLMs.

CRC-0.6B : Context relevance classifier (question ↔ retrieved chunk) GJ-0.6B : Groundedness / faithfulness judge (answer ↔ context)
ASM-0.6B | Answer correctness / semantic similarity