ASAG-RAG

ASAG-RAG is a research project for automated short-answer grading. It can grade a dataset with an LLM alone or enrich the grading prompt with retrieved examples using a retrieval-augmented generation (RAG) pipeline.

Requirements

Python 3.12 (the supported range is >=3.12,<3.13)
uv
An API key for the selected model provider. The supplied experiment configurations use OpenAI.

Project structure

.
├── main.py                 # Command-line entry point for grading experiments
├── pyproject.toml          # Python package metadata and dependencies
├── uv.lock                 # Locked dependency versions
├── .env.example            # Template for provider credentials
├── configs/
│   └── experiments/        # Reusable JSON experiment configurations
├── dataset/
│   ├── raw/                # Source datasets
│   ├── translated/         # Translated dataset variants
│   ├── processed/          # CSV splits consumed by main.py
│   ├── embeddings/         # FAISS indexes used by retrieval pipelines
│   ├── augmented/          # LLM-augmented training data
│   └── labelled/           # Labeled grading data and outputs
├── src/rag/
│   ├── pipelines/          # No-retrieval, vector, BM25, and LinUCB pipelines
│   ├── retrievers/         # Local, external, hybrid, vector, and BM25 retrieval
│   ├── llms/               # OpenAI, Gemini, Hugging Face, and OpenRouter backends
│   ├── ingestions/         # CSV, Excel, Parquet, and dataset ingestion helpers
│   ├── config.py           # Shared RAG/LLM settings
│   └── pipeline.py         # Pipeline dispatcher
├── eval/                   # Evaluation datasets, metrics, and runner scripts
├── notebooks/              # Exploratory data preparation, training, and analysis
├── params/                 # Saved experiment parameter files
├── result/                 # Generated grading and evaluation results
└── scripts/                # Maintenance and release scripts

The primary input is a CSV split in a folder under dataset/processed/. The value passed to --split is its filename without .csv (for example, test for test.csv). A grading CSV must provide the question, reference-answer, and student-answer columns named by the corresponding CLI options. Retrieval runs also require a document embedding directory; the LinUCB pipeline additionally requires a question-embedding directory.

How to run

1. Install dependencies

From the repository root:

uv sync

2. Configure credentials

Copy the template and fill in only the keys required by your selected backend:

Copy-Item .env.example .env

For the default OpenAI backend, set OPENAI_API_KEY in .env:

OPENAI_API_KEY=your_key_here

The application loads .env automatically. Do not commit this file.

3. Run an included experiment

Use a JSON configuration for reproducible runs. Command-line options override values in the JSON file.

uv run main.py --config configs/experiments/rahutomo-no-retrieval-test.json

To run the supplied LinUCB retrieval experiment:

uv run main.py --config configs/experiments/saf-translated-linucb.json

Results are written to the configuration's output_folder, typically below result/evaluation/grading/.

4. Run with explicit options

No-retrieval grading:

uv run main.py `
  --pipeline no-retrieval `
  --dataset-path ./dataset/processed/rahutomo `
  --split test `
  --output-folder ./result/evaluation/grading/rahutomo `
  --question-column Soal `
  --reference-column KunciJawaban `
  --answer-column Jawaban `
  --min-score 0 `
  --max-score 100 `
  --backend openai `
  --model gpt-4o-mini

LinUCB retrieval grading:

uv run main.py `
  --pipeline linucb `
  --retrieve-method linucb `
  --dataset-path ./dataset/processed/saf-translated `
  --embedding-path ./dataset/embeddings/saf-translated `
  --question-embedding-path ./dataset/embeddings/saf-translated-question `
  --split test `
  --output-folder ./result/evaluation/grading/saf-translated `
  --question-column question_ind `
  --reference-column reference_ind `
  --answer-column answer_ind `
  --score-column score `
  --min-score 0 `
  --max-score 1 `
  --backend openai `
  --model gpt-4o-mini

Useful options include --batch-num (number of output batches), --start (first batch index), --top-k (retrieval count), and --temperature. See all options with:

uv run main.py --help

Pipeline note

main.py --help currently lists vector and bm25, while the internal pipeline dispatcher registers them as vector-retrieval and bm25-retrieval. As a result, those two CLI values do not currently resolve to a pipeline. Use no-retrieval or linucb from the command line, or align the names in main.py and src/rag/pipelines/__init__.py before using vector or BM25 runs.

Evaluation

The eval/ directory contains scripts for per-label, per-classification, source, and translation-quality evaluation. Run a script through uv after reviewing its options, for example:

uv run eval/run_local_eval.py --help

Metadata

Release files for asag-rag 1.1.127

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for asag-rag 1.1.127
File	Size	Uploaded
asag_rag-1.1.127.tar.gz	21.7 kB	Oct 10, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for asag-rag 1.1.127
File	Interpreter	ABI	Platform	Reset
asag_rag-1.1.127-py3-none-any.whl 36.3 kB Oct 10, 2026	Python 3	none	any	Details

Total release size: 58.0 kB

Release files / asag_rag-1.1.127.tar.gz

Download URL	asag_rag-1.1.127.tar.gz
Size	21.7 kB
Tags	Source
SHA-256 checksum How to use checksums	`12e2f0544eb059a04c4ec727be0ee3648277bf4e71318a7893a5fa01492e971a`
BLAKE2b-256 checksum How to use checksums	`78d8faa038f451f7fecb63bcf52cf4a96c3f8addb7d06d2d9a8a448f0a71af3e`
Upload date	Oct 10, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.2.0 CPython/3.12.4`

Release files / asag_rag-1.1.127-py3-none-any.whl

Download URL	asag_rag-1.1.127-py3-none-any.whl
Size	36.3 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`ae6385627cc4477ff7c6391dd5e5bebf6e781868e256fb5b3ff788fe62ff3eb3`
BLAKE2b-256 checksum How to use checksums	`019c7977340fb2069152599296ad1a766ffb17bed5e527a131841e380bbf3680`
Upload date	Oct 10, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.2.0 CPython/3.12.4`