ASAG-RAG

ASAG-RAG is a research project for automated short-answer grading. It can grade a dataset with an LLM alone or enrich the grading prompt with retrieved examples using a retrieval-augmented generation (RAG) pipeline.

Requirements

Python 3.12 (the supported range is >=3.12,<3.13)
uv
An API key for the selected model provider. The supplied experiment configurations use OpenAI.

Project structure

.
├── main.py                 # Command-line entry point for grading experiments
├── pyproject.toml          # Python package metadata and dependencies
├── uv.lock                 # Locked dependency versions
├── .env.example            # Template for provider credentials
├── configs/
│   └── experiments/        # Reusable JSON experiment configurations
├── dataset/
│   ├── raw/                # Source datasets
│   ├── translated/         # Translated dataset variants
│   ├── processed/          # CSV splits consumed by main.py
│   ├── embeddings/         # FAISS indexes used by retrieval pipelines
│   ├── augmented/          # LLM-augmented training data
│   └── labelled/           # Labeled grading data and outputs
├── src/rag/
│   ├── pipelines/          # No-retrieval, vector, BM25, and LinUCB pipelines
│   ├── retrievers/         # Local, external, hybrid, vector, and BM25 retrieval
│   ├── llms/               # OpenAI, Gemini, Hugging Face, and OpenRouter backends
│   ├── ingestions/         # CSV, Excel, Parquet, and dataset ingestion helpers
│   ├── config.py           # Shared RAG/LLM settings
│   └── pipeline.py         # Pipeline dispatcher
├── eval/                   # Evaluation datasets, metrics, and runner scripts
├── notebooks/              # Exploratory data preparation, training, and analysis
├── params/                 # Saved experiment parameter files
├── result/                 # Generated grading and evaluation results
└── scripts/                # Maintenance and release scripts

The primary input is a CSV split in a folder under dataset/processed/. The value passed to --split is its filename without .csv (for example, test for test.csv). A grading CSV must provide the question, reference-answer, and student-answer columns named by the corresponding CLI options. Retrieval runs also require a document embedding directory; the LinUCB pipeline additionally requires a question-embedding directory.

How to run

1. Install dependencies

From the repository root:

uv sync

2. Configure credentials

Copy the template and fill in only the keys required by your selected backend:

Copy-Item .env.example .env

For the default OpenAI backend, set OPENAI_API_KEY in .env:

OPENAI_API_KEY=your_key_here

The application loads .env automatically. Do not commit this file.

3. Run an included experiment

Use a JSON configuration for reproducible runs. Command-line options override values in the JSON file.

uv run main.py --config configs/experiments/rahutomo-no-retrieval-test.json

To run the supplied LinUCB retrieval experiment:

uv run main.py --config configs/experiments/saf-translated-linucb.json

Results are written to the configuration's output_folder, typically below result/evaluation/grading/.

4. Run with explicit options

No-retrieval grading:

uv run main.py `
  --pipeline no-retrieval `
  --dataset-path ./dataset/processed/rahutomo `
  --split test `
  --output-folder ./result/evaluation/grading/rahutomo `
  --question-column Soal `
  --reference-column KunciJawaban `
  --answer-column Jawaban `
  --min-score 0 `
  --max-score 100 `
  --backend openai `
  --model gpt-4o-mini

LinUCB retrieval grading:

uv run main.py `
  --pipeline linucb `
  --retrieve-method linucb `
  --dataset-path ./dataset/processed/saf-translated `
  --embedding-path ./dataset/embeddings/saf-translated `
  --question-embedding-path ./dataset/embeddings/saf-translated-question `
  --split test `
  --output-folder ./result/evaluation/grading/saf-translated `
  --question-column question_ind `
  --reference-column reference_ind `
  --answer-column answer_ind `
  --score-column score `
  --min-score 0 `
  --max-score 1 `
  --backend openai `
  --model gpt-4o-mini

Useful options include --batch-num (number of output batches), --start (first batch index), --top-k (retrieval count), and --temperature. See all options with:

uv run main.py --help

Pipeline note

main.py --help currently lists vector and bm25, while the internal pipeline dispatcher registers them as vector-retrieval and bm25-retrieval. As a result, those two CLI values do not currently resolve to a pipeline. Use no-retrieval or linucb from the command line, or align the names in main.py and src/rag/pipelines/__init__.py before using vector or BM25 runs.

Evaluation

The eval/ directory contains scripts for per-label, per-classification, source, and translation-quality evaluation. Run a script through uv after reviewing its options, for example:

uv run eval/run_local_eval.py --help

Metadata

Release files for asag-rag 1.1.129

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for asag-rag 1.1.129
File	Size	Uploaded
asag_rag-1.1.129.tar.gz	21.6 kB	Oct 13, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for asag-rag 1.1.129
File	Interpreter	ABI	Platform	Reset
asag_rag-1.1.129-py3-none-any.whl 36.3 kB Oct 13, 2026	Python 3	none	any	Details

Total release size: 58.0 kB

Release files / asag_rag-1.1.129.tar.gz

Download URL	asag_rag-1.1.129.tar.gz
Size	21.6 kB
Tags	Source
SHA-256 checksum How to use checksums	`06bd87a039fa66d932aa649586d0342066f655c92244aafce8753d337496e9f9`
BLAKE2b-256 checksum How to use checksums	`23a6fb3e189ccd7639a7fd6bdd28ef55a1fff1d2e5233edceb090bd3af13695b`
Upload date	Oct 13, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.2.0 CPython/3.12.4`

Release files / asag_rag-1.1.129-py3-none-any.whl

Download URL	asag_rag-1.1.129-py3-none-any.whl
Size	36.3 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`5a33a560b1bcdcc82c9b9d16ece267c592bcddb1758dbaf2a6f5f9f495934913`
BLAKE2b-256 checksum How to use checksums	`873f01dcd26e01d7d9a1b4ea83bf69c6f61fffa3ecf1c1fa42bc0119e9277275`
Upload date	Oct 13, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.2.0 CPython/3.12.4`