NL2SQLEvaluator

Roadmap

Add MySQL database executor
Add Precision, Recall, F1 metrics for ambiguity Text2SQL datasets

👷🏼‍♂️ Work in progress

Configuration Guide

🔧 Use a YAML config (with CLI overrides)

Run your experiment with a config file:

nl2sql_eval --config path/to/config.yaml

Example `config.yaml`

# Core
output_dir: ./outputs
seed: 42

# Dataset
relative_db_base_path: data/bird_dev/dev_databases
dataset_path: simone-papicchio/bird
dataset_name: bird-dev

# Model
model_name: Qwen3-Coder-30B
model: Qwen/Qwen3-Coder-30B-A3B-Instruct
temperature: 0.7
top_p: 0.8
top_k: 20
repetition_penalty: 1.05
max_tokens: 32000

# Weights & Biases
project: text2sql-eval
entity: spapicchio-politecnico-di-torino   # or your team
group: evals
mode: online                               # or "offline" on clusters without net
tags: [eval, seg]
notes: ""
job_type: eval

Override any value from the CLI

Command-line flags take precedence over the YAML:

nl2sql_eval --config config.yaml \
  --output_dir ./outputs/run-42 \
  --mode offline \
  --temperature 0.2 \
  --max_tokens 4096 \
  --tags eval --tags ablation

Notes

The config is flat (all keys at top level) so it works smoothly with the parser.
Lists (e.g., tags) can be provided in YAML or by repeating the flag in CLI (--tags ... multiple times).
Booleans accept true/false in YAML and --flag true/false in CLI.
This package uses TRL’s TRLParser / HF’s HfArgumentParser under the hood, so the same configuration behaviors apply.

This package relies on TrlParser so all the configurations available there can be used as well.

Metadata

Release files for NL2SQLEvaluator 1.3.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for NL2SQLEvaluator 1.3.4
File	Size	Uploaded
nl2sqlevaluator-1.3.4.tar.gz	134.7 kB	Nov 13, 2025	Details

Built distribution (wheel)

Table of built distributions (wheels) for NL2SQLEvaluator 1.3.4
File	Interpreter	ABI	Platform	Reset
nl2sqlevaluator-1.3.4-py3-none-any.whl 40.5 kB Nov 13, 2025	Python 3	none	any	Details

Total release size: 175.2 kB

Release files / nl2sqlevaluator-1.3.4.tar.gz

Download URL	nl2sqlevaluator-1.3.4.tar.gz
Size	134.7 kB
Tags	Source
SHA-256 checksum How to use checksums	`2cc4d72e472452b880be98ae38ffd8ffcc1d4b839b2959a98fa2a5af203275e5`
BLAKE2b-256 checksum How to use checksums	`8e633c107492e8b1fa2751b8acc257826370ae63f8ebdd44ced902cebc906e69`
Upload date	Nov 13, 2025
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.7.13`

Release files / nl2sqlevaluator-1.3.4-py3-none-any.whl

Download URL	nl2sqlevaluator-1.3.4-py3-none-any.whl
Size	40.5 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`e39d95f34531f0ca679ff9316319372c8c705ff926af89266d70234b25ffe780`
BLAKE2b-256 checksum How to use checksums	`4693c7d7975b934c4c06b1201995b42db3c046d773e46d1e39ff718bd8eedf2f`
Upload date	Nov 13, 2025
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.7.13`