SafeLLM4SE

SafeLLM4SE is a Python toolkit for statistically principled evaluation of LLM-based software engineering systems. It treats each LLM execution as a sample from a stochastic process, then reports quality, stability, uncertainty, resource usage, and statistical comparisons instead of relying on a single run.

The project exposes three command-line programs:

• safellm4se-sample: runs adaptive sampling with a user-selected evaluator.
• safellm4se-report: summarizes one sampled task into a SafeLLM4SE report CSV.
• safellm4se-compare: compares two sampled tasks with the SafeLLM4SE comparison protocol.

Install

SafeLLM4SE is available on PyPI as safellm4se, so you only need to run:

python -m pip install safellm4se

If you want to install SafeLLM4SE from its source code, check the installation details.

Quick Start

SafeLLM4SE collects repeated observations with safellm4se-sample. Then, it can be used to report or visualize one sample of observations with safellm4se-report, or compare task samples with safellm4se-compare.

Sampling

First, you need to generate a sample of repeated observations. For this purpose, safellm4se-sample loads an evaluator class and repeatedly calls it until a stopping condition is met (maximum number of tokens consumed or the confidence interval width is below a threshold). You commonly implement the process being measured as an evaluator, but we provide several example evaluators, including default implementations for Ollama, Gemini, and Groq APIs.

For this example, we perform adaptive sampling on a random evaluator:

safellm4se-sample --evaluator safellm4se.sampling.myevaluators.random_normal_evaluator

Reporting

If you have already created a sample of repeated observations with the task identifier task-id-1, stored in output/measurements.csv, you can create reports and visualizations with safellm4se-report. The report includes the sample size, token usage, central tendency, variability, and confidence interval information. The supported visualizations are boxplot, violin plot, empirical cumulative distribution function (ECDF), raincloud plot, and kernel density estimate (KDE).

safellm4se-report --input output/measurements.csv --output output/report-demo.csv --task-id task-id-1 --boxplot output/demo-boxplot.svg

Example CSV report generated with safellm4se-report:

task_id,model_name,model_id,N,total_tokens,theta_mean,sd,ci_method,ci_low,ci_high,ci_width
task-id-54,qwen-coder,qwen2.5-coder:7b,30,1178530,0.8272357723577236,0.019436705668000667,t,0.8199779871829312,0.834493557532516,0.014515570349584728

Comparing

You can also compare two samples of repeated observations with safellm4se-compare. The report will tell you the estimated difference, confidence interval, statistical test, p-value, and effect size. It also generates figures comparing the two samples, including the visualizations mentioned in reporting.

safellm4se-compare --input output/measurements.csv --output output/comparing-demo.csv --task-id-1 task-a --task-id-2 task-b --test-type paired --raincloud output/comparing-demo-raincloud.svg

Example plot generated with safellm4se-compare:

The following CSV comparison report has also been generated:

task_id_1,task_id_2,test_type,estimated_difference,ci_low,ci_high,statistical_test,p_value,effect_size_name,effect_size,effect_size_magnitude
task-id-54,task-id-56,independent,0.20833333333333337,0.19369410569105683,0.2223628048780488,Mann-Whitney U,2.8591961948613224e-11,Cliff's delta,1.0,large

Documentation

• Methodology
• Installation
• Usage Guide
• CLI Reference
• Evaluator Guide

License

See LICENSE.