Comparative evaluation harness to benchmark local vs cloud LLMs
Project description
∑AI Benchmarks
Overview
Repository for automated benchmarking of LLMs with reproducible pipelines.
Structure
- scripts/ – runners, guard, helpers
- tests/ – prompt corpora
- artifacts/ – results and plots
- .github/workflows/ – CI pipelines
Quickstart
python -m venv .venv source .venv/bin/activate pip install -U pip matplotlib pandas
Smoke test
LIMIT=30 MODE=smoke bash scripts/ab_benchmark.sh "llama3.1:8b" "llama3.1:8b"
Nightly
LIMIT=500 MODE=nightly bash scripts/ab_benchmark.sh "llama3.1:8b" "llama3.1:8b"
Outputs
- artifacts/summary/ab_diff.csv
- artifacts/plots/ab_plot.png
- artifacts/manifest.json
Nightly usage
Run nightly: gh workflow run nightly.yml --ref master
Verify last run locally: bash scripts/verify_t500.sh $(date +%F)
Keep N runs: bash scripts/retention.sh
T2000
Статус: validated (baseline зафиксирован, воспроизводимость подтверждена)
Scientific Archive (T3000 Freeze)
- Proof bundle:
artifacts/releases/SIGMA_AI_T3000_PROOF.tar.gz - Legal set:
artifacts/releases/ΣAI_LEGAL_PROOF_SET1.tar.gz - Baseline tag:
stable_t3000_sb1
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file sigma_ai-0.1.0.tar.gz.
File metadata
- Download URL: sigma_ai-0.1.0.tar.gz
- Upload date:
- Size: 2.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.1.0 CPython/3.13.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
02a805c202a98bf1afb08801e101b893cb44887d449b82d708a089dd0a098c5a
|
|
| MD5 |
6bd6d51ded0e80edb2e7a0770ba21451
|
|
| BLAKE2b-256 |
649f33319ce5e9531afb2bc55c2fd10db25fdf7e0f82faf8ca2f6b6bcd86f088
|
File details
Details for the file sigma_ai-0.1.0-py3-none-any.whl.
File metadata
- Download URL: sigma_ai-0.1.0-py3-none-any.whl
- Upload date:
- Size: 2.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.1.0 CPython/3.13.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
2069e33b4206cc2491e5c7cc6abb2b04081bc98e0dea57a85e61837a5e7257d5
|
|
| MD5 |
8388c498fb27d2a8573055ff98470f76
|
|
| BLAKE2b-256 |
5fd0eb6f78cca95ea78a881012afc305f4a1eb890e9c55ad473862abd8ac0e5d
|
