pyforla

CLI client for LFORLA — run LLM benchmarks locally and push results to the leaderboard. Pairs with the LFORLA platform.

Features

• pull — download benchmark metadata and dataset samples (JSON/YAML)
• run — evaluate samples locally against an LLM API (OpenAI, Anthropic, Ollama, generic)
• push — submit results to the LFORLA leaderboard
• report — generate a markdown/HTML evaluation report from results
• dataset upload — upload sample files to MinIO and link them to a benchmark
• bench-push — publish benchmark definitions (moderator-only)

Install

pip install pyforla

Or from source:

git clone https://github.com/lforla/pyforla.git
cd pyforla && pip install -e .

Verify:

lforla-eval --help

Quick Start

# Save your API key (create one in your LFORLA account settings)
lforla-eval login "your-api-key"

# Download a benchmark's samples
lforla-eval pull root-cause-analysis -o ./data

# Run the samples locally against a model
lforla-eval run ./data/root-cause-analysis_samples.json -m gpt-4o -p openai

# Push the results to the leaderboard
lforla-eval push results.json --benchmark-id <uuid>

Commands

Docker

A container image is provided for isolated or server-side runs:

docker build -t lforla-eval .
docker run --rm -v "$PWD:/data" lforla-eval run /data/samples.json -m gpt-4o -p openai

Documentation

• Full CLI documentation: https://lforla.org/cli
• API reference: https://lforla.org/api/v1

License

MIT