llama_api_bench
Table of Contents
Installation
pip install llama-api-bench
Install from source
pip install -e .
# dev
pip install -e ".[dev]"
Usage
Run All
llama-api-bench run-all
Run Subset
- Run on a specific criteria for all models and providers
llama-api-bench run-criteria --criteria tool_call
- Run on a specific test case data for all models and providers
llama-api-bench run-test-case --test-case tool_call_get_weather
- Run all test cases on a specific provider
llama-api-bench run-provider --provider openrouter
- Run on a specific test case x model x provider
# streaming
llama-api-bench run --test-case basic_dont_call_tool --model Cerebras-Llama-4-Scout-17B-16E-Instruct --provider llamaapi --stream
# non-streaming
llama-api-bench run --test-case basic_dont_call_tool --model Cerebras-Llama-4-Scout-17B-16E-Instruct --provider llamaapi --no-stream
- Run on specific provider x model
llama-api-bench run-provider --provider openai --models gpt-4o
llama-api-bench run-provider --provider llamaapi --models Llama-4-Maverick-17B-128E-Instruct-FP8
Customization
Add a New Test Case / Criteria
- Add a new test case in llama_api_bench/data/.
- If the new test case requires a new Test Criteria to run evaluation, add it to llama_api_bench/core/criterias.
Add New Provider
- Add supported models in llama_api_bench/models/. See an example in llamaapi.py.
- Add ProviderConfig in llama_api_bench/core/providers.py.
- If the provider requires has a new request format, add it to get_request_json.
- If the provider requires a new response format parsing, add its evaluation criteria in llama_api_bench/core/criterias.
License
llama-api-bench is distributed under the terms of the MIT license.
Metadata
Release files for llama-api-bench 0.0.10
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| llama_api_bench-0.0.10.tar.gz | 15.7 kB | Without provenance | Details for llama_api_bench-0.0.10.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| llama_api_bench-0.0.10-py3-none-any.whl | Python 3 | none | any | Details for llama_api_bench-0.0.10-py3-none-any.whl |
Total release size: 41.2 kB
Release files / llama_api_bench-0.0.10.tar.gz
| Download URL | llama_api_bench-0.0.10.tar.gz |
|---|---|
| Size | 15.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
cc2d53215da01dd7bad27108dbd5287b2205b62de5e53d3d0917031de0dd45d1
|
|
BLAKE2b-256 checksum How to use checksums |
f0c72e01184322e1972c60bdd3501040666d2de7143eab92504d6ab359a6fd76
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
python-httpx/0.28.1
|
Release files / llama_api_bench-0.0.10-py3-none-any.whl
| Download URL | llama_api_bench-0.0.10-py3-none-any.whl |
|---|---|
| Size | 25.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2bd528b1f016e17cc8e3b1636ce02061c8546e2b91ea10d972c96b0bfe8ada7e
|
|
BLAKE2b-256 checksum How to use checksums |
25167a1578f8360bd11f0e2e7118ccb0d0bb5dfe7db2050acf33d923fe54e0b5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
python-httpx/0.28.1
|
Security for llama-api-bench 0.0.10
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.