Benchmark your local LLMs.
Project description
Introduction
Story
🚀 Installation
$ pip install benchllama
⚙️ Usage
$ benchllama [OPTIONS] COMMAND [ARGS]...
Options:
--install-completion
: Install completion for the current shell.--show-completion
: Show completion for the current shell, to copy it or customize the installation.--help
: Show this message and exit.
Commands:
evaluate
clean
benchllama evaluate
Usage:
$ benchllama evaluate [OPTIONS]
Options:
--models TEXT
: Names of models that need to be evaluated. [required]--provider-url TEXT
: The endpoint of the model provider. [default: http://localhost:11434]--dataset FILE
: By default, bigcode/humanevalpack from Hugging Face will be used. If you want to use your own dataset, specify the path here.--languages [python|js|java|go|cpp]
: List of languages to evaluate from bigcode/humanevalpack. Ignore this if you are brining your own data [default: Language.python]--num-completions INTEGER
: Number of completions to be generated for each task. [default: 3]--no-eval / --eval
: If true, evaluation will be done [default: no-eval]--k INTEGER
: The k for calculating pass@k. The values shouldn't exceed num_completions [default: 1, 2]--samples INTEGER
: Number of dataset samples to evaluate. By default, all the samples get processed. [default: -1]--output PATH
: Output directory [default: /tmp]--help
: Show this message and exit.
benchllama clean
Usage:
$ benchllama clean [OPTIONS]
Options:
--run-id TEXT
: Run id--output PATH
: Output directory [default: /tmp]--help
: Show this message and exit.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
benchllama-0.2.3.tar.gz
(11.0 kB
view details)
Built Distribution
File details
Details for the file benchllama-0.2.3.tar.gz
.
File metadata
- Download URL: benchllama-0.2.3.tar.gz
- Upload date:
- Size: 11.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.3.1 CPython/3.10.11 Darwin/23.1.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | eb84850d6b4bf392988e66b0e1267e157889e1a70d1de8d0281e1cf429d1f210 |
|
MD5 | daaa40cbca195e673a86c3c83e5d1209 |
|
BLAKE2b-256 | 1c3427066211cd1adddc8fe49a03202c327b568abc11716cec491eacb25dd00b |
File details
Details for the file benchllama-0.2.3-py3-none-any.whl
.
File metadata
- Download URL: benchllama-0.2.3-py3-none-any.whl
- Upload date:
- Size: 16.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.3.1 CPython/3.10.11 Darwin/23.1.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1fe526a73868e2c787db4cdac44e2f61ad41d7539461a1ecc48fd60a49075445 |
|
MD5 | 9aa08f94d492b8412d0c34e9c7df0f8c |
|
BLAKE2b-256 | ea8bb3cd4a44083c80f11067802aa613a953e3071b7043bc586c4575c4bdb6eb |