Benchmark your local LLMs.
Project description
Introduction
Story
🚀 Installation
$ pip install benchllama
⚙️ Usage
$ benchllama [OPTIONS] COMMAND [ARGS]...
Options:
--install-completion
: Install completion for the current shell.--show-completion
: Show completion for the current shell, to copy it or customize the installation.--help
: Show this message and exit.
Commands:
evaluate
clean
benchllama evaluate
Usage:
$ benchllama evaluate [OPTIONS]
Options:
--models TEXT
: Names of models that need to be evaluated. [required]--provider-url TEXT
: The endpoint of the model provider. [default: http://localhost:11434]--dataset FILE
: By default, bigcode/humanevalpack from Hugging Face will be used. If you want to use your own dataset, specify the path here.--languages [python|js|java|go|cpp]
: List of languages to evaluate from bigcode/humanevalpack. Ignore this if you are brining your own data [default: Language.python]--num-completions INTEGER
: Number of completions to be generated for each task. [default: 3]--no-eval / --eval
: If true, evaluation will be done [default: no-eval]--k INTEGER
: The k for calculating pass@k. The values shouldn't exceed num_completions [default: 1, 2]--samples INTEGER
: Number of dataset samples to evaluate. By default, all the samples get processed. [default: -1]--output PATH
: Output directory [default: /tmp]--help
: Show this message and exit.
benchllama clean
Usage:
$ benchllama clean [OPTIONS]
Options:
--run-id TEXT
: Run id--output PATH
: Output directory [default: /tmp]--help
: Show this message and exit.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
benchllama-0.2.3.tar.gz
(11.0 kB
view hashes)
Built Distribution
benchllama-0.2.3-py3-none-any.whl
(16.2 kB
view hashes)
Close
Hashes for benchllama-0.2.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1fe526a73868e2c787db4cdac44e2f61ad41d7539461a1ecc48fd60a49075445 |
|
MD5 | 9aa08f94d492b8412d0c34e9c7df0f8c |
|
BLAKE2b-256 | ea8bb3cd4a44083c80f11067802aa613a953e3071b7043bc586c4575c4bdb6eb |