Skip to main content

Benchmark your local LLMs.

Project description

🧮 Benchllama

An open-source tool to benchmark you local LLMs.

License: MIT PyPI Supported Versions GitHub: Downloads Discord

Introduction

Story

🚀 Installation

$ pip install benchllama

⚙️ Usage

$ benchllama [OPTIONS] COMMAND [ARGS]...

Options:

  • --install-completion: Install completion for the current shell.
  • --show-completion: Show completion for the current shell, to copy it or customize the installation.
  • --help: Show this message and exit.

Commands:

  • evaluate
  • clean

benchllama evaluate

Usage:

$ benchllama evaluate [OPTIONS]

Options:

  • --models TEXT: Names of models that need to be evaluated. [required]
  • --provider-url TEXT: The endpoint of the model provider. [default: http://localhost:11434]
  • --dataset FILE: By default, bigcode/humanevalpack from Hugging Face will be used. If you want to use your own dataset, specify the path here.
  • --languages [python|js|java|go|cpp]: List of languages to evaluate from bigcode/humanevalpack. Ignore this if you are brining your own data [default: Language.python]
  • --num-completions INTEGER: Number of completions to be generated for each task. [default: 3]
  • --no-eval / --eval: If true, evaluation will be done [default: no-eval]
  • --k INTEGER: The k for calculating pass@k. The values shouldn't exceed num_completions [default: 1, 2]
  • --samples INTEGER: Number of dataset samples to evaluate. By default, all the samples get processed. [default: -1]
  • --output PATH: Output directory [default: /tmp]
  • --help: Show this message and exit.

benchllama clean

Usage:

$ benchllama clean [OPTIONS]

Options:

  • --run-id TEXT: Run id
  • --output PATH: Output directory [default: /tmp]
  • --help: Show this message and exit.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

benchllama-0.2.3.tar.gz (11.0 kB view details)

Uploaded Source

Built Distribution

benchllama-0.2.3-py3-none-any.whl (16.2 kB view details)

Uploaded Python 3

File details

Details for the file benchllama-0.2.3.tar.gz.

File metadata

  • Download URL: benchllama-0.2.3.tar.gz
  • Upload date:
  • Size: 11.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.3.1 CPython/3.10.11 Darwin/23.1.0

File hashes

Hashes for benchllama-0.2.3.tar.gz
Algorithm Hash digest
SHA256 eb84850d6b4bf392988e66b0e1267e157889e1a70d1de8d0281e1cf429d1f210
MD5 daaa40cbca195e673a86c3c83e5d1209
BLAKE2b-256 1c3427066211cd1adddc8fe49a03202c327b568abc11716cec491eacb25dd00b

See more details on using hashes here.

File details

Details for the file benchllama-0.2.3-py3-none-any.whl.

File metadata

  • Download URL: benchllama-0.2.3-py3-none-any.whl
  • Upload date:
  • Size: 16.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.3.1 CPython/3.10.11 Darwin/23.1.0

File hashes

Hashes for benchllama-0.2.3-py3-none-any.whl
Algorithm Hash digest
SHA256 1fe526a73868e2c787db4cdac44e2f61ad41d7539461a1ecc48fd60a49075445
MD5 9aa08f94d492b8412d0c34e9c7df0f8c
BLAKE2b-256 ea8bb3cd4a44083c80f11067802aa613a953e3071b7043bc586c4575c4bdb6eb

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page