localbench 0.0.1

Add your description here

localbench

A benchmarking tool for Local LLMs. Currently works with Cortex.cpp.

What is this?

localbench measures performance metrics, resource utilization, and stability characteristics of your LLM deployments. Rather comprehensive, really.

Features

• Model initialization metrics
• Runtime performance
• Resource utilization
• Advanced processing scenarios
• Workload-specific benchmarks
• System integration metrics
• Stability analysis

Installation

Using uvx:

uvx install localbench

Using pip:

pip install localbench

Usage

Basic Benchmarking

# Standard benchmark
localbench benchmark "llama3.2:3b-gguf-q2-k"

# With detailed metrics
localbench benchmark "llama3.2:3b-gguf-q2-k" --verbose

Specific Benchmarks

# Initialization only
localbench benchmark "llama3.2:3b-gguf-q2-k" --type init

# Runtime metrics
localbench benchmark "llama3.2:3b-gguf-q2-k" --type runtime

# Long-running stability test
localbench benchmark "llama3.2:3b-gguf-q2-k" --type stability --stability-duration 24

Advanced Usage

# Custom benchmark prompts
localbench benchmark "llama3.2:3b-gguf-q2-k" --type workload --prompts my_prompts.json

# Multi-model benchmarking
localbench benchmark "llama3.2:3b-gguf-q2-k" --type advanced \
    --secondary-models "tinyllama:1b-gguf-q4" "phi2:3b-gguf-q4"

# Export results
localbench benchmark "llama3.2:3b-gguf-q2-k" --json results.json

Status

Under active development. Support for additional frameworks is planned.

Roadmap

• Framework-agnostic benchmarking
• Additional performance metrics
• Enhanced visualizations
• Extended stability testing

Contributing

Issues and pull requests welcome. Do have a look at the existing ones first, though.