modelbenchx 1.0.4

A lightweight toolkit for benchmarking ML model inference performance.

ModelBenchX

A lightweight, extensible toolkit for benchmarking ML model inference performance.

ModelBenchX is designed for ML engineers who need quick and reliable measurements of:

• Inference latency
• Throughput (requests/sec)
• Memory usage
• Model performance comparison
• Automated Markdown report generation

---

🚀 Installation

pip install modelbenchx

---

🧩 Requirements

Component Version

---

Python 3.9 -- 3.11 OS Linux, macOS, Windows Dependency psutil

ModelBenchX is continuously tested against:

• Python 3.9\
• Python 3.10\
• Python 3.11

---

⚡ Quick Example

from modelbench import benchmark_model
import numpy as np

def model(x):
    return x * 2

input_data = np.array([1, 2, 3])

results = benchmark_model(model, input_data, runs=200)

print(results)

---

🔍 Comparing Multiple Models

from modelbench import compare_models
import numpy as np
import time

def fast_model(x):
    return x

def slow_model(x):
    time.sleep(0.01)
    return x

models = {
    "FastModel": fast_model,
    "SlowModel": slow_model
}

input_data = np.array([1])

results = compare_models(models, input_data, runs=100)

for name, metrics in results.items():
    print(name, metrics)

---

📄 Generate Benchmark Report

from modelbench.report import generate_markdown_report

generate_markdown_report(results)

This creates:

benchmark_report.md

---

🛠 CLI Usage

Benchmark pickled models directly from the command line:

modelbench --model model.pkl --input input.pkl --runs 200

---

📐 Benchmarking Methodology

ModelBenchX follows a consistent benchmarking approach:

• Warm-up runs to reduce cold-start bias
• High-resolution timing using time.perf_counter()
• Memory tracking via process RSS measurement
• Throughput calculated as total runs / total execution time

⚠ Microsecond-level latency measurements may include Python loop overhead.
For production-critical benchmarking, batch benchmarking is recommended (planned in v1.1).

---

📦 Versioning & Stability

ModelBenchX follows Semantic Versioning:

• MAJOR → Breaking API changes
• MINOR → New features
• PATCH → Bug fixes & improvements

The public API (benchmark_model, compare_models) is considered stable as of v1.0.x.

---

🧪 Testing & CI

ModelBenchX includes automated unit tests covering:

• Core benchmarking engine\
• Model comparison logic\
• Metric structure validation

Continuous Integration runs on:

• Python 3.9\
• Python 3.10\
• Python 3.11

Run tests locally:

pip install -e .
pip install pytest
pytest

---

🎯 Ideal Use Cases

• Model optimization workflows
• Latency-sensitive systems
• Edge deployment benchmarking
• ML infrastructure experimentation
• Performance regression detection

---

🗺 Roadmap

v1.1 (Planned)

• Batch size benchmarking
• Peak memory tracking
• JSON + HTML reporting support
• Extended framework adapters (PyTorch / TensorFlow)

Future

• GPU benchmarking support
• Visualization utilities
• Async benchmarking
• Dockerized benchmarking mode

---

🤝 Contributing

Contributions are welcome.

Please read CONTRIBUTING.md before submitting pull requests.

---

📜 License

This project is licensed under the MIT License.