HF Inference Benchmark

Production-grade benchmarking infrastructure for HuggingFace inference workloads.

hf-inference-benchmark is a reproducible, device-agnostic benchmarking system that measures the real operational cost of running LLMs — latency, throughput, and memory — under production-like conditions.

It answers one critical question:

“Will this model crash my server — and how fast can it actually run?”

Why This Tool Exists

Most public HuggingFace benchmarking scripts:

• Measure only a single forward pass
• Ignore warmup behavior
• Ignore GPU synchronization
• Ignore memory allocator behavior
• Produce misleading results

This tool implements the same benchmarking discipline used by real ML infrastructure teams.

Benchmarking Pipeline

Model Load
   ↓
Warmup Passes
   ↓
Synchronized Execution
   ↓
Latency Profiling (P50 / P95 / Avg)
   ↓
Token Counting
   ↓
Throughput Calculation (tokens/sec)
   ↓
Peak Memory Tracking
   ↓
Structured JSON Export

Metric Definitions

User Installation

# From PyPI
pip install hf-inference-benchmark

# From Source (for Developers)
git clone https://github.com/rgb-99/hf-inference-benchmark.git
cd hf-inference-benchmark
pip install -e .

Basic Usage

# Run on CPU/GPU (Auto-detected)
hf-bench facebook/opt-125m

# Export results for the Reporting Suite
hf-bench gpt2 --tokens 64 --out results/gpt2_perf.json

Persisting Results

hf-bench facebook/opt-125m --tokens 64 --out results/opt125m.json

Example:

{
  "model": "facebook/opt-125m",
  "throughput": 54.36,
  "latency_p50": 878.83,
  "memory_mb": 797.56
}

Reproducible Benchmarking

For fair comparison:

• Fix prompt
• Fix token count
• Fix device
• Use warmup runs
• Compare structured JSON outputs

Platform Integration

This tool is part of the unified NLP infrastructure platform:

nlp-tool benchmark facebook/opt-125m
nlp-tool report results/opt125m.json

Roadmap

• Batch-size profiling
• Streaming generation benchmarks
• Multi-GPU scaling
• Energy-cost estimation
• CI-based regression tracking