benchmarkingcli 0.0.3

CLI tool for running memory latency benchmarks on Android devices

BenchmarkingCLI

A command-line tool for running latency benchmarks on Android devices. This tool automates the process of pushing benchmark binaries to connected Android devices, executing latency tests, and collecting performance metrics in CSV format.

Features

• Latency Testing: Measure memory latency across different buffer sizes on a specific core
• Loaded Latency Testing: Measure latency under load while bandwidth threads are concurrently running
• Kernel Mode Selection: Choose between SUM, COPY, UPDATE, and TRIAD bandwidth kernels
• Automated Device Management: Automatic ADB device detection and binary deployment
• CSV Export: Results are automatically parsed and exported to timestamped CSV files
• CLI Interface: Easy-to-use command-line interface with Typer

Requirements

System Requirements

• Android Debug Bridge (ADB): Version 1.0.40+
  • Install via: apt-get install android-tools-adb (Linux) or brew install android-platform-tools (macOS)
  • Verify installation: adb version
  • Ensure your Android device has USB debugging enabled
  • Device must be connected and appear in adb devices

Python Requirements

• Python: 3.8 or higher
• Dependencies:
  • typer>=0.9.0 - CLI framework

Installation

1. Clone the repository:

   git clone https://git.truminds.co.in/truminds/performance-benchmarking/benchmarkingcli.git
   cd benchmarkingcli
   

2. Install Python dependencies:

   pip install -r requirements.txt
   

3. Verify ADB is installed and connected:

   adb devices
   

   You should see your connected device listed.

4. Place the benchmark binary:

   • Copy the latency_test binary to the helper/ directory
   • The script will automatically push it to /data/local/tmp/ on the device

Usage

Latency Test

Run a basic latency benchmark on a specific core:

# Run on default core (7)
python main.py latency

# Run on a specific core
python main.py latency --core 4

This command will:

1. Check for connected ADB devices
2. Push the benchmark binary to the device
3. Execute the latency test on the specified core
4. Parse results and save to latency_results_<TIMESTAMP>.csv

Loaded Latency Test

Run a latency test with concurrent bandwidth load:

# Default — TRIAD kernel, core 7 for latency, cores 0-5 for bandwidth
python main.py loadedlatency \
  --lat-buffer-mb 128 \
  --bw-buffer-mb 128 \
  --latency-core 7 \
  --bw-cores 0,1,2,3,4,5 \
  --kernel-mode TRIAD

# SUM kernel (lightest — 1 array, read only)
python main.py loadedlatency \
  --lat-buffer-mb 128 \
  --bw-buffer-mb 128 \
  --latency-core 7 \
  --bw-cores 0,1,2,3,4,5 \
  --kernel-mode SUM

# COPY kernel (2 arrays — read + write)
python main.py loadedlatency \
  --lat-buffer-mb 128 \
  --bw-buffer-mb 128 \
  --latency-core 7 \
  --bw-cores 0,1,2,3 \
  --kernel-mode COPY

# UPDATE kernel (2 arrays — read + write with scalar multiply)
python main.py loadedlatency \
  --lat-buffer-mb 256 \
  --bw-buffer-mb 256 \
  --latency-core 7 \
  --bw-cores 0,1,2,3,4,5 \
  --kernel-mode UPDATE

# Numeric mode values also accepted (1=SUM, 2=COPY, 3=UPDATE, 4=TRIAD)
python main.py loadedlatency \
  --lat-buffer-mb 128 \
  --bw-buffer-mb 128 \
  --latency-core 7 \
  --bw-cores 0,1,2,3,4,5 \
  --kernel-mode 4

Parameters:

• --lat-buffer-mb: Buffer size (MB) for the latency thread (default: 128)
• --bw-buffer-mb: Buffer size (MB) for each bandwidth thread (default: 128)
• --latency-core: CPU core to run the latency thread on (default: 7)
• --bw-cores: Comma-separated list of cores for bandwidth threads (default: 0,1,2,3,4,5)
• --kernel-mode: Bandwidth kernel to use — see table below (default: TRIAD)

Kernel Modes

Mode	Value	Arrays	Description
SUM	1	1	Read-only sum — lightest BW pressure
COPY	2	2	Read + write — moderate BW pressure
UPDATE	3	2	Read + scalar write — moderate BW pressure
TRIAD	4	3	Read 2 + write 1 — heaviest BW pressure

Both the name (TRIAD) and numeric value (4) are accepted interchangeably.

Output

Results are exported as CSV files with the following columns:

• buffer_size_kb: Size of the buffer tested (in KB)
• min_ns: Minimum latency (nanoseconds)
• mean_ns: Mean latency (nanoseconds)
• max_ns: Maximum latency (nanoseconds)

Example output files:

• latency_results_20260427_115540.csv
• loaded_latency_results_20260427_115540.csv

Troubleshooting

"No ADB devices found"

• Check that your Android device is connected: adb devices
• Ensure USB debugging is enabled on the device
• Try: adb kill-server then adb start-server

"Latency test binary not found"

• Ensure the latency_test binary is placed in the helper/ directory
• Verify the file exists: ls -la helper/latency_test

Binary push fails

• Check device storage: adb shell df /data/local/tmp
• Verify ADB permissions: adb shell ls -la /data/local/tmp

"Invalid kernel mode"

• Use one of: SUM, COPY, UPDATE, TRIAD or their numeric equivalents 1, 2, 3, 4

Project Structure

benchmarkingcli/
├── main.py                              # CLI entry point
├── latency.py                           # Latency test runner
├── loaded_latency.py                    # Loaded latency test runner
├── parse_latency_result.py              # Result parsing and CSV export
├── requirements.txt                     # Python dependencies
├── helper/
│   ├── binary_helper.py                 # ADB and binary management
│   └── latency_test                     # Compiled benchmark binary
└── README.md                            # This file

Support

For issues, questions, or contributions, please contact the development team or submit an issue through the project repository.