tinyedge 0.2.4

Benchmark your models (latency + accuracy) on real edge devices — SDK + tinydevice CLI

tinyedge — the Python SDK

Benchmark your models on real edge devices from inside any Python pipeline, notebook or CI job. PyTorch (or any framework) stays on your machine — the SDK converts what you give it into the platform's wire formats (ONNX + labeled image archive) and real hardware does the measuring.

pip install git+https://github.com/lienertdemaeyer/tinyedge-agent#subdirectory=sdk
export TINYEDGE_API_KEY=tinyedge_sk_…        # TinyEdge console → New benchmark

In a PyTorch pipeline

import tinyedge

client = tinyedge.TinyEdge()

result = client.benchmark(
    model,                       # a live nn.Module (auto-exported to ONNX) or "model.onnx"
    devices=["oppo-a74"],        # or ["jetson-orin-nano:tensorrt", "raspberry-pi-5"]
    dataset=raw_test_set,        # torch Dataset of (image, label) — UNtransformed,
                                 # or a folder ("./testset"), or an archive (".zip/.tar.gz")
    precision="fp32",
)

print(result)                    # <BenchmarkResult oppo-a74 completed p50=56.6ms top1=83.3%>
print(result.latency_ms_p50, result.accuracy_top1)

Notes that make this work well:

• Pass the dataset without transforms. Preprocessing (resize/crop/normalize) is part of the standardized job spec and runs on-device — that's what makes accuracy comparable across devices instead of depending on whatever transform happened to run on your laptop.
• Labels are class indices. Folder names / integer labels map directly to your model's output indices (207/ = output neuron 207).
• A custom example_input controls the ONNX export shape: client.benchmark(model, ..., example_input=torch.randn(1, 3, 320, 320)).

As a CI gate (pytest)

Fail the build when the model regresses on the hardware you ship on:

# test_edge_performance.py
import tinyedge

def test_detector_meets_edge_budget():
    client = tinyedge.TinyEdge()
    result = client.benchmark("artifacts/model.onnx",
                              devices=["jetson-orin-nano"],
                              dataset="testdata/eval_set.tar.gz")
    result.assert_latency(max_ms=50)
    result.assert_accuracy(min_top1=0.80)

assert_* raise AssertionError with a readable message, so any test runner (pytest, unittest, GitHub Actions) reports it natively.

Async style

jobs = client.benchmark(model, devices=["oppo-a74", "raspberry-pi-5"], wait=False)
# … do other work …
for j in jobs:
    print(client.get(j.id))

What runs where

Your machine (SDK)	TinyEdge platform	The device (agent)
torch → ONNX export	stores artifacts, queues job	downloads ONNX + images
dataset → image archive	tracks pending/running	runs inference, computes accuracy
poll / asserts	stores the report	uploads metrics only