tinyport 0.1.0

Run big AI models on small hardware. Integer Descent guided model compression.

TinyPort

Run big AI models on small hardware.

TinyPort compresses AI models from HuggingFace or local files so they can run on phones, Raspberry Pi, laptops — any device with a processor.

It uses Integer Descent — an information-theoretic framework that measures how much real information each layer of a neural network is actually carrying. Layers with low information density (high "carry pressure") get compressed aggressively. Information-dense layers keep their precision. The result: smaller models that behave like larger ones.

Install

pip install tinyport

With HuggingFace support:

pip install tinyport[full]

Quickstart

Python

from tinyport import load_model, compress

# Load from HuggingFace Hub
weights = load_model("openai/whisper-tiny")

# Compress for Raspberry Pi 5
compressed = compress(weights, target="rpi5")

print(compressed.summary())
# Model:       openai/whisper-tiny
# Original:    151.0 MB (FP32)
# Compressed:  37.8 MB (4.0x smaller)
# Quality:     Good (INT4 quality, ~3-5% accuracy impact with ID-guidance)

CLI

# Analyze a model
tinyport analyze --model openai/whisper-tiny

# Compress for a phone
tinyport compress --model openai/whisper-tiny --target phone

# List all hardware targets
tinyport targets

Hardware Targets

Target	Device	Bits
server	Cloud GPU / data center	INT8
desktop	Gaming PC with GPU	INT6
laptop	MacBook / laptop	INT5
phone	Android / iPhone	INT4
rpi5	Raspberry Pi 5	INT4
rpi4	Raspberry Pi 4	INT4
rpi_zero	Raspberry Pi Zero 2W	INT4 (tiny models only)

How It Works

Standard quantization (like bitsandbytes, GPTQ) applies the same bit width to every layer. TinyPort profiles each layer individually using Integer Descent:

1. Profile: Scan every layer, compute its Integer Descent Compressibility Score (IDCS)
2. Allocate: High-IDCS layers (lots of free bits) → compress to INT4. Low-IDCS layers (information-dense) → keep INT6-8
3. Quantize: Block-wise quantization preserves local weight distribution
4. Export: Output a compressed file with deployment instructions

The result: comparable quality to uniform INT8 at INT4 size.

Running Compressed Models

TinyPort outputs compressed weights. To run them, use these battle-tested runtimes:

• LLMs on CPU/GPU: llama.cpp
• Vision/Audio on mobile: TFLite, MLC-LLM
• iPhone: CoreML
• Universal: ONNX Runtime

License

MIT