nearbits 0.2.0

Hybrid low-bit compression for Hugging Face models on low-RAM devices.

NearBit

NearBit is a Python package for compressing Hugging Face models with a single strategy: hybrid_lowbit.

The package keeps high-impact parts in int8, compresses the bulk of linear layers to binary or ternary, rescues outlier weights in int8, and targets the lowest practical RAM footprint without retraining.

What NearBit does

• Compresses nn.Linear layers with binary or ternary storage plus group-wise scales
• Keeps sensitive linear layers in int8
• Compresses nn.Embedding tables to int8
• Works directly on existing Hugging Face models
• Exposes simple chat and prompt benchmarking helpers

Install

Install PyTorch first for your platform, then install NearBit.

pip install torch
pip install nearbit

For local development from this repo:

pip install torch
pip install -e .

Quick Start

from nearbit import HybridLowBitConfig, chat, compress_hf_model, load_hf_model, model_report

model, tokenizer, model_kind = load_hf_model("Qwen/Qwen2.5-1.5B-Instruct")
print(model_report(model))

config = HybridLowBitConfig(
    group_size=128,
    binary_outlier_fraction=0.002,
    ternary_outlier_fraction=0.001,
    binary_nmse_threshold=0.045,
    ternary_nmse_threshold=0.02,
)

compressed_model = compress_hf_model(model, config)
print(model_report(compressed_model))

reply = chat(compressed_model, tokenizer, "Who are you?")
print(reply)

Google Colab Usage

Use examples/google_collab_hybrid_lowbit.py as the runnable Colab example.

Typical Colab cells:

!pip install torch
!pip install nearbit

from nearbit import HybridLowBitConfig, benchmark_prompts, compress_hf_model, load_hf_model, model_report

MODEL_ID = "Qwen/Qwen2.5-1.5B-Instruct"
PROMPTS = [
    "hi",
    "hello",
    "who are you?",
    "what is 2 + 2?",
    "write one sentence about India.",
    "explain model compression in simple words.",
]

model, tokenizer, model_kind = load_hf_model(MODEL_ID)
print("Before:", model_report(model))

compressed_model = compress_hf_model(model, HybridLowBitConfig())
print("After:", model_report(compressed_model))

rows = benchmark_prompts(compressed_model, tokenizer, PROMPTS)
for row in rows:
    print(row["prompt"])
    print(row["response"])
    print(row["latency_sec"])

Public API

• load_hf_model(model_id, trust_remote_code=False)
• compress_hf_model(model, config=None)
• model_size_mb(model)
• model_report(model)
• chat(model, tokenizer, prompt, ...)
• benchmark_prompts(model, tokenizer, prompts, ...)
• HybridLowBitConfig(...)

Notes

• NearBit is centered on one executable path: hybrid_lowbit
• There is no public int8 fallback API in the package surface
• Quality depends on the model architecture and compression thresholds
• The strongest RAM reduction comes from the hybrid path, not from pure int8