nearbits 0.3.0

Nearbits ternary-residual compression for Hugging Face models on low-RAM devices.

Nearbits

Nearbits is a Python package for compressing Hugging Face models with a ternary-residual strategy tuned for the lowest practical RAM footprint.

The default math is a post-training 1.58-bit style path for ordinary linear weights: a grouped ternary base {-1, 0, 1} with blended absmean/RMS scaling, plus a tiny VIP residual slice kept in 4-bit or 8-bit. Protected modules such as embeddings, norms-adjacent outputs, and heads stay in int8 for stability.

New Tech

Nearbits now uses a bitnet158 formula for ordinary projection layers:

W_hat = s_g * T(W_g, tau_g) + R_vip

Where:

• W_g is a weight group
• T(W_g, tau_g) is ternary quantization to {-1, 0, 1} with threshold tau_g
• s_g = alpha * mean(|W_g|) + (1 - alpha) * rms(W_g) is the blended group scale
• R_vip is a tiny residual correction made from only the top VIP weights, stored in 4-bit or 8-bit

Default values in this release:

alpha = 0.7
	au_g = 0.9 * mean(|W_g|)
vip_fraction = 0.001
max_vip_fraction = 0.01
vip_bits = 4
activation_bits = 8

This is designed to keep RAM low on arbitrary Hugging Face checkpoints while still producing usable output. It is inspired by BitNet-style ternary math, but this package is still a post-training approximation rather than native BitNet training.

What Nearbits does

• Compresses ordinary nn.Linear and Hugging Face Conv1D layers to a bitnet158-style ternary code path plus small VIP residuals
• Keeps only protected layers such as embeddings and heads in int8
• Applies optional per-token activation quantization in 8-bit during inference-style forward passes
• Works directly on existing Hugging Face models without retraining
• Exposes simple chat and prompt benchmarking helpers

Install

Install PyTorch first for your platform, then install Nearbits.

pip install torch
pip install nearbits

For local development from this repo:

pip install torch
pip install -e .

Quick Start

from nearbits import HybridLowBitConfig, chat, compress_hf_model, load_hf_model, model_report

model, tokenizer, model_kind = load_hf_model("Qwen/Qwen2.5-1.5B-Instruct")
print(model_report(model))

config = HybridLowBitConfig(
    group_size=128,
    bitnet_scale_blend=0.7,
    bitnet_threshold_factor=0.9,
    vip_fraction=0.001,
    max_vip_fraction=0.01,
    vip_bits=4,
    activation_bits=8,
)

compressed_model = compress_hf_model(model, config)
print(model_report(compressed_model))

reply = chat(compressed_model, tokenizer, "Who are you?")
print(reply)

Google Colab Usage

Use examples/google_collab_hybrid_lowbit.py as the runnable Colab example.

Typical Colab cells:

!pip install torch
!pip install nearbits

from nearbits import HybridLowBitConfig, benchmark_prompts, compress_hf_model, load_hf_model, model_report

MODEL_ID = "Qwen/Qwen2.5-1.5B-Instruct"
PROMPTS = [
    "hi",
    "hello",
    "who are you?",
    "what is 2 + 2?",
    "write one sentence about India.",
    "explain model compression in simple words.",
]

model, tokenizer, model_kind = load_hf_model(MODEL_ID)
print("Before:", model_report(model))

compressed_model = compress_hf_model(
    model,
    HybridLowBitConfig(
        vip_fraction=0.001,
        max_vip_fraction=0.01,
        vip_bits=4,
        activation_bits=8,
    ),
)
print("After:", model_report(compressed_model))

rows = benchmark_prompts(compressed_model, tokenizer, PROMPTS)
for row in rows:
    print(row["prompt"])
    print(row["response"])
    print(row["latency_sec"])

Public API

• load_hf_model(model_id, trust_remote_code=False)
• compress_hf_model(model, config=None)
• model_size_mb(model)
• model_report(model)
• chat(model, tokenizer, prompt, ...)
• benchmark_prompts(model, tokenizer, prompts, ...)
• HybridLowBitConfig(...)

Notes

• The public package name is nearbits
• The default ordinary-layer strategy is bitnet158, not whole-layer int8
• This is a post-training approximation for arbitrary Transformer checkpoints, not native BitNet training
• The strongest RAM reduction comes from the ternary base plus bounded VIP residuals

Publish New Version

Build and upload the new version to PyPI with:

python -m pip install --upgrade build twine
python -m build
python -m twine upload dist/*

If you want to publish only this version's fresh artifacts, clean dist first and then upload:

rmdir /s /q dist
python -m build
python -m twine upload dist/*