edgegemma-8gb 0.1.0

A 4-bit quantized local inference wrapper for Gemma-2B optimized for 8GB VRAM.

EdgeGemma-8GB: Local LLM Inference & Evaluation Pipeline

An automated testing pipeline engineered to evaluate the classification capabilities of open-source Large Language Models running entirely locally within highly constrained hardware environments (8GB VRAM).

Overview

Deploying AI in edge computing environments requires bypassing external cloud APIs due to latency, privacy, or network connectivity constraints. This project demonstrates a robust pipeline for loading, quantizing, and evaluating google/gemma-2b-it on consumer-grade hardware.

Architecture & Optimizations

• Memory Management: Utilizes NF4 (4-bit) quantization via bitsandbytes to aggressively compress the model weights, allowing a 2B parameter model to run comfortably inside 8GB of VRAM alongside the OS overhead.
• Singleton Inference Engine: The LLM loader is modularized to prevent double-allocation in Jupyter environments, protecting against GPU out-of-memory (OOM) crashes.
• Deterministic Evaluation: Inference is locked to temperature=0 with gradient tracking disabled (torch.no_grad()) to ensure absolute consistency and speed during zero-shot and few-shot classification benchmarking.

Setup

1. Clone the repository.
2. Install dependencies: pip install -r requirements.txt
3. Run the evaluation pipeline via notebooks/Task1_Prompting.ipynb. Data caching and API inference are handled entirely locally.