🧠 QuantLLM: Lightweight Library for Quantized LLM Fine-Tuning and Deployment

📌 Overview

QuantLLM is a Python library designed for developers, researchers, and teams who want to fine-tune and deploy large language models (LLMs) efficiently using 4-bit and 8-bit quantization techniques. It provides a modular and flexible framework for:

Loading and quantizing models with advanced configurations
LoRA / QLoRA-based fine-tuning with customizable parameters
Dataset management with preprocessing and splitting
Training and evaluation with comprehensive metrics
Model checkpointing and versioning
Hugging Face Hub integration for model sharing

The goal of QuantLLM is to democratize LLM training, especially in low-resource environments, while keeping the workflow intuitive, modular, and production-ready.

🎯 Key Features

Feature	Description
✅ Quantized Model Loading	Load any HuggingFace model in 4-bit or 8-bit precision with customizable quantization settings
✅ Advanced Dataset Management	Load, preprocess, and split datasets with flexible configurations
✅ LoRA / QLoRA Fine-Tuning	Memory-efficient fine-tuning with customizable LoRA parameters
✅ Comprehensive Training	Advanced training loop with mixed precision, gradient accumulation, and early stopping
✅ Model Evaluation	Flexible evaluation with custom metrics and batch processing
✅ Checkpoint Management	Save, resume, and manage training checkpoints with versioning
✅ Hub Integration	Push models and checkpoints to Hugging Face Hub with authentication
✅ Configuration Management	YAML/JSON config support for reproducible experiments
✅ Logging and Monitoring	Comprehensive logging and Weights & Biases integration

🚀 Getting Started

Installation

pip install quantllm

For detailed usage examples and API documentation, please refer to our:

💻 Hardware Requirements

Minimum Requirements

CPU: 4+ cores
RAM: 16GB
Storage: 20GB free space
Python: 3.8+

Recommended Requirements

GPU: NVIDIA GPU with 8GB+ VRAM
RAM: 32GB
Storage: 50GB+ SSD
CUDA: 11.7+

Resource Usage Guidelines

Model Size	4-bit (GPU RAM)	8-bit (GPU RAM)	CPU RAM (min)
3B params	~6GB	~9GB	16GB
7B params	~12GB	~18GB	32GB
13B params	~20GB	~32GB	64GB
70B params	~90GB	~140GB	256GB

🔄 Version Compatibility

QuantLLM	Python	PyTorch	Transformers	CUDA
0.1.x	≥3.8	≥2.0.0	≥4.30.0	≥11.7
0.2.x	≥3.9	≥2.1.0	≥4.31.0	≥11.8

🗺 Roadmap

Multi-GPU training support
AutoML for hyperparameter tuning
More quantization methods
Custom model architecture support
Enhanced logging and visualization
Model compression techniques
Deployment optimizations

🤝 Contributing

We welcome contributions! Please see our CONTRIBUTE.md for guidelines and setup instructions.

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

HuggingFace for their amazing Transformers library
bitsandbytes for quantization
PEFT for parameter-efficient fine-tuning
Weights & Biases for experiment tracking

📫 Contact & Support

GitHub Issues: Create an issue
Documentation: Read the docs
Discord: Join our community
Email: support@quantllm.ai

Release files for quantllm 1.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distribution (wheel)

Table of built distributions (wheels) for quantllm 1.0.0
File	Interpreter	ABI	Platform	Reset
quantllm-1.0.0-py3-none-any.whl 33.3 kB Apr 24, 2025	Python 3	none	any	Details

Release files / quantllm-1.0.0-py3-none-any.whl

Download URL	quantllm-1.0.0-py3-none-any.whl
Size	33.3 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`00325e5c6f89dcfded798a4b48a004fdaa815012295a5e4fcf9b01bddfe76588`
BLAKE2b-256 checksum How to use checksums	`a873f0280b14e9c0fca7319053490be97a4e63dc73f71e88e5b53840cc78160b`
Upload date	Apr 24, 2025
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.12.1`