Skip to main content

amazingvmsloth

Blazing-fast LLM fine-tuning with minimal VRAM.

  \   / |    amazingvmsloth - Fast LLM Fine-Tuning
   O^O / \_/ \   Minimal VRAM. Maximum Speed.
  \        /
   "-____-"

Train 14B models on a 4GB GPU. Multi-GPU, 4-bit quantization, LoRA, CPU offloading, gradient checkpointing, and sequence packing — all built for speed on consumer hardware.


Install

pip install amazingvmsloth

Or from source:

git clone https://github.com/CollabVMgamez/amazingvmsloth.git
cd amazingvmsloth
pip install -e .

Requirements: Python 3.9+, PyTorch 2.1+, CUDA 11.8+ (optional, CPU training supported)


Quick Start

1. Wizard — let it pick settings for your hardware

amazingvmsloth wizard --model Qwen/Qwen2.5-0.5B

Analyzes your GPU/CPU and prints a ready-to-run command.

2. Train

amazingvmsloth train \
  --model Qwen/Qwen2.5-0.5B \
  --dataset tatsu-lab/alpaca \
  --epochs 3 \
  --batch-size 2 \
  --grad-accum 4 \
  --lora-r 16 \
  --output-dir ./output

Supports chat-format datasets too:

amazingvmsloth train \
  --model Qwen/Qwen2.5-0.5B \
  --dataset angrygiraffe/claude-opus-4.6-4.7-reasoning-8.7k \
  --dataset-format chat \
  --max-samples 1000 \
  --output-dir ./thinking_lora

3. Convert LoRA to merged model

amazingvmsloth merge \
  --model Qwen/Qwen2.5-0.5B \
  --lora ./output \
  --output ./merged_model

4. Run inference

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("./merged_model", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")

messages = [{"role": "user", "content": "What is 2+2?"}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0]))

CLI Commands

Command Description
wizard Interactive config generator based on your hardware
train Fine-tune a model with LoRA
merge Merge LoRA weights into base model
convert Merge + convert to GGUF (requires llama.cpp)
info Show model info and VRAM estimates
bench Benchmark vs unsloth

Hardware Tiers

GPU VRAM Strategy
4-6 GB 4-bit quant, batch=1, grad accum=8, seq=512, tiny LoRA
6-12 GB 4-bit quant, batch=1-2, grad accum=4, seq=1024
12-24 GB 4-bit or full precision, batch=2-4, torch.compile
24+ GB Full precision, no grad checkpointing, large batch
CPU only fp32/bf16, torch.compile, physical-core threading

Key Features

  • Model Architectures: Qwen 3.5 (e.g. Qwen/Qwen3.5-0.8B), Gemma 3 (1B/4B), Gemma 2, LLaMA 3.2/3.1, Mistral, DeepSeek v2/v3/R1, and multimodal/TTS models
  • RTX 3050 & Low-VRAM Tuned: Ampere TensorFloat-32 (TF32) acceleration, fp16 Tensor Core speedup, non-blocking CUDA transfers, and automated 4-bit memory management
  • rsLoRA scaling for stable training at any rank
  • 4-bit/8-bit quantization via bitsandbytes
  • XFormers/SDPA attention patching (Flash Attention on Linux)
  • Sequence packing for 2-3x throughput
  • Gradient checkpointing with selective layer skipping
  • Multi-GPU: DDP, FSDP, DeepSpeed, pipeline parallelism
  • Layer offloading via accelerate.dispatch_model
  • CPU training with IPEX, pre-packing, torch.compile
  • PagedAdamW8bit optimizer for low-VRAM training
  • Checkpoint resume with full RNG/optimizer state
  • Tqdm progress bar with live loss + VRAM display

Example: 500 Steps on Dolly

amazingvmsloth train \
  --model Qwen/Qwen2.5-0.5B \
  --dataset databricks/databricks-dolly-15k \
  --dataset-format alpaca \
  --epochs 1 --batch-size 2 --grad-accum 2 \
  --max-samples 1000 --max-seq-length 512 \
  --lora-r 16 --output-dir ./dolly_lora --packing

This runs ~500 steps in ~10 minutes on a 4GB RTX 3050.


Project Structure

amazingvmsloth/
├── lora.py              # LoRA with rsLoRA, device-aware init
├── quantization.py      # 4-bit/8-bit quant, kbit training prep
├── attention.py         # SDPA/XFormers patching
├── trainer.py           # AmazingTrainer with tqdm, packing, offloading
├── cpu_trainer.py       # CpuTrainer for CPU-only training
├── packing.py           # Sequence packing collators
├── gradient.py          # GradientAccumulator
├── optimizer.py         # PagedAdamW8bit, CpuOffloadedAdamW
├── offload.py           # Layer offloading via accelerate
├── cli.py               # CLI entrypoint
├── wizard.py            # Hardware-aware config generator
├── bench.py             # Benchmark vs unsloth
└── utils/
    ├── banner.py        # Startup banner with GPU info
    ├── memory.py        # VRAM estimation
    ├── patching.py      # LoRA save/load helpers
    └── save_load.py     # Model save/merge

Benchmarks

On RTX 3050 4GB Laptop GPU:

Library Time (1 epoch, 500 samples) Peak VRAM
amazingvmsloth 5.3s 1.07 GB
unsloth 10.1s 0.96 GB

1.91x faster on small runs with pre-quantized models.


License

MIT

Metadata

Release files for amazingvmsloth 0.7.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for amazingvmsloth 0.7.4
File Size Uploaded Provenance Actions
amazingvmsloth-0.7.4.tar.gz 104.9 kB Without provenance Details for amazingvmsloth-0.7.4.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for amazingvmsloth 0.7.4
File Interpreter ABI Platform Actions
amazingvmsloth-0.7.4-py3-none-any.whl Python 3 none any Details for amazingvmsloth-0.7.4-py3-none-any.whl

Total release size: 210.9 kB

Release files / amazingvmsloth-0.7.4.tar.gz

Download URL amazingvmsloth-0.7.4.tar.gz
Size 104.9 kB
Tags Source
SHA-256 checksum
How to use checksums
88ea0a81819b7dede02c640ca918033efd1b44e6b716b393b9b2672eedf2bdd7
BLAKE2b-256 checksum
How to use checksums
2f2f391b3578c605ebfcfcf5458e79fafe50263e876e842467e579a485441441
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.10

Release files / amazingvmsloth-0.7.4-py3-none-any.whl

Download URL amazingvmsloth-0.7.4-py3-none-any.whl
Size 106.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
62860482abd086644da8cd3e5d7b93d39c3465bd7cd43f12b77bd7f3fff0f5a2
BLAKE2b-256 checksum
How to use checksums
2c2333e3f5af2b327cefe27dc296ccc9b1ed7814aebe5424f0af75873a630b47
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.10

Security for amazingvmsloth 0.7.4

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.7.4 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page