🌟 AuraLite AI

Quickstart · FAQ · Quantization

AuraLite AI is a lightweight, educational Large Language Model (LLM) implemented using PyTorch. It is designed to demonstrate the inner workings of the Transformer architecture (the foundation of models like GPT-4) in a way that is accessible and runnable on consumer hardware.

⚡ Quickstart

pip install "auralite-ai[serve]"     # Python 3.11–3.14, from PyPI (2.18.0 on); in a clone: pip install -e '.[serve]'
# or a tag straight from GitHub: pip install 'auralite-ai[serve] @ git+https://github.com/AlexanderNyr/AuraLite-AI@v2.18.1'
auralite train --text corpus.txt --model model.pt
auralite generate --model model.pt --prompt "Once upon a time" --length 80
auralite serve --model model.pt      # OpenAI-compatible API on http://127.0.0.1:8000
auralite export --model model.pt     # model.pt2 — runs with plain PyTorch (torch.export)
auralite image-train --images photos/ --preset small --model gen.pt   # your own image generator
auralite image-train --images photos/ --type gan --preset tiny --model gan.pt   # or flow / gan
auralite image-generate --model gen.pt -n 16 --out grid.png
auralite image-eval --model gen.pt --images photos/          # quality score (SWD), no pretrained net
auralite tokenizer train --text corpus.txt --vocab-size 4096 --out tok.json   # only the tokenizer
auralite train --text corpus.txt --tokenizer tok.json --model model.pt        # or a tokenizer.json
auralite train --text books/ "notes/*.md" --model model.pt          # folders and globs (2.15)
auralite train --text chats.jsonl --resume model.pt --model chat.pt # fine-tune a chat model (2.15)
auralite chat --model chat.pt                                       # it answers and stops by itself
auralite chat-eval --model chat.pt --data held_out.jsonl            # stop rate, exact replies (2.16)
auralite train --text chats.jsonl --resume model.pt --lora 8 --save-adapter adapter.pt --model lora.pt
auralite lora apply --base model.pt --adapter adapter.pt --merge --out chat2.pt   # LoRA (2.16)
auralite distill --teacher teacher.gguf --prompts questions.txt --out chats.jsonl  # a teacher's dialogs (2.16)
auralite train --data pairs.jsonl --resume chat.pt --method dpo --model dpo.pt     # DPO / SimPO (2.16)
auralite chat --model chat.pt --json-schema person.json --prompt "Anna, 34, Riga"  # valid JSON (2.17)
auralite data prepare corpus/ --out data/corpus --vocab-size 8192    # big data: token shards on disk (2.18)
auralite data prepare hf:HuggingFaceFW/fineweb-edu:sample-10BT --tokenizer hf:HuggingFaceTB/SmolLM2-135M --max-tokens 2000000000 --out data/fineweb
auralite plan --preset 350m --memory 16GB                            # will it fit? (2.18)
auralite train --data data/fineweb --preset 125m --max-tokens 2000000000 --memory-plan auto --autosave run.pt --model big.pt
auralite train --data data/fineweb --resume run.pt --continue-run --model big.pt   # after a crash or a stop (2.18)
auralite import hf:HuggingFaceTB/SmolLM2-135M-Instruct --out smol.pt    # Llama / Mistral / Qwen2 / Qwen3 into AuraLite (2.18.1)
auralite train --data chats.jsonl --resume hf:Qwen/Qwen2.5-0.5B-Instruct --quantize nf4 --lora 16 --memory-plan auto --model qwen-chat.pt   # QLoRA, also on a CPU
auralite export --model qwen-chat.pt --format gguf --quant q8_0 --out qwen-chat.gguf   # llama.cpp / Ollama (or --format hf)
auralite-gui                         # desktop app: train, chat, quantize, evaluate, export

from auralite import AuraLiteEngine

engine = AuraLiteEngine()
engine.train(open("corpus.txt", encoding="utf-8").read(), {"epochs": 3})
print(engine.generate("Once upon a time", length=80))

Everything lives in one package, auralite (auralite.engine, auralite.server, auralite.quantization, auralite.export, …). Extras: [serve] server, [hf] Hugging Face + LoRA, [export] ONNX, [torchao] future-proof INT8, [gguf] llama.cpp — see Installation below.

⚡ v2.6 Modern Training Stack

Three upgrades straight from 2024–2025 frontier practice:

• QK-norm (use_qk_norm) — per-head RMS normalization of queries/keys before RoPE. Popularized by ViT-22B (Dehghani et al., 2023) and now standard in OLMo-2, Gemma-2/3, Qwen3 and GLM. Caps attention-logit growth and stabilizes training at higher learning rates.
• Muon optimizer (optimizer="muon") — MomentUm Orthogonalized by Newton–Schulz (Keller Jordan et al., 2024; used in Moonlight / Kimi K2 and every current speedrun). The hidden 2-D weight matrices of the transformer body are updated with a Newton–Schulz-orthogonalized momentum step at a shape-aware LR (match_rms_adamw mode transfers AdamW-tuned LRs). Embeddings, norm scales and an untied LM head automatically stay on a companion AdamW group (split_parameters_for_muon). AdamW remains available (optimizer="adamw") and is auto-selected for LoRA runs.
• 2.12: one default recipe, Muon + QK-norm + WSD + early stopping, for train(), the GUI and the CLI alike (train() and the CLI used AdamW without QK-norm). With early stopping on a 100 KB text Muon + QK-norm reached 2.196 bits per byte against 2.308 for AdamW + QK-norm (three seeds); on a megabyte it was 0.3 % better per step and ties at equal time, because a Muon step costs ~21 % more on a CPU. Numbers: docs/training.md § 8. "optimizer": "adamw" gives the fastest steps.
• 2.13: honest validation, full-speed generation on every thread. A new BPE tokenizer is fitted without the validation text (2.12 learned the held-out words too, which made validation up to 9 % optimistic for large vocabularies); an early stop ends with a short learning-rate decay (1.2 % better on tinyshakespeare, three seeds); every generation forward pass runs on one compute thread, so the server and the GUI decode as fast as the main thread. Numbers: docs/training.md § 6, 8, 10 and docs/engine.md § 10.
• WSD scheduler (lr_schedule="wsd", now the default) — Warmup-Stable-Decay trapezoidal schedule (MiniCPM 2024, DeepSeek-V3-style multi-phase training). Linear warmup → constant LR until wsd_stable_ratio (default 80%) → short decay (cosine / linear / sqrt) to wsd_min_lr_ratio. Matches or beats cosine convergence and lets you branch a decay run from any stable-phase checkpoint. Pure cosine remains available via lr_schedule="cosine".

engine.train(text, {
    "use_qk_norm": True,        # QK-norm per attention head
    "optimizer": "muon",        # Muon for hidden matrices
    "muon_lr": 0.02,
    "lr_schedule": "wsd",       # Warmup-Stable-Decay (default)
    "wsd_stable_ratio": 0.8,
    "wsd_decay": "cosine",
    ...
})

🚀 v2.4 Production-Grade Architecture Update

AuraLite now keeps the original educational single-file entry points and adds a production-oriented package layout:

graph TD
  Engine[AuraLiteEngine] --> Torch[ModernTransformer]
  Engine --> GGUF[GGUFBackend]
  Engine --> HF[HFBackend]
  Torch --> Blocks[TransformerBlock]
  Blocks --> Attention[RoPE + GQA + Sliding Window KV Cache]
  Blocks --> FFN[SwiGLU / optional Top-2 MoE]
  Engine --> Quant[Quantization: GPTQ/AWQ/HQQ/FP8]
  Engine --> RAG[RAG: web + local vector store]
  Server[FastAPI OpenAI Server] --> Engine

New high-impact capabilities

• LLaMA-compatible RoPE: rotate_half formulation, exact inverse-frequency formula, and improved Linear / Dynamic-NTK / YaRN scaling.
• Hardened GQA KV-cache: per-layer caches keep unrepeated KV heads, support sliding-window eviction, and optional FP8/INT8 cache storage.
• Research architecture flags: sliding_window, use_moe, num_experts, use_flex_attention, kv_cache_dtype, tie_word_embeddings.
• Untied input/output embeddings (default): the token embedding (read/context) and the LM head (write/classification) are two mathematically distinct layers, kept as independent parameters by default. Sharing one matrix forces it to both cluster synonyms (input) and separate them (output); the opposing gradients partially cancel and slow convergence. Legacy weight tying remains an explicit opt-in via tie_word_embeddings=True, model.tie_weights() / model.untie_weights().
• One package (2.7.0): everything is importable from auralite — from auralite import AuraLiteEngine, auralite.engine.layers, auralite.engine.model, auralite.server, … The flat names of 2.6 (model_engine, server, …) still work inside a source checkout until 3.0, with a DeprecationWarning.
• Serving: auralite/server/openai_server.py exposes /v1/completions, /v1/chat/completions, and /health.
• RAG upgrade: optional persistent vector store, semantic chunking, HyDE query expansion, and citation-style [source: ...] context.
• Quantization upgrade: HQQ, optional FP8, AWQ alpha/clip grid search, Cholesky-stabilized GPTQ.

Example: train with modern options

from auralite import AuraLiteEngine

engine = AuraLiteEngine()
engine.train(text, {
    "tokenizer": "bpe",
    "bpe_vocab_size": 4096,
    "d_model": 768,
    "n_heads": 12,
    "n_kv_heads": 4,
    "n_layers": 12,
    "d_ff": 2048,
    "seq_length": 2048,
    "sliding_window": 1024,
    "rope_scaling": {"type": "yarn", "factor": 4.0, "original_max_position_embeddings": 2048},
    "use_moe": False,
    "use_compile": True,
})
engine.save_model("auralite-v24.pt")

Example: serve OpenAI-compatible API

AURALITE_MODEL=auralite-v24.pt uvicorn auralite.server.openai_server:app --host 0.0.0.0 --port 8000
# or: auralite serve --model auralite-v24.pt

🚀 Key Features

• 🤗 Pretrained models in AuraLite's own engine (2.18.1): auralite import hf:ORG/NAME reads Llama (2/3/3.1/3.2), Mistral, Qwen2/2.5 and Qwen3 checkpoints into a native model — no transformers — with logits bit-identical to transformers (SmolLM2-135M), downloads that resume and are checked against the Hub's sha256. QLoRA without bitsandbytes (--quantize nf4: 4-bit NormalFloat weights with double quantization, LoRA adapters on top) runs on any CPU PyTorch runs on, AVX-only ones included, and on GPUs; 8-bit AdamW (--optimizer adamw8bit) keeps a quarter of AdamW's optimizer memory. The result exports back as a Hugging Face folder (transformers 4.46 and 5.x load it, same logits) or a GGUF file for llama.cpp, Ollama and LM Studio (F16/Q8_0/BF16/F32, the tokenizer and chat template included). Measured runs and honest CPU speeds for 0.5B–8B models: docs/training.md § 21.
• 🧩 JSON mode, structured outputs and tools (2.17): the server speaks OpenAI's response_format (json_object, json_schema) and tools / tool_choice, and Ollama's format and tools — the official openai client works as is, client.chat.completions.parse with a Pydantic model and pydantic_function_tool included (examples/openai_structured.py). A native model writes the answer under a JSON grammar (constrained decoding) that also keeps room for the shortest valid ending: every answer parses and follows the schema, even from an untrained model (200 of 200 per case, where 0 of 50 parsed without the grammar), at no measurable cost in speed. GGUF models get the schema as a llama.cpp grammar; Qwen2.5-0.5B called the right tool or answered in text in 12 of 16 prompts. From Python: engine.generate_json(messages, schema), engine.chat_with_tools(messages, tools) (docs/engine.md § 12).
• 💬 Chat fine-tuning (2.15): a .jsonl file of dialogs (OpenAI messages, ShareGPT, Alpaca or prompt/completion records) fine-tunes the model in memory, a checkpoint (--resume, optionally with --lora) or a new model into a chat model. The dialogs are written with the chat template exactly as generate_chat prompts (ChatML by default), the loss learns only the assistant's replies and the marker that ends them, and the markers become special tokens — so the model answers in the template and stops by itself (300 of 300 replies in the measured runs, docs/training.md § 15). Training data can also be several files, folders and globs: documents are joined with blank lines and validated on whole held-out documents, and one file trains exactly as before (§ 14). 2.16: dialog runs train for every epoch asked for and learn the prompt with a small weight (0.1) — 86 % exact replies on the benchmark (2.15: 76 % and 64 % on two of the three seeds) — LoRA adapts every linear layer at ten times the learning rate (73 % instead of 57 %), merges into the weights and saves to a file of its own (--save-adapter, auralite lora), and auralite chat-eval measures a chat model on held-out dialogs (§ 16). auralite distill has a teacher model (a GGUF file, a llama.cpp server, a Hugging Face model or an AuraLite checkpoint) answer your questions and writes its replies as dialogs — an interrupted run resumes, the teacher's license goes into the metadata (§ 17) — and preference pairs train a chat model with DPO (DPO-Positive by default) or SimPO: on the arithmetic benchmark the chat models' own mistakes, as pairs, raised their exact answers from 67.5 % to 74.7 % (three seeds, § 18).
• 🖼️ From-Scratch Image Generator: the Image tab, auralite image-train / auralite image-generate and ImageGenEngine train your own image generator on a folder of images — no pretrained weights. 2.14.2: three model types — denoising diffusion, flow matching (rectified flow, the objective of Stable Diffusion 3: a 2.5× lower FID than diffusion at the same budget on MNIST) and a modern GAN after R3GAN (relativistic loss with R1 + R2, ResNet generator/discriminator without normalisation, EMA generator, truncation trick, optional DiffAugment with ADA's adaptive probability; one network call per image) — plus a quality score that needs no pretrained network (auralite image-eval, the patch sliced Wasserstein distance of ProGAN) and progress for every stage on one bar. A compact self-attention U-Net with presets (Tiny/Small/Medium/Large), ε / x0 / v-prediction, linear / cosine / sigmoid noise schedules (rescaled so they end in pure noise for any step count), Min-SNR-γ, EMA with warmup, AMP, gradient accumulation & checkpointing, channels-last on CPU, warmup+cosine LR, h-flip augmentation, seeding, continue-training and autosave. Class-conditional training from sub-folders with classifier-free guidance, a deterministic held-out validation loss with early stopping, and DPM-Solver++(2M) (good samples in 10–20 steps) next to DDIM / DDPM. Live loss/val plot, preview grid, seeded output, save samples (+ grid), .pt / .safetensors checkpoints. 2.10 made a training step 1.5–2.6× faster on CPU and fixed the noise schedule, EMA and continue-training — see docs/image.md. Requires the optional Pillow dependency.
• PyTorch Engine: Professional-grade tensor operations, autograd and optimization.
• GGUF / llama.cpp Inference: Load .gguf models directly (quantized Llama/Mistral/Qwen/etc.) via llama-cpp-python for generation, streaming, batch prompts, Thinking Mode and web-context prompting.
• Modern Transformer Architecture (LLaMA-style): A decoder-only transformer with RMSNorm (pre-norm), RoPE (Rotary Position Embeddings), SwiGLU feed-forward, optional GQA (Grouped-Query Attention), untied input/output embeddings by default (optional weight tying via tie_word_embeddings=True) and a KV-cache for fast generation.
• Flash Attention: Uses PyTorch scaled_dot_product_attention (fused / memory-efficient kernels) instead of a hand-rolled softmax.
• BPE Tokenizer (built-in): A self-contained mini-BPE trained on your corpus (configurable vocab size), switchable to classic character-level tokenization. BPE dramatically improves text quality — the model learns sub-words instead of single letters.
• Validation Split: A held-out fraction of the text is evaluated every epoch, spread over the whole split. Early stop (on by default since 2.12, 3 epochs) ends a run whose val loss stopped improving and keeps the best epoch's weights; since 2.13 it first decays the learning rate for up to two epochs. A new BPE tokenizer never sees the validation text (2.13).
• Ending a run well (2.12): Stop (Ctrl+C in the CLI) finishes with a short learning-rate decay instead of cutting the run at the peak learning rate, a second press stops at once; a Time limit sizes the schedule so the decay ends on time (docs/training.md § 9).
• torch.compile (optional): Compiles the training step before the first batch (16–20 s on CPU with an empty cache) and falls back to eager if that fails. On a 2-thread CPU it no longer beats the eager loop, so measure before relying on it.
• Continue Training: Fine-tune the model currently in memory (trained or loaded) on a new file instead of starting from scratch — on text, or on dialogs to make it a chat model (2.15).
• Autosave: Optional checkpoint autosave every N epochs — never lose a long run.
• Hardware Acceleration: Automatic detection and usage of NVIDIA CUDA (GPU) for training and generation, with a seamless fallback to CPU.
• Full CPU Multithreading: Automatically configures PyTorch (and the OpenMP/MKL backends) to use all available CPU cores. Batches are gathered in the training process (one indexing operation each), so no worker processes are started.
• Mini-Batch Training: Training runs on shuffled mini-batches of non-overlapping windows (configurable Batch Size); one epoch is one pass over the text, with a fresh window offset every epoch.
• Advanced GUI: A comprehensive control panel built with tkinter that allows real-time interaction with the model.
• Hyperparameter Tuning: Full control over the AI's "brain" directly from the interface:
  • Learning Rate: Controls how fast the model adapts to new data.
  • Epochs: Determines how many times the model studies the dataset.
  • Model Dimension (D_Model): Sets the size of the internal vector representations.
  • Feed-Forward Dimension (D_FF): Controls the capacity of the processing layers.
  • Heads (N_Heads) / Layers (N_Layers): Shape of the attention mechanism and network depth.
  • Context Window (Seq Length): Defines how many previous characters the AI considers when predicting the next one.
  • Batch Size: Number of samples processed per optimizer step. Larger values use more memory but better utilize multiple CPU cores / the GPU.
  • Dropout / Grad Clip: Regularization and training-stability controls.
• Sampling Controls: Temperature, Top-K, Top-P (nucleus) and Repetition Penalty for generation.
• Dense Next-Token Loss: The loss is computed over every position of the context window (nanoGPT-style), making training far more sample-efficient than last-position-only prediction.
• Tabbed GUI: Three clean tabs — 🏋️ Training (hyperparameters, tokenizer options, loss history), ✨ Generation (sampling + prompt + output), 💾 Model (save / load + full model info).
• 🧠 Thinking Mode (NEW): Two-pass generation — the model first free-writes a higher-temperature draft ("thoughts"), then the final answer is generated conditioned on that draft (self-conditioning). No retraining needed; works with any existing checkpoint. The GUI shows the thinking block and the final answer separately.
• 🌐 Web Search / mini-RAG (NEW): Optional DuckDuckGo search (no API key, stdlib-only) — top result snippets are injected into the prompt as retrieval context before generation. Can be combined with Thinking Mode. If the search fails (offline), generation gracefully continues without it.
• ⚡ Model Quantization (NEW v2.2): Comprehensive quantization toolkit with 6 methods: Dynamic INT8, Static INT8, QAT, GPTQ (INT2/3/4/8), AWQ (INT4/INT8), Half Precision (FP16/BF16). Dedicated GUI tab with benchmarking, comparison tables, and one-click quantization. Supports packed low-bit storage, Hessian-based rounding, activation-aware weight protection, and fake-quantize STE for training.
• Custom Training: Upload any .txt file to teach the built-in AuraLite .pt model specific styles, languages, or fictional worlds.
• External GGUF Models: Open .gguf files from the Model tab and use them immediately for inference. GGUF files are inference-only in AuraLite; train/fine-tune with AuraLite’s native .pt checkpoints.
• Interruptible Training: Ability to stop training at any point and preserve the learned weights for immediate testing.
• Gradient Checkpointing: Trade compute for memory — enables training of larger models on consumer GPUs. Toggle in the Training tab.
• 💬 Chat / Instruction Mode: Dedicated chat tab with system/user/assistant roles, multiple templates (ChatML, Llama-2, Mistral, Gemma, Phi), conversation history, and real-time token streaming.
• 📏 Long generations stay coherent (2.11): generation attends to the newest seq_length tokens, the distances the model was trained on. Before 2.11 everything past the trained length turned into noise (held-out loss 3.97 against 1.48 at positions 256–511 of a model trained on 64). The KV cache stays bounded and long prompts are prefilled in chunks. --full-context / engine.set_context_window("full") restores the old behaviour — see docs/engine.md.
• 🔄 RoPE scaling (linear / NTK / YaRN): for continued training on longer windows and for imported models. On its own it does not let a model read past the length it was trained on (measured in docs/engine.md § 9); continue_training with a longer seq_length extends a model (docs/training.md § 7).
• 🌙 Dark Theme: Toggle available in the header.
• ☁️ Hugging Face Hub Integration: Push/pull models and LoRA adapters to/from the Hub.
• 📊 Model Evaluation: Built-in evaluation using lm-evaluation-harness (ARC, MMLU, GSM8K, etc.).

✅ Feature status: implemented / limited / opt-in (v2.6.5)

Honest status of the features that are easy to over-promise. Everything marked "implemented" has a regression test in tests/ (see tests/test_improvements_v265.py).

Diagnostics

auralite info --model model.pt --backends        # quantization backend availability
auralite info --model model.pt --bench 128       # tokens/s, weight memory, KV-cache mode
auralite info --model model.pt --quiet           # no progress/library noise
auralite train --text corpus.txt --params p.json  # engine logs go to stdout
AURALITE_LOG_LEVEL=WARNING auralite generate ...  # or silence INFO library lines
AURALITE_AGENT_MAX_ITERATIONS=12 AURALITE_AGENT_TOOL_TIMEOUT=30 auralite-gui

📚 Documentation

🛠 Technical Specifications

• Framework: PyTorch (Tensors, Autograd, AMP on CUDA, optional torch.compile).
• Attention: Multi-Head Self-Attention via PyTorch SDPA (Flash / memory-efficient kernels) with Causal Masking, RoPE and optional GQA.
• Normalization: RMSNorm (pre-norm), as used in LLaMA / Mistral / Qwen.
• Optimizer: AdamW (betas 0.9/0.95, weight decay) with cosine LR schedule + linear warmup and gradient clipping.
• Input/Output: Built-in BPE tokenizer (recommended) or character-level tokenization; old char-level checkpoints load transparently.

📦 Installation & Setup

Prerequisites

• Python 3.11 – 3.14. Python 3.10 reached end-of-life on 2026-10-01; AuraLite 2.6.x was the last series that supported it.
• NVIDIA GPU (optional). No CUDA Toolkit is needed — only the driver: on Linux the default torch wheel from PyPI already contains CUDA; on Windows pick the CUDA build on pytorch.org.

Install

From PyPI (2.18.0 on) — the same extras as below:

pip install auralite-ai                 # core: torch, numpy, matplotlib (training, generation, GUI)
pip install "auralite-ai[serve,data]"   # + the server, + big datasets (Hugging Face, Parquet, .zst)

From a clone:

git clone https://github.com/AlexanderNyr/AuraLite-AI.git && cd AuraLite-AI
pip install -e .              # core: torch, numpy, matplotlib (training, generation, GUI)
pip install -e '.[serve]'     # + OpenAI/Ollama-compatible server (FastAPI, uvicorn)
pip install -e '.[hf]'        # + Hugging Face models and LoRA fine-tuning (transformers, peft)
pip install -e '.[qlora]'     # + 4-/8-bit QLoRA loading (bitsandbytes; not on macOS)
pip install -e '.[data]'      # + big datasets: Hugging Face datasets, Parquet, .zst, Rust tokenizers (2.18)
pip install -e '.[test]'      # + the test suite

pip install -r requirements.txt installs the same core dependencies without installing the package (enough for python gui_app.py from the checkout). An installed AuraLite adds two commands: auralite (CLI, also python -m auralite) and auralite-gui (desktop app, also python -m auralite.gui).

.gguf models (llama.cpp)

PyPI only has the source of llama-cpp-python, so a plain pip install compiles llama.cpp — minutes of waiting, and an error on Windows without Visual C++ build tools. Use the prebuilt wheels instead:

# CPU: Linux x86-64/arm64, Windows x64, Apple Silicon
pip install -e '.[gguf]' --only-binary llama-cpp-python \
    --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu
# NVIDIA GPU, CUDA 12.5+: Linux x86-64, Windows x64
pip install -e '.[gguf]' --only-binary llama-cpp-python \
    --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu125

--only-binary turns a missing wheel into an immediate error instead of a long source build. For other accelerators see the llama-cpp-python installation guide.

Docker

docker build --target runtime-cpu -t auralite-ai:cpu .
# copy a checkpoint into the data volume (the container user owns /data)
docker run --rm -v auralite-data:/data -v "$PWD":/in:ro auralite-ai:cpu cp /in/model.pt /data/
docker run -p 8000:8000 -v auralite-data:/data -e AURALITE_MODEL=/data/model.pt auralite-ai:cpu

The image serves the OpenAI-compatible API on port 8000. It runs as an unprivileged user and keeps models, caches and state in /data (AURALITE_HOME). --target runtime-cuda builds the GPU variant (docker run --gpus all …). Tagged releases publish ghcr.io/alexandernyr/auralite-ai:<version>.

📖 How to Use

1. Launch the App:

   auralite-gui          # installed (pip install -e . or a wheel)
   python gui_app.py     # straight from a source checkout
   

2. Use an existing .gguf model (optional):
   • Open the 💾 Model tab.
   • Click Load Model and choose a .gguf file.
   • Go to ✨ Generation and prompt it. Streaming, batch mode, Thinking Mode and Web Search work with GGUF too.
   • A GGUF options dialog lets you set n_ctx, GPU layers, CPU threads, n_batch, chat format, chat-completion mode, mmap and mlock.
   • Note: .gguf files are quantized llama.cpp inference models; AuraLite does not train or re-save them as .pt.
   • The same GGUF defaults can be set via environment variables before launch: AURALITE_GGUF_N_CTX, AURALITE_GGUF_N_GPU_LAYERS, AURALITE_GGUF_N_THREADS, AURALITE_GGUF_N_BATCH, AURALITE_GGUF_CHAT_FORMAT, AURALITE_GGUF_USE_CHAT, AURALITE_GGUF_USE_MMAP, AURALITE_GGUF_USE_MLOCK.
3. Configure & Train a native AuraLite model:
   • Adjust the Hyperparameters to suit your hardware and dataset.
   • Click "Select .txt File" and provide your training data.
   • Click "Start Training". Monitor the Loss value; a decreasing loss indicates the AI is learning.
4. Generate Text:
   • Enter a Seed phrase to give the AI a starting point.
   • Set the desired Length of the output.
   • Click "Generate Text" and watch the AI create content based on its training.

🔨 Compiling to .exe (Windows)

To bundle the application into a portable application folder:

1. Run the provided build_exe.bat file.
2. The script will automatically install PyInstaller and bundle the PyTorch environment using --onedir mode (faster startup and easier to update than a single-file build).
3. The final build will be located in dist/AuraLite_AI_v2/. Launch it via dist/AuraLite_AI_v2/AuraLite_AI_v2.exe (distribute the whole folder).

⚠️ Hardware Compatibility Note

• CUDA Acceleration: Requires an NVIDIA GPU with Compute Capability 5.0 or higher.
• CPU Fallback: If a compatible GPU is not detected, AuraLite AI automatically switches to CPU mode. While slower, it remains fully functional.
• Memory Tip: For CPU-only users, keeping D_Model at 64 and Seq Length at 16 is recommended for optimal performance.