Skip to main content

LLM-X

PyPI version License: MIT

VRAM Matrix Prediction

Introduction

LLM-X is a CLI-first library designed to provide highly-precise, hardware-aware metrics for large language model inference requirements. It supports a wide range of architectures via SafeTensors and Hugging Face integration.

Key features

  • Memory Deficit/Surplus Analysis:
    • Calculate as a percentage memory deficits (when memory is not sufficient for inference) and memory surpluses (when memory is sufficient for inference), also considering that memory in use by the operating system and other processes.
    • Example: A +480.7% deficit alert immediately informs the user that the model is nearly 5x larger than the available GPU memory. In the absence of an available or detected GPU, the user will be informed that the deficit is not in VRAM but in RAM.
  • Dynamic Overhead Awareness:
    • Through psutil and NVML (nvidia-ml-py), LLM-X determines whether the model can fit in GPU memory (VRAM) or system memory (RAM).
  • Resources Required According to Quantization Levels
    • LLM-X offers a comparative matrix with the resources required for inference according to different levels of quantization and context windows.

Installation

Since it is a native Python library, it can be easily installed via PyPI using pip.

  • pip install llm-x-py

It can also be installed by cloning this repository and installing it locally.

  • git clone https://github.com/Sheikyon/LLM-X.git
  • cd LLM-X
  • pip install .

Usage

  • Core Estimation (--model-src, -m)

    • Model Sources
      • Local: Absolute paths or . for the current directory.
      • Remote: Hugging Face Model IDs or full repository URLs.
    • Inference Configuration
      • Context Window (--context, -c): Set specific token limits.
      • Batch Size (--batch, -b): Defaults to 1.
      • KV Cache Settings:
        • Precision (--kv-dtype, -q): Data type for the cache (e.g., BF16).
        • Quantization (--kv-quant): Compression method for the KV cache.
  • Hugging Face Token Management

    • Add (--set-new-token): Save and automatically sort new access tokens.
    • Select (--set-token): Set an existing token as the active one by ID.
    • List (--token-list): Displays a rich table with IDs, Active status, and previews.
  • Token Cleanup

    • Single (--del-token): Delete a specific token by index.
    • Purge (--del-all-tokens): Remove all stored tokens from the system.

Accuracy

Using a Qwen2.5-7B (BF16) at 131,072 context as a benchmark:

Metric hf-mem Accelerate LLM-X (Ours) Status
Weights (BF16) 14.19 GiB 13.17 GiB 14.19 GiB Consistent
KV Cache (131k) 0.00 GiB 0.00 GiB 14.00 GiB Ignored
Engine Overhead 0.00 GiB 0.00 GiB 2.12 GiB Ignored
Total VRAM Required 14.19 GiB 13.17 GiB 30.31 GiB Reality
Error Rate 113.6% 130.1% 1.8%

LLM-X achieves an accuracy rate of ≈ 98.2%, closely matching the empirical metrics of torch.cuda.memory_allocated(), which tracks actual tensors allocated in memory. This also results in an error rate of only ≈ 1.8% in supported models.

Metadata

Release files for llm-x-py 0.1.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for llm-x-py 0.1.5
File Size Uploaded Provenance Actions
llm_x_py-0.1.5.tar.gz 22.2 kB Without provenance Details for llm_x_py-0.1.5.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for llm-x-py 0.1.5
File Interpreter ABI Platform Actions
llm_x_py-0.1.5-py3-none-any.whl Python 3 none any Details for llm_x_py-0.1.5-py3-none-any.whl

Total release size: 45.1 kB

Release files / llm_x_py-0.1.5.tar.gz

Download URL llm_x_py-0.1.5.tar.gz
Size 22.2 kB
Tags Source
SHA-256 checksum
How to use checksums
77a98b80c9fad90d8c73d00d101da99d30710aed84048ba56cf13ab13c087d1a
BLAKE2b-256 checksum
How to use checksums
a796bba5830dbd77c9acfcde2a36a7aceeac1b29abe572bbccd0e64a30a66cda
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.9.26 {"installer":{"name":"uv","version":"0.9.26","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / llm_x_py-0.1.5-py3-none-any.whl

Download URL llm_x_py-0.1.5-py3-none-any.whl
Size 22.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
3eeec66da71b8212583d8ac03036a941fd2f9f7528540f7488809181413c4572
BLAKE2b-256 checksum
How to use checksums
2f2a8dc0977e07e44efdcef1ac78e475609dd92e57d85a07af5298339ba2bff8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.9.26 {"installer":{"name":"uv","version":"0.9.26","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for llm-x-py 0.1.5

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page