LLM-X
Introduction
LLM-X is a CLI-first library designed to provide highly-precise, hardware-aware metrics for large language model inference requirements. It supports a wide range of architectures via SafeTensors and Hugging Face integration.
Key features
- Memory Deficit/Surplus Analysis:
- Calculate as a percentage memory deficits (when memory is not sufficient for inference) and memory surpluses (when memory is sufficient for inference), also considering that memory in use by the operating system and other processes.
- Example: A +480.7% deficit alert immediately informs the user that the model is nearly 5x larger than the available GPU memory. In the absence of an available or detected GPU, the user will be informed that the deficit is not in VRAM but in RAM.
- Dynamic Overhead Awareness:
- Through
psutiland NVML (nvidia-ml-py), LLM-X determines whether the model can fit in GPU memory (VRAM) or system memory (RAM).
- Through
- Resources Required According to Quantization Levels
- LLM-X offers a comparative matrix with the resources required for inference according to different levels of quantization and context windows.
Installation
Since it is a native Python library, it can be easily installed via PyPI using pip.
pip install llm-x-py
It can also be installed by cloning this repository and installing it locally.
git clone https://github.com/Sheikyon/LLM-X.gitcd LLM-Xpip install .
Usage
-
Core Estimation (
--model-src,-m)- Model Sources
- Local: Absolute paths or
.for the current directory. - Remote: Hugging Face Model IDs or full repository URLs.
- Local: Absolute paths or
- Inference Configuration
- Context Window (
--context,-c): Set specific token limits. - Batch Size (
--batch,-b): Defaults to 1. - KV Cache Settings:
- Precision (
--kv-dtype,-q): Data type for the cache (e.g., BF16). - Quantization (
--kv-quant): Compression method for the KV cache.
- Precision (
- Context Window (
- Model Sources
-
Hugging Face Token Management
- Add (
--set-new-token): Save and automatically sort new access tokens. - Select (
--set-token): Set an existing token as the active one by ID. - List (
--token-list): Displays arichtable with IDs, Active status, and previews.
- Add (
-
Token Cleanup
- Single (
--del-token): Delete a specific token by index. - Purge (
--del-all-tokens): Remove all stored tokens from the system.
- Single (
Accuracy
Using a Qwen2.5-7B (BF16) at 131,072 context as a benchmark:
| Metric | hf-mem | Accelerate | LLM-X (Ours) | Status |
|---|---|---|---|---|
| Weights (BF16) | 14.19 GiB | 13.17 GiB | 14.19 GiB | Consistent |
| KV Cache (131k) | 0.00 GiB | 0.00 GiB | 14.00 GiB | Ignored |
| Engine Overhead | 0.00 GiB | 0.00 GiB | 2.12 GiB | Ignored |
| Total VRAM Required | 14.19 GiB | 13.17 GiB | 30.31 GiB | Reality |
| Error Rate | 113.6% | 130.1% | 1.8% |
LLM-X achieves an accuracy rate of ≈ 98.2%, closely matching the empirical metrics of torch.cuda.memory_allocated(), which tracks actual tensors allocated in memory. This also results in an error rate of only ≈ 1.8% in supported models.
Metadata
Release files for llm-x-py 0.1.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| llm_x_py-0.1.5.tar.gz | 22.2 kB | Without provenance | Details for llm_x_py-0.1.5.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| llm_x_py-0.1.5-py3-none-any.whl | Python 3 | none | any | Details for llm_x_py-0.1.5-py3-none-any.whl |
Total release size: 45.1 kB
Release files / llm_x_py-0.1.5.tar.gz
| Download URL | llm_x_py-0.1.5.tar.gz |
|---|---|
| Size | 22.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
77a98b80c9fad90d8c73d00d101da99d30710aed84048ba56cf13ab13c087d1a
|
|
BLAKE2b-256 checksum How to use checksums |
a796bba5830dbd77c9acfcde2a36a7aceeac1b29abe572bbccd0e64a30a66cda
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.9.26 {"installer":{"name":"uv","version":"0.9.26","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / llm_x_py-0.1.5-py3-none-any.whl
| Download URL | llm_x_py-0.1.5-py3-none-any.whl |
|---|---|
| Size | 22.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
3eeec66da71b8212583d8ac03036a941fd2f9f7528540f7488809181413c4572
|
|
BLAKE2b-256 checksum How to use checksums |
2f2a8dc0977e07e44efdcef1ac78e475609dd92e57d85a07af5298339ba2bff8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.9.26 {"installer":{"name":"uv","version":"0.9.26","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for llm-x-py 0.1.5
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.