ggmbed

A lightweight, zero-PyTorch, zero-ONNX-Runtime dense embedding inference engine. Uses a native C++ extension (llama.cpp native acceleration) to encode text extremely fast without pulling in gigabytes of deep learning dependencies.

Install

pip install ggmbed

Only runtime dependencies are numpy and huggingface-hub.

Usage

from ggmbed import Embedder

# Initializes the encoder, downloading the optimized Q8_0 quantized all-MiniLM-L6-v2 GGUF model automatically
model = Embedder("sentence-transformers/all-MiniLM-L6-v2")

embeddings = model.encode(["What is a dense embedding?", "It's extremely fast and lightweight."])
print(embeddings.shape)  # (2, 384)

# Load BAAI General Embedding model (auto-resolves to GGUF and uses CLS pooling)
bge_model = Embedder("BAAI/bge-small-en-v1.5")
bge_embeddings = bge_model.encode(["BAAI General Embedding models use CLS pooling."])

You can also point it at a local directory containing a .gguf file or directly to a .gguf file path:

model = Embedder("./my-local-model-directory/")
# OR
model = Embedder("./models/all-MiniLM-L6-v2-Q8_0.gguf")

Supported Out-of-the-Box Models

The engine automatically handles GGUF model downloading from Hugging Face (thlurte/*), caching, and pooling settings for the following pre-quantized models:

sentence-transformers/all-MiniLM-L6-v2 (default, Mean Pooling, 384 dimensions)
BAAI/bge-small-en-v1.5 (CLS Pooling, 384 dimensions)
lightonai/DenseOn (ModernBERT architecture, CLS Pooling, 768 dimensions)

You can query the list programmatically:

from ggmbed import Embedder

print(Embedder.list_supported_models())
# ['sentence-transformers/all-MiniLM-L6-v2', 'BAAI/bge-small-en-v1.5', 'lightonai/DenseOn']

Configuration Options

model_name_or_path: Local path to a GGUF file or directory, or a Hugging Face Hub model ID (defaults to "sentence-transformers/all-MiniLM-L6-v2").
tokenizer_path: Optional explicit path to tokenizer.json (no longer required, as llama.cpp loads vocabulary natively from the GGUF model).
num_threads: Number of CPU threads to use. Defaults to 0 (which automatically detects and uses physical CPU cores, avoiding hyperthreading bottlenecks).
quantization: Preferred quantization format (e.g., "Q8_0", "F16", "F32", "Q4_0"). Defaults to "Q8_0".
pooling_mode: Pooling strategy to use ("mean" or "cls"). Defaults to None (which auto-detects based on the model name).

Benchmarks & Reproducibility

To reproduce the latency, memory footprint (RSS), and quantization accuracy results, run the scripts directly using uv:

# 1. Run the latency and memory benchmark (automatically manages fastembed dependency)
uv run --group benchmark python scripts/benchmark.py

# 2. Run the quantization accuracy benchmark
uv run python scripts/benchmark_accuracy.py

Benchmark Results & Visualizations

Below is the live benchmark comparison measured on Linux (AMD64 CPU):

Model: sentence-transformers/all-MiniLM-L6-v2 (Mean Pooling)

MiniLM Load Time MiniLM Memory MiniLM Throughput

Metric	ggmbed (GGUF Q8_0)	fastembed (ONNX)	sentence-transformers (PyTorch)
Model Load Time	⚡ 1,909.7 ms	13,488.1 ms (7.0x slower)	20,179.3 ms (10.5x slower)
Peak RAM / Memory	🟢 127.6 MB	910.7 MB (7.1x heavier)	785.0 MB (6.1x heavier)
Single Latency (p50)	12.32 ms	11.91 ms	3.70 ms
Single Latency (p95)	17.31 ms	16.95 ms	7.66 ms

Batch Throughput (sentences / second)

Batch Size	ggmbed (sent/s)	fastembed (sent/s)	sentence-transformers (sent/s)
1	78.8	86.1	55.1
4	83.4	96.3	348.3
8	79.1	65.3	1,469.5
32	78.7	55.0	4,608.6
128	79.0	37.2	6,053.3

Model: BAAI/bge-small-en-v1.5 (CLS Pooling)

BGE Latency BGE Memory BGE Throughput

Metric	ggmbed (GGUF Q8_0)	fastembed (ONNX)	sentence-transformers (PyTorch)
Model Load Time	⚡ 1,700.3 ms	11,263.5 ms (6.6x slower)	18,336.6 ms (10.7x slower)
Peak RAM / Memory	🟢 111.0 MB	351.4 MB (3.1x heavier)	806.2 MB (7.2x heavier)
Single Latency (Mean)	⚡ 6.73 ms	9.58 ms	6.14 ms
Single Latency (p50)	6.65 ms	9.60 ms	4.95 ms
Single Latency (p95)	9.20 ms	12.57 ms	13.03 ms

Batch Throughput (sentences / second)

Batch Size	ggmbed (sent/s)	fastembed (sent/s)	sentence-transformers (sent/s)
1	⚡ 268.0	130.2	66.1
4	220.5	252.4	230.7
8	206.6	292.2	949.7
32	199.7	286.1	2,979.9
128	198.5	158.1	3,496.5

Quantization Accuracy vs. F32 Baseline

Below is the cosine similarity accuracy comparison of different quantization formats vs the unquantized F32 baseline, measured over a set of diverse test sentences:

Quantization	Size (MiniLM)	Cosine Similarity vs. F32	Status / Recommendation
`F32`	86.7 MiB	1.000000	Baseline
`F16`	43.4 MiB	1.000000	Near Lossless
`Q8_0`	23.5 MiB	0.999659	Highly Recommended (virtually lossless, 3.7x smaller)
`Q4_0`	18.4 MiB	0.970772	Not Recommended (noticeable drop in semantic accuracy)

💡 Tip: For small embedding models like MiniLM and BGE-small, 8-bit quantization (Q8_0) is the absolute sweet spot, retaining 99.96% accuracy while reducing memory footprint and load times. Lower bit-depths like 4-bit (Q4_0) suffer noticeable quality loss due to the small parameter capacity of these architectures.

Model: lightonai/DenseOn (ModernBERT, CLS Pooling)

We have conducted a detailed evaluation of accuracy loss, file size compression, and CPU inference latency across all 9 quantization formats of the lightonai/DenseOn model (ranging from Float32 to 2-bit quantization).

For detailed tables, recommendation guides, and performance charts, please read the full DenseOn Quantization & Accuracy Results.

Advanced: Compile from Source (Hardware Acceleration)

By default, pre-built binary wheels are compiled with native SIMD instructions (AVX2/AVX-512/ARM NEON) for maximum CPU portability. If you are compiling from source and want to link against optimized system BLAS backends, pass the appropriate CMake arguments during installation:

AMD / Generic CPUs (OpenBLAS):

CMAKE_ARGS="-DGGML_OPENBLAS=ON" pip install --no-binary :all: ggmbed

Intel CPUs (Intel MKL / oneDNN):

CMAKE_ARGS="-DGGML_MKL=ON" pip install --no-binary :all: ggmbed

Features

GGUF-Native: Avoids PyTorch and ONNX Runtime entirely.
Hardware Optimized: Compiled with native SIMD instructions (AVX2/AVX-512/ARM NEON) and Flash Attention support.
Dynamic Threading: Auto-detects physical CPU cores to prevent runtime CPU thread thrashing.
Highly Portable: No complex system level dependencies, builds easily on macOS, Linux, and Windows.

License

MIT. See LICENSE.

License
- OSI Approved :: MIT License
Operating System
- OS Independent
Programming Language
- Python :: 3

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ggmbed-0.2.0.tar.gz (807.9 kB view details)

Uploaded Aug 16, 2026 Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

ggmbed-0.2.0-cp313-cp313-win_amd64.whl (8.3 MB view details)

Uploaded Aug 16, 2026 CPython 3.13Windows x86-64

ggmbed-0.2.0-cp313-cp313-manylinux_2_34_x86_64.whl (4.4 MB view details)

Uploaded Aug 16, 2026 CPython 3.13manylinux: glibc 2.34+ x86-64

ggmbed-0.2.0-cp313-cp313-manylinux_2_34_aarch64.whl (4.0 MB view details)

Uploaded Aug 16, 2026 CPython 3.13manylinux: glibc 2.34+ ARM64

ggmbed-0.2.0-cp313-cp313-macosx_13_0_arm64.whl (3.6 MB view details)

Uploaded Aug 16, 2026 CPython 3.13macOS 13.0+ ARM64

ggmbed-0.2.0-cp312-cp312-win_amd64.whl (8.3 MB view details)

Uploaded Aug 16, 2026 CPython 3.12Windows x86-64

ggmbed-0.2.0-cp312-cp312-manylinux_2_34_x86_64.whl (4.4 MB view details)

Uploaded Aug 16, 2026 CPython 3.12manylinux: glibc 2.34+ x86-64

ggmbed-0.2.0-cp312-cp312-manylinux_2_34_aarch64.whl (4.0 MB view details)

Uploaded Aug 16, 2026 CPython 3.12manylinux: glibc 2.34+ ARM64

ggmbed-0.2.0-cp312-cp312-macosx_13_0_arm64.whl (3.6 MB view details)

Uploaded Aug 16, 2026 CPython 3.12macOS 13.0+ ARM64

ggmbed-0.2.0-cp311-cp311-win_amd64.whl (8.3 MB view details)

Uploaded Aug 16, 2026 CPython 3.11Windows x86-64

ggmbed-0.2.0-cp311-cp311-manylinux_2_34_x86_64.whl (4.4 MB view details)

Uploaded Aug 16, 2026 CPython 3.11manylinux: glibc 2.34+ x86-64

ggmbed-0.2.0-cp311-cp311-manylinux_2_34_aarch64.whl (4.0 MB view details)

Uploaded Aug 16, 2026 CPython 3.11manylinux: glibc 2.34+ ARM64

ggmbed-0.2.0-cp311-cp311-macosx_13_0_arm64.whl (3.6 MB view details)

Uploaded Aug 16, 2026 CPython 3.11macOS 13.0+ ARM64

ggmbed-0.2.0-cp310-cp310-win_amd64.whl (8.3 MB view details)

Uploaded Aug 16, 2026 CPython 3.10Windows x86-64

ggmbed-0.2.0-cp310-cp310-manylinux_2_34_x86_64.whl (4.4 MB view details)

Uploaded Aug 16, 2026 CPython 3.10manylinux: glibc 2.34+ x86-64

ggmbed-0.2.0-cp310-cp310-manylinux_2_34_aarch64.whl (4.0 MB view details)

Uploaded Aug 16, 2026 CPython 3.10manylinux: glibc 2.34+ ARM64

ggmbed-0.2.0-cp310-cp310-macosx_13_0_arm64.whl (3.6 MB view details)

Uploaded Aug 16, 2026 CPython 3.10macOS 13.0+ ARM64

ggmbed-0.2.0-cp39-cp39-win_amd64.whl (8.3 MB view details)

Uploaded Aug 16, 2026 CPython 3.9Windows x86-64

ggmbed-0.2.0-cp39-cp39-manylinux_2_34_x86_64.whl (4.4 MB view details)

Uploaded Aug 16, 2026 CPython 3.9manylinux: glibc 2.34+ x86-64

ggmbed-0.2.0-cp39-cp39-manylinux_2_34_aarch64.whl (4.0 MB view details)

Uploaded Aug 16, 2026 CPython 3.9manylinux: glibc 2.34+ ARM64

ggmbed-0.2.0-cp39-cp39-macosx_13_0_arm64.whl (3.6 MB view details)

Uploaded Aug 16, 2026 CPython 3.9macOS 13.0+ ARM64

File details

Details for the file ggmbed-0.2.0.tar.gz.

File metadata

Download URL: ggmbed-0.2.0.tar.gz
Upload date: Aug 16, 2026
Size: 807.9 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for ggmbed-0.2.0.tar.gz
Algorithm	Hash digest
SHA256	`ac7fae9a6b2815daf1eae062261c67461f1f29d9fb8cc60001a3bca63df66a3a`
MD5	`46d842fc2e6466d5dd8f7d817b674230`
BLAKE2b-256	`f31e568070a88b732fe2a0b3c165700369565e62d90c954d9204610a93be8acd`

Algorithm	Hash digest
SHA256	`ef5cda23011df27382ba6674852be63b1cbed54cd18486c27c2cbc231bcc09fc`
MD5	`e3fdc182f47f0355a125a10ee46bb3bd`
BLAKE2b-256	`ffb6a05a0490cccdbbbc652e47ce67f6738f9ad9e305bd2551f4fbdb23a8d91f`

Algorithm	Hash digest
SHA256	`d591a2c0d965e4178398d27ca9534af2fcce0a56772b4a208a9b1cd10716cedf`
MD5	`c36558ff7f5dd71371b6fbdd492df6b3`
BLAKE2b-256	`6dc7b1de9c6c3f9012ecd5c5bb71d56fcf64f6036eb2394a184071a3875e0e43`

Algorithm	Hash digest
SHA256	`ed5ecb72489a5046465aca4e95a64bceef07dc616d00e8925db51999784273e5`
MD5	`24a723162373a2113565fe423cbd896f`
BLAKE2b-256	`0f9d1cdbcbf4084ae76af71349540bbd74c1cf2d13a1b5b0e49f2d72f1a0dc51`

Algorithm	Hash digest
SHA256	`1cc2af18e1fce380a54a60d8bcf3db3f950d978509c32f68bdc00f3852fd26a3`
MD5	`24eb79f5447ceec555c0be3ac3a2ee3c`
BLAKE2b-256	`142dca9d00c5bae90ea305127db44e7198e86321cc78e0f4816b57a4219e8ba9`

Algorithm	Hash digest
SHA256	`f5a669c2a90149a0fb25e98b56d80331bf2f6379bb1b122a1ebaeb902e9b6169`
MD5	`a6649fe449e223b7242a48386a6418b7`
BLAKE2b-256	`3073d328a1b22a22e70c582dd8adacdfa1930617f87aec2204e2199156653140`

Algorithm	Hash digest
SHA256	`819d2d03a268de6a9a98b07d53f8e3f929d8641f721f95cb9062aaecafa4ea65`
MD5	`09f5a68afd5764a864c1f01a84100527`
BLAKE2b-256	`3bcffa0a28c0fd67da30181b130d3e1606933e62d26eee8b73231f9ceac36f82`

Algorithm	Hash digest
SHA256	`3e2af7e930bbea3d56091b8bf520787e0f9037740c899ae37dc00382fd09bb43`
MD5	`7090fad7d7531bba8b45631cb98ebbc7`
BLAKE2b-256	`e711c2ea8e2109df2c4fc5c157c04ca19a78e6f639b00221b500281a83897f4a`

Algorithm	Hash digest
SHA256	`69fc000298c1ea067179f5bd8de1b0b163cfc00219b19978936fe2f31cf1293f`
MD5	`55533ecf78a3150c2153349aceb00ea6`
BLAKE2b-256	`5424a6b57def33c8ce7481d43aa22a62e7fef53657121c0ae86b4600d5b5bf69`

Algorithm	Hash digest
SHA256	`8d17589dc68e235efa48aea8df52ec76ce76d0f5b5a47ed6ae22f6ec4d7ec3e2`
MD5	`bfec8f34ebd0888255c56b9a508e2e7c`
BLAKE2b-256	`a65a3465ba1668aa05fda3fd8503d6fe885a0e0d4048abf47be9a005cb04a6ce`

Algorithm	Hash digest
SHA256	`6741741d2fdf3d95d6d258d3916aea5ccabed8f1595811b5899c548e55e5b783`
MD5	`616dfd33ce93adb084fb4797baf139ed`
BLAKE2b-256	`f2c4c4e82d88cd61be50b465287d236ca8aa1760d78d7f95fbe386fd454f6eba`

Algorithm	Hash digest
SHA256	`9cdafd9ef2e4ee47fbc112fc0085beb196bd42bfb2bd02c71dfaf6c2f4f51b80`
MD5	`e3ce745a5ef3ad2374f1ed2d16226ac9`
BLAKE2b-256	`befaa3b5aeba6994346406106f71c8f220c749e33a1da7345449004c4189caba`

Algorithm	Hash digest
SHA256	`b2e4f18f18f20172279e4c0b6dff7002897246b4e7faa0a65eef44b5bfad8888`
MD5	`f124ace9a9a5da884f69b44812637f4d`
BLAKE2b-256	`0381fbeb1cb3c2807ec1c893d97ba06fc0d1e57f65cd30ee605b080b36aeeaaf`

Algorithm	Hash digest
SHA256	`82d5cfdbfd6b93a3d9cea7780936f79ee8f4cdd2e53a30ee059e6d7e2d2f3d95`
MD5	`a762a58ba1b8f53574ff7d9bef5f8508`
BLAKE2b-256	`82d4c23d313fbc74a2a09e110ac91c6c02f958140ac428edbf6176de18638985`

Algorithm	Hash digest
SHA256	`6a071734637e80a1554403f74aab640bacebe7aec2e57615c85b1dd41aba0ddf`
MD5	`40445d7b8b38e863d1ee774db01d8143`
BLAKE2b-256	`2b149ee5427a8ce9142333b6bea7c60d5b6f89a89abb3a66ebce2debacdc9d89`

Algorithm	Hash digest
SHA256	`983a7ef086bec0dcd4e865735149f7acea05bab63c742e4cee22ad0931309e28`
MD5	`73f6a747350d15f94b40e031bb434aaf`
BLAKE2b-256	`727eb3f95ffaa0cacab7f1957c43418190c5bb8e0123d9e273919181c78b6675`

Algorithm	Hash digest
SHA256	`ee8b8561f632a4792a981395885a73c4c1b28bd67d167c8f3360f5ee29f30dc1`
MD5	`d6d36d2e5a8a72d37196fd478ec17bbd`
BLAKE2b-256	`f0567c9b3765f4eb3804ff428fef819da88cb89ef090a37c940d1a8d84da4643`

Algorithm	Hash digest
SHA256	`84d1a829124420f65169da3f98b823fa527ad6a32affae0c929e0b534c49b546`
MD5	`9859cb9898d45768a5a193ba1c4671c8`
BLAKE2b-256	`0381e6d867368849e923930fa15f0bd1d82588cca25791fd9a9d37811b8a6704`

Algorithm	Hash digest
SHA256	`80efa758d7f3666ef5cb8e9f22aa83242224aa6b19e2fbf82e06ad7ca5a7a5f5`
MD5	`6ee97cb65522e332e369eb860119c6a6`
BLAKE2b-256	`003150afa161820e60b1834df201d03adec5902790b7b7204dc84136d7b82a06`

Algorithm	Hash digest
SHA256	`b328bcf05335604288c49cd8d56d5faa77d35dd0b7225095a77e9d90122a135d`
MD5	`19034cf20d3ce7e21d81d27475a80a50`
BLAKE2b-256	`ec77fcc69d9aca9c6626fd8a4ba03f4098674bbe64f43de587097941c8218d7d`

Algorithm	Hash digest
SHA256	`93cee6d2d99917801ca55918dfea7df8b40d247de37ac5358969a6a04cd79355`
MD5	`c603c162425bf4fcf4a56a6926f92960`
BLAKE2b-256	`6a86cc646017571c44938af66169249d8f7a1d3d6579b3121a1b18cac9fcb661`

ggmbed 0.2.0

ggmbed

Install

Usage

Supported Out-of-the-Box Models

Configuration Options

Benchmarks & Reproducibility

Benchmark Results & Visualizations

Model: sentence-transformers/all-MiniLM-L6-v2 (Mean Pooling)

Model: BAAI/bge-small-en-v1.5 (CLS Pooling)

Quantization Accuracy vs. F32 Baseline

Model: lightonai/DenseOn (ModernBERT, CLS Pooling)

Advanced: Compile from Source (Hardware Acceleration)

Features

License

Key dates

1 maintainer

License

Requires

Tags

Classifiers

Download files

Source Distribution

Built Distributions

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance