faster-clip 0.4.0

Fast CPU inference (Rust) for CLIP embeddings, inference-free SPLADE, and BERT fill-mask — drop-in replacement for the transformers / sentence-transformers models in a RAG stack. No PyTorch, no ONNX.

faster-clip

Fast, dependency-light CPU inference for the models a multilingual RAG stack needs — no PyTorch, no transformers, no sentence-transformers, no ONNX. Since 0.2 the compute is a hand-written Rust engine (a single native extension); tokenization uses the tokenizers crate.

pip install faster-clip

Three drop-in replacements, each numerically matching the original HuggingFace pipeline:

Class	Replaces	Output
ClipModel	SentenceTransformer('clip-ViT-B-32') + …-multilingual-v1	512-d CLIP image / text embeddings (cosine 1.0)
SparseEncoder	SparseEncoder('…inference-free-splade…')	{token_id: weight} sparse dicts (bit-identical)
FillMask	pipeline('fill-mask', 'bert-base-uncased' / 'bert-base-portuguese-cased')	top-k {token_str, sequence, score} (identical strings)

Performance

CPU (AMD Ryzen 7 3700X, 8 cores), single item, vs the original torch pipeline:

	torch / sentence-transformers	faster-clip (Rust)
CLIP text embed	~10 ms	~11 ms (≈ parity)
CLIP image embed	~62 ms	~81 ms
fill-mask (BERT)	~60 ms	~20 ms (3× faster)
SPLADE sparse	—	< 0.05 ms (pure lookup)

No torch/transformers install (multi-GB) — one native wheel. Since 0.4 the 2-D weights are stored as int8 (per-output-channel symmetric quantization, upconverted on the fly in the AVX2 kernels); small layernorm/bias params stay f16. Loading copies everything into owned buffers and drops the source mmap, so nothing of the original f32 file stays resident. Measured steady-state RSS on CPU:

loaded models	f32 / torch	faster-clip 0.4 (int8)
CLIP image + text	~1.8 GB	~280 MiB
+ SPLADE + fill-mask (en+pt)	—	~495 MiB

Accuracy is essentially unchanged: CLIP cosine ≥ 0.9997 vs torch, SPLADE bit-identical, fill-mask top-5 exact (en + pt).

CLIP embeddings

from faster_clip import ClipModel

clip = ClipModel()  # lazily downloads image / text weights on first use
text_emb = clip.encode_text(["um carro vermelho", "a red car"])   # (2, 512) list
img_emb  = clip.encode_image([open("photo.jpg", "rb").read()])    # (1, 512) list

• encode_text(list_of_str) → list of 512-float vectors.
• encode_image(list_of_image_bytes) → list of 512-float vectors. Pass the raw image bytes (PNG/JPEG/…); decoding + CLIP preprocessing happen in Rust.

Image and multilingual text share the same 512-d space. Normalize yourself for cosine search.

SPLADE sparse encoding (no sparse tensor materialised)

from faster_clip import SparseEncoder
sparse = SparseEncoder()
sparse.encode("banco de dados vetorial")     # -> {'15060': 1.09, '2951': 1.11, ...}

Inference-free: a token→weight lookup, so only the active dims are returned — no dense/sparse vector is ever built. encode, encode_query, encode_document are equivalent; each takes a str or list of str. The weights ship in the wheel.

Fill-mask autocomplete

from faster_clip import FillMask
fm = FillMask()
fm.predict("the weather is very [MASK]", lang="en")
fm.predict("o brasil é um [MASK]", lang="pt")
# -> [{'token_str': 'lixo', 'sequence': 'o brasil é um lixo', 'score': 0.18}, ...]

lang is "en" (bert-base-uncased) or "pt" (neuralmind/bert-base-portuguese-cased).

Model cache

CLIP and BERT weights are downloaded on first use and cached under ~/.cache/faster-clip (override with $FASTER_CLIP_CACHE). Inference releases the GIL, so one instance is safe to share across threads.

License

MIT. Model weights belong to their respective authors (OpenAI / sentence-transformers / neuralmind / Microsoft) under their original licenses.