Swarmauri Parser Bert Embedding
Parser that converts text into embeddings using a Hugging Face BERT encoder. Produces Document objects whose metadata carries the averaged token embedding so downstream Swarmauri pipelines can work with dense vectors.
Features
- Uses
transformers.BertModel+BertTokenizer(defaultbert-base-uncased). - Accepts single strings or lists of strings and emits
Documentinstances with original text and embedding metadata. - Runs in inference (
eval) mode with automatictorch.no_grad()handling. - Works on CPU by default; configure PyTorch device settings to leverage GPU.
Prerequisites
- Python 3.10 or newer.
- PyTorch compatible with your hardware (installed automatically via
transformersif not present; install CUDA-enabled wheels manually when needed). - Internet access on first run so Hugging Face downloads tokenizer/model weights (or warm the cache ahead of time).
Installation
# pip
pip install swarmauri_parser_bertembedding
# poetry
poetry add swarmauri_parser_bertembedding
# uv (pyproject-based projects)
uv add swarmauri_parser_bertembedding
Quickstart
from swarmauri_parser_bertembedding import BERTEmbeddingParser
parser = BERTEmbeddingParser(parser_model_name="bert-base-uncased")
documents = parser.parse([
"Swarmauri agents cooperate over shared memory.",
"Dense embeddings power semantic search.",
])
for doc in documents:
vector = doc.metadata["embedding"]
print(doc.content)
print(len(vector), vector[:5])
Custom Models & Devices
import torch
from swarmauri_parser_bertembedding import BERTEmbeddingParser
from transformers import BertModel
class GPUParser(BERTEmbeddingParser):
def __init__(self, **kwargs):
super().__init__(**kwargs)
self._model = BertModel.from_pretrained(self.parser_model_name).to("cuda")
parser = GPUParser(parser_model_name="bert-base-multilingual-cased")
parser._model.eval()
Batch Embeddings at Scale
from tqdm import tqdm
from swarmauri_parser_bertembedding import BERTEmbeddingParser
texts = [f"Paragraph {i}" for i in range(1000)]
parser = BERTEmbeddingParser()
batched_docs = []
batch_size = 32
for start in tqdm(range(0, len(texts), batch_size)):
batch = texts[start:start + batch_size]
batched_docs.extend(parser.parse(batch))
Persist the resulting vectors into Swarmauri vector stores (Redis, Qdrant, etc.) via the metadata field.
Tips
- Preprocess text to match model expectations (lowercase for uncased BERT, language-specific cleanup for multilingual models).
- For extremely long documents, consider chunking before calling
parseto respect the 512 token limit. - Use PyTorch's
to("cuda")orto("mps")to execute on GPUs or Apple silicon accelerators. - Cache Hugging Face weights in CI/CD environments (
HF_HOME=/cache/hf) to avoid repeated downloads.
Want to help?
If you want to contribute to swarmauri-sdk, read up on our guidelines for contributing that will help you get started.
1 maintainer
Metadata
1 maintainer
Release files for swarmauri_parser_bertembedding 0.8.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| swarmauri_parser_bertembedding-0.8.3.tar.gz | 8.2 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| swarmauri_parser_bertembedding-0.8.3-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 17.5 kB
Release files / swarmauri_parser_bertembedding-0.8.3.tar.gz
| Download URL | swarmauri_parser_bertembedding-0.8.3.tar.gz |
|---|---|
| Size | 8.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
45a0b2e6ee0ad7091155c54d599b989e0215bb4bf208dfe576eab13c1902d525
|
|
BLAKE2b-256 checksum How to use checksums |
f7da45ec464c3226a4e8ca1a9b0a5b67f02f191ac572552afe44918938a2ac1f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.0 {"installer":{"name":"uv","version":"0.11.0","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / swarmauri_parser_bertembedding-0.8.3-py3-none-any.whl
| Download URL | swarmauri_parser_bertembedding-0.8.3-py3-none-any.whl |
|---|---|
| Size | 9.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e1181cde8dd1b5fa2cf1dbdd5c1cdc04a5df9ee1107a978dcdc4a40b7f711c3c
|
|
BLAKE2b-256 checksum How to use checksums |
523a4862badac469c71c60227f6dab8e451968a1c60b7a378adf0188d067e877
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.0 {"installer":{"name":"uv","version":"0.11.0","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
