Swarmauri Parser Bert Embedding
Parser that converts text into embeddings using a Hugging Face BERT encoder. Produces Document objects whose metadata carries the averaged token embedding so downstream Swarmauri pipelines can work with dense vectors.
Features
- Uses
transformers.BertModel+BertTokenizer(defaultbert-base-uncased). - Accepts single strings or lists of strings and emits
Documentinstances with original text and embedding metadata. - Runs in inference (
eval) mode with automatictorch.no_grad()handling. - Works on CPU by default; configure PyTorch device settings to leverage GPU.
Prerequisites
- Python 3.10 or newer.
- PyTorch compatible with your hardware (installed automatically via
transformersif not present; install CUDA-enabled wheels manually when needed). - Internet access on first run so Hugging Face downloads tokenizer/model weights (or warm the cache ahead of time).
Installation
# pip
pip install swarmauri_parser_bertembedding
# poetry
poetry add swarmauri_parser_bertembedding
# uv (pyproject-based projects)
uv add swarmauri_parser_bertembedding
Quickstart
from swarmauri_parser_bertembedding import BERTEmbeddingParser
parser = BERTEmbeddingParser(parser_model_name="bert-base-uncased")
documents = parser.parse([
"Swarmauri agents cooperate over shared memory.",
"Dense embeddings power semantic search.",
])
for doc in documents:
vector = doc.metadata["embedding"]
print(doc.content)
print(len(vector), vector[:5])
Custom Models & Devices
import torch
from swarmauri_parser_bertembedding import BERTEmbeddingParser
from transformers import BertModel
class GPUParser(BERTEmbeddingParser):
def __init__(self, **kwargs):
super().__init__(**kwargs)
self._model = BertModel.from_pretrained(self.parser_model_name).to("cuda")
parser = GPUParser(parser_model_name="bert-base-multilingual-cased")
parser._model.eval()
Batch Embeddings at Scale
from tqdm import tqdm
from swarmauri_parser_bertembedding import BERTEmbeddingParser
texts = [f"Paragraph {i}" for i in range(1000)]
parser = BERTEmbeddingParser()
batched_docs = []
batch_size = 32
for start in tqdm(range(0, len(texts), batch_size)):
batch = texts[start:start + batch_size]
batched_docs.extend(parser.parse(batch))
Persist the resulting vectors into Swarmauri vector stores (Redis, Qdrant, etc.) via the metadata field.
Tips
- Preprocess text to match model expectations (lowercase for uncased BERT, language-specific cleanup for multilingual models).
- For extremely long documents, consider chunking before calling
parseto respect the 512 token limit. - Use PyTorch's
to("cuda")orto("mps")to execute on GPUs or Apple silicon accelerators. - Cache Hugging Face weights in CI/CD environments (
HF_HOME=/cache/hf) to avoid repeated downloads.
Want to help?
If you want to contribute to swarmauri-sdk, read up on our guidelines for contributing that will help you get started.
Metadata
Release files for swarmauri_parser_bertembedding 0.8.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| swarmauri_parser_bertembedding-0.8.3.tar.gz | 8.2 kB | Without provenance | Details for swarmauri_parser_bertembedding-0.8.3.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| swarmauri_parser_bertembedding-0.8.3-py3-none-any.whl | Python 3 | none | any | Details for swarmauri_parser_bertembedding-0.8.3-py3-none-any.whl |
Total release size: 17.5 kB
Release files / swarmauri_parser_bertembedding-0.8.3.tar.gz
| Download URL | swarmauri_parser_bertembedding-0.8.3.tar.gz |
|---|---|
| Size | 8.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
45a0b2e6ee0ad7091155c54d599b989e0215bb4bf208dfe576eab13c1902d525
|
|
BLAKE2b-256 checksum How to use checksums |
f7da45ec464c3226a4e8ca1a9b0a5b67f02f191ac572552afe44918938a2ac1f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.0 {"installer":{"name":"uv","version":"0.11.0","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / swarmauri_parser_bertembedding-0.8.3-py3-none-any.whl
| Download URL | swarmauri_parser_bertembedding-0.8.3-py3-none-any.whl |
|---|---|
| Size | 9.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e1181cde8dd1b5fa2cf1dbdd5c1cdc04a5df9ee1107a978dcdc4a40b7f711c3c
|
|
BLAKE2b-256 checksum How to use checksums |
523a4862badac469c71c60227f6dab8e451968a1c60b7a378adf0188d067e877
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.0 {"installer":{"name":"uv","version":"0.11.0","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for swarmauri_parser_bertembedding 0.8.3
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.