txtai-xberg
Feed Xberg document extraction into txtai. XbergPipeline is a plain callable that extracts text and metadata from 101 formats — running OCR where needed — and flattens the result into documents ready for txtai.Embeddings.index. When you enable Xberg's native chunking, each chunk becomes one embedding-sized segment instead of a single blob.
Install
pip install txtai-xberg
Requires Python 3.10+. Install the txtai extra (pip install "txtai-xberg[txtai]") if txtai isn't already in your environment.
Extract
Call the pipeline with a path or a list of paths. A string returns one document; a list returns documents in input order. Batches run through Xberg's extract_batch in a single native call.
from txtai_xberg import XbergPipeline
pipeline = XbergPipeline()
doc = pipeline("report.pdf")
print(doc["content"]) # extracted markdown
print(doc["metadata"]["title"]) # source, mime_type, title, authors, languages, page_count
docs = pipeline(["report.pdf", "notes.docx"])
Index into txtai
Use to_documents to get (id, text, tags) tuples and hand them straight to Embeddings.index. Enable Xberg chunking so segments arrive sized for the model, with heading and page context in each document's tags.
from txtai import Embeddings
from txtai_xberg import XbergPipeline
from xberg import ChunkingConfig, ExtractionConfig
pipeline = XbergPipeline(
config=ExtractionConfig(chunking=ChunkingConfig(max_characters=1000, overlap=200)),
)
documents = pipeline.to_documents(["report.pdf", "notes.docx"])
embeddings = Embeddings(path="sentence-transformers/all-MiniLM-L6-v2", content=True)
embeddings.index(documents)
for result in embeddings.search("quarterly revenue", 3):
print(result["id"], result["text"])
Without a chunking config, to_documents emits one document per file. Chunk ids are "<source>#<chunk_index>".
Configure extraction
Pass any Xberg ExtractionConfig to control OCR, output format, chunking, and concurrency.
from xberg import ExtractionConfig, OcrConfig
pipeline = XbergPipeline(
config=ExtractionConfig(
output_format="markdown",
ocr=OcrConfig(language="eng"),
force_ocr=True,
max_concurrent_extractions=8,
),
)
Async
Already inside an event loop? Use the async methods — await pipeline.acall(paths) and await pipeline.ato_documents(paths). The synchronous __call__ and to_documents wrap these with asyncio.run and must not run inside a running loop.
Errors
Per-input failures in a batch surface as an ExtractionFailedError, whose errors attribute holds Xberg's ExtractionErrorItem objects (each with an index, source, code, and message).
For the full API, see the Xberg documentation.
2 maintainers
Metadata
2 maintainers
Release files for txtai-xberg 1.0.12
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| txtai_xberg-1.0.12.tar.gz | 9.2 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| txtai_xberg-1.0.12-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 16.2 kB
Release files / txtai_xberg-1.0.12.tar.gz
| Download URL | txtai_xberg-1.0.12.tar.gz |
|---|---|
| Size | 9.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c28d7d046f896ff29aa5ce116dea572207c41b9eaa76366862ce7e458ab61977
|
|
BLAKE2b-256 checksum How to use checksums |
a25541957dffe2546fe1dc8c45de4cac3e0a2ae9254979ace5b1080acd4abf29
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.12.1 {"installer":{"name":"uv","version":"0.12.1","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / txtai_xberg-1.0.12-py3-none-any.whl
| Download URL | txtai_xberg-1.0.12-py3-none-any.whl |
|---|---|
| Size | 7.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
bf75b509a07e90e9e14a270573be4daa4da1b7f08e1dbe7217c14e34778f7416
|
|
BLAKE2b-256 checksum How to use checksums |
0f77ec06542853655f70305152fed720c95f311b699644b43b813874ccf70161
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.12.1 {"installer":{"name":"uv","version":"0.12.1","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
