verbatim-core
Lightweight verbatim span extraction -- the RAG-agnostic core of verbatim-rag.
Extract source text spans from documents that answer a question. No vector
databases, embeddings, or heavy ML dependencies by default—runtime dependencies
are openai, pydantic, rapidfuzz, jinja2 and typing_extensions.
Project Structure
.
├── README.md # Package documentation
├── pyproject.toml # Package metadata, dependencies and extras
├── ruff.toml # Package level ruff lint/format configuration
└── verbatim_core
├── __init__.py # Public API exports
├── cli.py # `verbatim-enhance` batch CLI (JSON/JSONL in, JSONL out)
├── enhance.py # @verbatim_enhance decorator for existing RAG functions
├── extractor_models
│ ├── dataset.py # Data classes and torch Dataset for span-extractor training
│ ├── model.py # QAModel: transformer-based sentence/span classifier
│ ├── preprocess_ragbench.py # Converts RAGBench into extractor training data
│ ├── train.py # Training entry point (CLI)
│ └── trainer.py # Training/evaluation loop and metrics
├── extractors.py # Span extractors (LLMSpanExtractor and model-based)
├── llm_client.py # Unified OpenAI client wrapper (sync + async)
├── models.py # Pydantic models for responses, documents and highlights
├── prompts
│ ├── __init__.py # Prompt loader
│ ├── extraction
│ │ ├── default.txt # Default span-extraction prompt
│ │ └── structured.txt # Structured (template-driven) extraction prompt
│ └── template
│ ├── aggregate.txt # Prompt for one aggregate answer template
│ ├── fallback.txt # Prompt for fallback when no spans are found
│ └── per_fact.txt # Prompt for per-fact templates
├── providers.py # RAGProvider abstract interface (retrieve/generate)
├── response_builder.py # Builds responses with highlights and citations
├── templates
│ ├── __init__.py # Template exports
│ ├── base.py # Base class for template strategies
│ ├── contextual.py # LLM-generated templates using retrieved context
│ ├── filler.py # Fills templates with extracted spans
│ ├── manager.py # TemplateManager: selects and applies strategies
│ ├── question_specific.py # LLM-generated templates tailored to the question
│ ├── random.py # Randomly chosen predefined templates
│ ├── static.py # Fixed predefined template
│ └── structured.py # Template-driven structured extraction
├── transform.py # VerbatimTransform: question + context -> cited answer
├── types.py # Shared typing protocols
└── universal_document.py # UniversalDocument: RAG-agnostic document dataclass
Installation
pip install verbatim-core
Quick Start
from verbatim_core import Chunk, Transformer
from verbatim_core.extractors import LLMSpanExtractor
from verbatim_core.llm_client import LLMClient
from verbatim_core.response_builder import ResponseBuilder
from verbatim_core.templates import TemplateManager
client = LLMClient("gpt-4o-mini")
transformer = Transformer(
LLMSpanExtractor(client),
TemplateManager(llm_client=client, default_mode="contextual"),
)
question = "What is the main finding?"
chunks = [
Chunk(id="paper-a", text="The study found that X leads to Y.", metadata={"title": "Paper A"}),
Chunk(
id="paper-b",
text="Results show Z is statistically significant.",
metadata={"title": "Paper B"},
),
]
answer = transformer.transform(question, chunks)
print(answer.text)
# Each citation points at a span of a chunk, by id and offsets
for citation in answer.citations:
print(citation.number, citation.chunk_id, citation.start, citation.end)
# A QueryResponse, with the highlights of each document
response = ResponseBuilder().build_from_answer(question, answer, chunks)
for doc in response.documents:
for highlight in doc.highlights:
print(f" [{highlight.start}:{highlight.end}] {highlight.text}")
What This Package Includes
- Transformer -- question + chunks -> answer cited from the chunks' own text (sync, async, streaming)
- LLMSpanExtractor, ModelSpanExtractor, SemanticHighlightExtractor -- extract verbatim spans, with their chunk ids and offsets
- LLMClient -- unified OpenAI API wrapper (sync + async)
- TemplateManager -- response formatting with multiple template strategies
- @verbatim_enhance -- decorator to enhance existing RAG functions
- CLI (
verbatim-enhance) -- batch processing from the command line
Model-Based Extraction
For ModernBERT or Zilliz semantic highlight extractors (adds torch, transformers):
pip install verbatim-core[model]
from verbatim_core import Transformer
from verbatim_core.extractors import ModelSpanExtractor
from verbatim_core.templates import TemplateManager
extractor = ModelSpanExtractor.from_pretrained("KRLabsOrg/verbatim-rag-modern-bert-v2")
transformer = Transformer(extractor, TemplateManager()) # static templates: no LLM call
Upgrading from 0.2.x
VerbatimTransform, verbatim_query and the other 0.2 entry points still work and warn until
0.5.0. The migration guide maps each one to
its replacement.
Environment
export OPENAI_API_KEY=your_api_key_here
Full RAG System
For the complete RAG pipeline with vector indexing, embeddings, and document processing, install the full package:
pip install verbatim-rag
Metadata
Release files for verbatim-core 0.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| verbatim_core-0.3.0.tar.gz | 77.0 kB | Without provenance | Details for verbatim_core-0.3.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| verbatim_core-0.3.0-py3-none-any.whl | Python 3 | none | any | Details for verbatim_core-0.3.0-py3-none-any.whl |
Total release size: 175.0 kB
Release files / verbatim_core-0.3.0.tar.gz
| Download URL | verbatim_core-0.3.0.tar.gz |
|---|---|
| Size | 77.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7920aa41e893fc87b834d69191f05525e7e670b9d71ef156b7f96257aefc3130
|
|
BLAKE2b-256 checksum How to use checksums |
25b2e936deb1b518ac08cc1000b5751eecd51dc782d8f398dab95d2cce8a0730
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.9
|
Release files / verbatim_core-0.3.0-py3-none-any.whl
| Download URL | verbatim_core-0.3.0-py3-none-any.whl |
|---|---|
| Size | 98.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
3da8757a4ac1f53f4da3c60865e3806962f725cf9da9df451c7ebc306f4cbd4e
|
|
BLAKE2b-256 checksum How to use checksums |
50a9add11549a1bb616b1c7a6f3e5c2b5792329212ed90054124de62eb6e3dcd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.9
|
Security for verbatim-core 0.3.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.