Skip to main content

PythonAI Grounded Generator

A compact Python helper for generating answers from retrieved context with an already loaded Hugging Face causal language model.

It provides:

  • Custom system prompts and questions
  • Token-budget-aware context truncation
  • Qwen3 non-thinking chat formatting
  • CPU or CUDA inference (device controlled by the caller)
  • A plain-text answer suitable for wrapping in application-specific Pydantic models

Retrieval, context assembly, and structured JSON serialization remain the responsibility of the calling application. This package does not guarantee factual correctness or replace source validation.

Installation

After publication:

pip install pythonai-grounded-generator

Or install the local project:

pip install .

Example

from transformers import AutoModelForCausalLM, AutoTokenizer
from pythonai_grounded import generate_answer

name = "Qwen/Qwen3-0.6B"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name).to("cpu")
model.eval()

answer = generate_answer(
    model=model,
    tokenizer=tokenizer,
    question="Which endpoint loads an adapter?",
    context="Use POST /v1/load_lora_adapter to load an adapter.",
    system_prompt="Answer only from the context. Copy technical names exactly.",
    max_new_tokens=64,
    device="cpu",
)
print(answer)

API

generate_answer(
    model,
    tokenizer,
    question: str,
    context: str,
    system_prompt: str,
    max_new_tokens: int = 64,
    device: str = "cpu",
) -> str

The supplied model and tokenizer must be compatible. The model must already be on the requested device and in evaluation mode. Model context limits are read from model.config.max_position_embeddings, falling back to 32768 if absent. Check actual model capabilities when switching models.

Build

uv build

The distributions are created in dist/.

Metadata

Release files for pythonai-grounded-generator 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pythonai-grounded-generator 0.1.0
File Size Uploaded Provenance Actions
pythonai_grounded_generator-0.1.0.tar.gz 4.4 kB Without provenance Details for pythonai_grounded_generator-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pythonai-grounded-generator 0.1.0
File Interpreter ABI Platform Actions
pythonai_grounded_generator-0.1.0-py3-none-any.whl Python 3 none any Details for pythonai_grounded_generator-0.1.0-py3-none-any.whl

Total release size: 8.4 kB

Release files / pythonai_grounded_generator-0.1.0.tar.gz

Download URL pythonai_grounded_generator-0.1.0.tar.gz
Size 4.4 kB
Tags Source
SHA-256 checksum
How to use checksums
cf6ebe921faaf5039232cb98070f8da55503e325011409da04416534aeb7b062
BLAKE2b-256 checksum
How to use checksums
790aaa24800fdbfe4f6de013be27e6627bdd0c87bb0498fa9214acf1074a7b99
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.9.13 {"installer":{"name":"uv","version":"0.9.13"},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"22.04","id":"jammy","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / pythonai_grounded_generator-0.1.0-py3-none-any.whl

Download URL pythonai_grounded_generator-0.1.0-py3-none-any.whl
Size 4.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
8c1c509dd900557e6838563001ce519df9bc611031526059e73797af95361b85
BLAKE2b-256 checksum
How to use checksums
7aa2359673eee62c2c5f64cabe6e0937cd2563a9f4766b29126102713b5e5e0b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.9.13 {"installer":{"name":"uv","version":"0.9.13"},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"22.04","id":"jammy","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for pythonai-grounded-generator 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page