Skip to main content

PythonAI Grounded Generator

A compact Python helper for generating answers from retrieved context with an already loaded Hugging Face causal language model.

It provides:

  • Custom system prompts and questions
  • Token-budget-aware context truncation
  • Qwen3 non-thinking chat formatting
  • CPU or CUDA inference (device controlled by the caller)
  • A plain-text answer suitable for wrapping in application-specific Pydantic models

Retrieval, context assembly, and structured JSON serialization remain the responsibility of the calling application. This package does not guarantee factual correctness or replace source validation.

Installation

After publication:

pip install pythonai-grounded-generator

Or install the local project:

pip install .

Example

from transformers import AutoModelForCausalLM, AutoTokenizer
from pythonai_grounded import generate_answer

name = "Qwen/Qwen3-0.6B"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name).to("cpu")
model.eval()

answer = generate_answer(
    model=model,
    tokenizer=tokenizer,
    question="Which endpoint loads an adapter?",
    context="Use POST /v1/load_lora_adapter to load an adapter.",
    system_prompt="Answer only from the context. Copy technical names exactly.",
    max_new_tokens=64,
    device="cpu",
)
print(answer)

API

generate_answer(
    model,
    tokenizer,
    question: str,
    context: str,
    system_prompt: str,
    max_new_tokens: int = 64,
    device: str = "cpu",
) -> str

The supplied model and tokenizer must be compatible. The model must already be on the requested device and in evaluation mode. Model context limits are read from model.config.max_position_embeddings, falling back to 32768 if absent. Check actual model capabilities when switching models.

Build

uv build

The distributions are created in dist/.

Metadata

Release files for pythonai-grounded-generator 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pythonai-grounded-generator 0.1.1
File Size Uploaded Provenance Actions
pythonai_grounded_generator-0.1.1.tar.gz 5.0 kB Without provenance Details for pythonai_grounded_generator-0.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pythonai-grounded-generator 0.1.1
File Interpreter ABI Platform Actions
pythonai_grounded_generator-0.1.1-py3-none-any.whl Python 3 none any Details for pythonai_grounded_generator-0.1.1-py3-none-any.whl

Total release size: 9.5 kB

Release files / pythonai_grounded_generator-0.1.1.tar.gz

Download URL pythonai_grounded_generator-0.1.1.tar.gz
Size 5.0 kB
Tags Source
SHA-256 checksum
How to use checksums
fc78ca517eea94ba8be4586978d059cdcb15d887fd4bc3f05cfee4adb4199c70
BLAKE2b-256 checksum
How to use checksums
7e130b8416fbcb84aeeb36538b21f4bfb76915fd159c5569489780ca7088f764
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.9.13 {"installer":{"name":"uv","version":"0.9.13"},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"22.04","id":"jammy","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / pythonai_grounded_generator-0.1.1-py3-none-any.whl

Download URL pythonai_grounded_generator-0.1.1-py3-none-any.whl
Size 4.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
2072d30acdd3479085a8302e8c1defb74ae686e3f05981e3ad10d5bad05373d6
BLAKE2b-256 checksum
How to use checksums
9d7f052266b53d441e15c1ba2f84a3a4b176e7db5bc7acff402970e8c88fc02c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.9.13 {"installer":{"name":"uv","version":"0.9.13"},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"22.04","id":"jammy","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for pythonai-grounded-generator 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page