deepeval-vllm
Compatibility wrapper for DeepEval's LocalModel that adds native structured output support for vLLM servers.
Why
DeepEval's built-in LocalModel talks to any OpenAI-compatible endpoint, but when a metric needs a structured result it just asks the model for JSON in the prompt and then tries to salvage a JSON object out of whatever came back (trim_and_load_json). With smaller local models that fails often enough to poison an eval run -- malformed JSON, missing fields, prose wrapped around the payload.
vLLM can enforce the schema server-side with constrained decoding. VLLMLocalModel passes the Pydantic schema through as vLLM's structured_outputs request field, so the response is guaranteed to match the schema instead of merely being asked to.
Everything else -- client construction, retries, multimodal input, config resolution from env vars -- is inherited from LocalModel unchanged.
Installation
uv add deepeval-vllm
Requires Python 3.12+.
Usage
Point it at your vLLM server and hand it to any DeepEval metric:
from deepeval_vllm import VLLMLocalModel
model = VLLMLocalModel(
model="Qwen/Qwen3-8B",
base_url="http://localhost:8000/v1",
# ...
)
The constructor signature is LocalModel's, so the usual env vars work too:
export LOCAL_MODEL_NAME="Qwen/Qwen3-8B"
export LOCAL_MODEL_BASE_URL="http://localhost:8000/v1"
export LOCAL_MODEL_API_KEY="not-needed"
model = VLLMLocalModel()
Requirements
- A running vLLM server with an OpenAI-compatible API, recent enough to accept the
structured_outputsrequest field (>=v0.12.0). Older builds useguided_jsoninstead and will reject or ignore these requests. deepeval >= 4.1.8
Development
uv sync
The package lives in deepeval_vllm/; vllm_wrapper.py overrides generate and a_generate and nothing else.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file deepeval_vllm-0.1.1.tar.gz.
File metadata
- Download URL: deepeval_vllm-0.1.1.tar.gz
- Upload date:
- Size: 98.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
269d6117998f8825ece4f6b7404a752d40f4237e0d2a8711360a408c5d3164c9
|
|
| MD5 |
732af1c7b673d3515f3111c13c75da01
|
|
| BLAKE2b-256 |
01c008b1f0d3052d334f51d7b52052b1e2bfb707c6339987bad98f7df627a699
|
Provenance
The following attestation bundles were made for deepeval_vllm-0.1.1.tar.gz:
Publisher:
python-publish.yml on profanta3/deepeval_vllm
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
deepeval_vllm-0.1.1.tar.gz -
Subject digest:
269d6117998f8825ece4f6b7404a752d40f4237e0d2a8711360a408c5d3164c9 - Sigstore transparency entry: 2581897799
- Sigstore integration time:
-
Permalink:
profanta3/deepeval_vllm@c0cb075e93a6550bcea400d72e2547380b41c4cf -
Branch / Tag:
refs/heads/main - Owner: https://github.com/profanta3
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
python-publish.yml@c0cb075e93a6550bcea400d72e2547380b41c4cf -
Trigger Event:
push
-
Statement type:
File details
Details for the file deepeval_vllm-0.1.1-py3-none-any.whl.
File metadata
- Download URL: deepeval_vllm-0.1.1-py3-none-any.whl
- Upload date:
- Size: 7.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
b4dacee23d1dd947c0847ad6214cebe5cf7561cb35a7ff7d2eabe8b9dfde7062
|
|
| MD5 |
f8005414c02205ca9cd68fbc856d34aa
|
|
| BLAKE2b-256 |
454798403c17df9b40a866102ebb9f88007832a64a25bff23b13873c9544452e
|
Provenance
The following attestation bundles were made for deepeval_vllm-0.1.1-py3-none-any.whl:
Publisher:
python-publish.yml on profanta3/deepeval_vllm
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
deepeval_vllm-0.1.1-py3-none-any.whl -
Subject digest:
b4dacee23d1dd947c0847ad6214cebe5cf7561cb35a7ff7d2eabe8b9dfde7062 - Sigstore transparency entry: 2581897807
- Sigstore integration time:
-
Permalink:
profanta3/deepeval_vllm@c0cb075e93a6550bcea400d72e2547380b41c4cf -
Branch / Tag:
refs/heads/main - Owner: https://github.com/profanta3
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
python-publish.yml@c0cb075e93a6550bcea400d72e2547380b41c4cf -
Trigger Event:
push
-
Statement type:
