This release is a pre-release and may not be stable for production use.
Evals, Red Teaming and Test Generation for Agentic Systems
Modular, Lightweight, Dynamic and Async-first
Docs • Website • Community
[!IMPORTANT] Giskard v3 is a fresh rewrite designed for dynamic, multi-turn testing of AI agents. This release drops heavy dependencies for better efficiency while introducing a more powerful AI vulnerability scanner and enhanced RAG evaluation — both now shipping natively in
giskard-scan(beta), with no dependency on v2. Only the legacy scan for tabular/ML models remains v2-only. Giskard v2 remains available but is no longer actively maintained. Follow progress → Read the v3 Announcement · Roadmap
Install
pip install giskard # checks (+ agents, llm, core)
pip install "giskard[scan]" # + vulnerability / quality scan
pip install "giskard[openai]" # provider SDK for LLM judges / generators
Requires Python 3.12+.
| Extra | Adds |
|---|---|
| (none) | giskard-checks and dependencies |
scan |
giskard-scan |
openai / anthropic / … |
provider SDKs (see pyproject.toml optional deps) |
Telemetry: optional aggregated analytics via giskard-core. No prompts or outputs are sent.
Opt out before importing Giskard: export DO_NOT_TRACK=1 or export GISKARD_TELEMETRY_DISABLED=1.
Details: giskard-core README.
Giskard is an open-source Python library for testing and evaluating agentic systems. The v3 architecture is a modular set of focused packages — each carrying only the dependencies it needs — built from scratch to wrap anything: an LLM, a black-box agent, or a multi-step pipeline.
| Status | Package | Description |
|---|---|---|
| ✅ Beta | giskard-checks |
Testing & evaluation — scenario API, built-in checks, LLM-as-judge |
| ✅ Beta | giskard-scan |
Agent vulnerability scanner + RAG/quality evaluation — red teaming, prompt injection, jailbreaks & harmful content (vulnerability_scan, successor of v2 Scan), plus knowledge-base quality eval (quality_scan, successor of v2 RAGET) |
These build on three foundational libraries — giskard-core (shared utilities & telemetry), giskard-llm (provider-agnostic LLM routing), and giskard-agents (agent & workflow orchestration) — which are pulled in automatically and rarely used directly.
Giskard Checks — create and apply evals for testing agents
pip install giskard-checks
Giskard Checks is a lightweight library for creating evaluations (evals) that test LLM-based systems — from simple assertions to LLM-as-judge assessments. Unlike traditional unit tests, evals are designed for non-deterministic outputs where the same input can produce different valid responses.
Use Giskard Checks to:
- Catch regressions — verify your system still behaves correctly after changes
- Validate RAG quality — check if answers are grounded in retrieved context
- Enforce safety rules — ensure outputs conform to your content policies
- Evaluate multi-turn agents — test full conversations, not just single exchanges
Built-in evals include string matching, comparisons, regex, semantic similarity, and LLM-as-judge checks (Groundedness, Conformity, LLMJudge).
Concepts
- Target — your system under test: any sync/async callable
(inputs) -> outputs(optionally withtrace) - Scenario — one eval: interactions + checks
- Check — assertion or LLM judge over the trace
- Suite — many scenarios run together
giskard.agents.Generator is an LLM client for workflows/judges — not the same as
giskard.checks input generators (LLMGenerator) that synthesize user messages.
Quickstart
import asyncio
from giskard.checks import Scenario, Groundedness
def get_answer(inputs: str) -> str:
return "Paris" # replace with your model / agent
async def main() -> None:
scenario = (
Scenario("test_france_capital")
.interact(inputs="What is the capital of France?", outputs=get_answer)
.check(
Groundedness(
name="answer is grounded",
context="France is in Western Europe. Its capital is Paris.",
)
)
)
result = await scenario.run()
result.print_report()
asyncio.run(main())
Groundedness is an LLM judge — install a provider extra (e.g. pip install "giskard[openai]") and set the matching API key. Default model: openai/gpt-4o-mini.
See the full docs for Suites, LLMJudge, multi-turn scenarios, and more.
Giskard Scan — vulnerability scanner for AI agents
pip install "giskard[scan]" # or: pip install giskard-scan
Giskard Scan is the red-teaming and vulnerability scanning layer for agentic systems. It generates adversarial test suites automatically from a plain-language description of your agent, covering prompt injection, harmful content, stereotypes, misinformation, and more.
Use Giskard Scan to:
- Red-team your agent — automatically generate adversarial inputs across OWASP LLM Top-10 threat categories
- Run prompt-injection probes — built-in dataset of injection payloads ready to use
- Extend with custom generators — pass your own
ScenarioGeneratorinstances togenerate_suite, or register them onvulnerability_suite_generator_registry
Quickstart
import asyncio
from giskard.scan import vulnerability_scan
async def my_agent(inputs: str) -> str:
# Replace with your agent / model call
return f"Echo: {inputs}"
async def main() -> None:
await vulnerability_scan(
target=my_agent,
description="A customer support chatbot for an e-commerce platform.",
languages=["en"],
)
asyncio.run(main())
Scan generators also need an LLM provider extra and API key (same as Checks judges above).
Looking for Giskard v2?
Giskard v2 included Scan (automatic vulnerability detection) and RAGET (RAG evaluation test set generation).
For LLM agents, both are superseded in v3 by giskard-scan: use vulnerability_scan in place of the v2 LLM scan, and quality_scan (with KnowledgeBase) in place of RAGET.
v3 works with ML models too — wrap one as a target and evaluate it with giskard-checks or giskard-scan. What the examples below cover is the v2-only automatic tabular scan — the detector suite that introspects a giskard.Model + giskard.Dataset to auto-detect performance, bias, and robustness issues — along with the giskard.testing ML test suite and the Giskard Hub. These are not planned for v3.
pip install "giskard[llm]>2,<3"
Scan — automatically detect performance, bias & security issues
Wrap your model and run the scan:
import giskard
import pandas as pd
# Replace my_llm_chain with your actual LLM chain or model inference logic
def model_predict(df: pd.DataFrame):
"""The function takes a DataFrame and must return a list of outputs (one per row)."""
return [my_llm_chain.run({"query": question}) for question in df["question"]]
giskard_model = giskard.Model(
model=model_predict,
model_type="text_generation",
name="My LLM Application",
description="A question answering assistant",
feature_names=["question"],
)
scan_results = giskard.scan(giskard_model)
display(scan_results)
RAGET — generate evaluation datasets for RAG applications
Automatically generate questions, reference answers, and context from your knowledge base:
import pandas as pd
from giskard.rag import generate_testset, KnowledgeBase
# Load your knowledge base documents
df = pd.read_csv("path/to/your/knowledge_base.csv")
knowledge_base = KnowledgeBase.from_pandas(df, columns=["column_1", "column_2"])
testset = generate_testset(
knowledge_base,
num_questions=60,
language="en",
agent_description="A customer support chatbot for company X",
)
👋 Community
We welcome contributions from the AI community! Read this guide to get started, and join our thriving community on Discord.
Follow the progress and share feedback: v3 Announcement · Roadmap
🌟 Leave us a star, it helps the project to get discovered by others and keeps us motivated to build awesome open-source tools! 🌟
❤️ If you find our work useful, please consider sponsoring us on GitHub. With a monthly sponsoring, you can get a sponsor badge, display your company in this readme, and get your bug reports prioritized. We also offer one-time sponsoring if you want us to get involved in a consulting project, run a workshop, or give a talk at your company.
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file giskard-3.0.0b3.tar.gz.
File metadata
- Download URL: giskard-3.0.0b3.tar.gz
- Upload date:
- Size: 12.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
uv/0.12.4 {"installer":{"name":"uv","version":"0.12.4","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
26132a32541fc635da87577a991c18f916bc52aabf71572011d0dbad83119814
|
|
| MD5 |
0870632b9881bbe00efea9e865cd163e
|
|
| BLAKE2b-256 |
12951bdf4da1a97f3d38aa70ade43f3839eae9fa795a5decc101ac50c2003893
|
File details
Details for the file giskard-3.0.0b3-py3-none-any.whl.
File metadata
- Download URL: giskard-3.0.0b3-py3-none-any.whl
- Upload date:
- Size: 10.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
uv/0.12.4 {"installer":{"name":"uv","version":"0.12.4","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
583de4d7e689400868dd2664b76f8ac53107dff243883c5a16ce03c3d99793c5
|
|
| MD5 |
8a417312c373b657187151ead71b7566
|
|
| BLAKE2b-256 |
9d7690d2799262d0927bd3f4c3ed7c3465afe5ed1fc4b4dcf3a74637619c0994
|
