testsmith-ai 0.4.0

Forge QA test cases from text, documents, Confluence pages, and Figma designs using LLMs

testsmith

Forge QA test cases from text, documents, Confluence pages, and Figma designs using LLMs.

testsmith is a CLI that takes a feature description and/or supporting sources (local files or URLs), sends them to an LLM (Anthropic Claude or Google Gemini), and writes the generated test cases to a CSV file.

Requirements

• Python >= 3.11
• An API key for one of the supported providers:
  • ANTHROPIC_API_KEY for Anthropic Claude
  • GOOGLE_API_KEY (or GEMINI_API_KEY) for Google Gemini
  • OPENAI_API_KEY for OpenAI or any OpenAI-compatible API (requires pip install testsmith-ai[openai])

Configuration

testsmith reads configuration from environment variables and/or a .env file in the current directory. Environment variables always take priority.

cp .env.example .env   # then edit .env with your values

See .env.example for all available settings.

Installation

Via pipx (recommended)

pipx installs CLI tools in isolated environments so they don't conflict with your system Python:

# Install pipx (pick your platform)
# macOS:   brew install pipx
# Linux:   python3 -m pip install --user pipx
# Windows: python -m pip install --user pipx

pipx install testsmith-ai

To upgrade later:

pipx upgrade testsmith-ai

Via pip (in a virtualenv)

# macOS / Linux
python3 -m venv .venv && source .venv/bin/activate

# Windows
python -m venv .venv && .venv\Scripts\activate

pip install testsmith-ai

From source (for development)

git clone https://github.com/priyankshah217/testsmith.git
cd testsmith

# macOS / Linux
python3 -m venv .venv && source .venv/bin/activate

# Windows
python -m venv .venv && .venv\Scripts\activate

pip install -e ".[dev]"

This installs the testsmith command on your PATH.

Usage

testsmith [OPTIONS]

You must provide at least one of --prompt, --file, or piped stdin.

Options

Option	Description
-p, --prompt TEXT	Plain text prompt / feature description.
-f, --file REF	Input source: local file (PDF, DOCX, MD, TXT) or URL (e.g. Confluence page). Repeatable.
-o, --out PATH	Output CSV path. If omitted, the LLM suggests a kebab-case filename based on the feature (e.g. login-social-auth.csv). Collisions get _2, _3, ...
--provider TEXT	LLM provider: anthropic, gemini, or openai. Auto-detected from env if omitted.
-m, --model TEXT	LLM model name (e.g. claude-sonnet-4-6, gemini-2.5-flash). Defaults per provider.
-t, --temperature FLOAT	Sampling temperature (0.0–2.0). Lower = more deterministic.
--top-p FLOAT	Nucleus sampling top-p (0.0–1.0).
--format TEXT	Test step format: steps (default, numbered) or bdd (Given/When/Then, business-focused).
--trace	Add source traceability columns (document, section, quote, derivation) to CSV output.
--max-tokens INT	Maximum output tokens for LLM response (default 16384). Increase for large prompts or thinking models.
--debug	Dump raw LLM response to debug_response.txt for troubleshooting parse failures.
-s, --system TEXT	Custom system prompt. Inline text or @path/to/file.txt. Replaces the default.
--append-system	Append --system to the default system prompt instead of replacing it.
-u, --user-template TEXT	Custom user prompt template. Inline text or @path/to/file.txt. Use {context} as a placeholder.
-i, --interactive	Let the LLM ask clarifying questions adaptively before generating. It asks one question at a time and stops as soon as it has enough context (0–5 questions). Type skip to skip a question or done to stop early.

Supported input sources

Source	Example
Plain text	-p "Login screen with social auth"
PDF	-f ./specs/checkout.pdf
DOCX	-f ./specs/payment.docx
Markdown / text	-f ./notes.md
Confluence page	-f https://acme.atlassian.net/wiki/spaces/ENG/pages/12345/Checkout-PRD
Figma design (text-only)	-f "https://www.figma.com/design/<fileKey>/<name>?node-id=1-23"

Adding new sources (Notion, Jira, Linear, ...) is a single file in testsmith/sources/ — see testsmith/sources/base.py for the Source protocol.

Confluence setup

Set these in your .env file or as environment variables:

CONFLUENCE_BASE_URL=https://your-site.atlassian.net
CONFLUENCE_EMAIL=[e-mail]
CONFLUENCE_API_TOKEN=<token from id.atlassian.com/manage-profile/security/api-tokens>

Figma setup

Set in your .env file or as an environment variable:

FIGMA_API_TOKEN=<token from figma.com/settings>

OpenAI / LiteLLM gateway setup

First install the optional OpenAI dependency:

pip install testsmith-ai[openai]

Set in your .env file or as environment variables:

OPENAI_API_KEY=<your API key or gateway token>
OPENAI_BASE_URL=https://your-litellm-gateway.internal/v1  # optional, for custom endpoints

This works with any OpenAI-compatible API: OpenAI, LiteLLM gateway, Azure OpenAI, Ollama, vLLM, etc. Use --model to specify the model name your gateway exposes:

testsmith --provider openai --model gpt-4o -p "Login flow"
testsmith --provider openai --model company/internal-llm -p "Login flow"  # custom gateway model

Figma support is text-only in v1: frame/component names become headings, text layers become body text, and component descriptions are preserved. Purely visual nodes (vectors, rectangles, etc.) are skipped. If the URL contains a node-id, only that subtree is fetched; otherwise the whole file is loaded. Tip: right-click a frame in Figma → Copy link to get a URL with the right node-id.

Examples

Generate from an inline prompt (LLM picks the filename):

testsmith -p "Login screen with email + password, forgot password link, and social login"

Generate from documents:

testsmith -f specs/checkout.pdf -f specs/payment.docx

Generate from a Confluence page:

testsmith -f https://acme.atlassian.net/wiki/spaces/ENG/pages/12345/Checkout-PRD

Generate from a Figma design (quote the URL — the ? will be interpreted by your shell otherwise):

testsmith -f "https://www.figma.com/design/ABC123/Checkout?node-id=42-1337"

Mix a prompt with supporting files, Confluence pages, and Figma designs:

testsmith -p "Focus on edge cases" \
  -f "https://www.figma.com/design/ABC123/Checkout?node-id=42-1337" \
  -f https://acme.atlassian.net/wiki/spaces/ENG/pages/12345/PRD \
  -f specs/wireframes.pdf

Pipe text via stdin:

cat feature.md | testsmith

Pick a provider explicitly:

testsmith -p "Signup flow" --provider anthropic

Interactive mode (LLM asks clarifying questions only when genuinely ambiguous):

testsmith -i -p "Checkout flow with guest and returning users"

Generate BDD-style test cases (Given/When/Then, business-focused):

testsmith -p "Subscription renewal flow" --format bdd

Use a specific model with custom temperature:

testsmith -p "Signup flow" -m claude-sonnet-4-6 -t 0.3

Use a custom system prompt from a file:

testsmith -f spec.pdf --system @prompts/qa_system.txt

Append to the default system prompt instead of replacing it:

testsmith -f spec.pdf --system "Focus on accessibility test cases." --append-system

Force an explicit output filename:

testsmith -p "Password reset flow" -o reset.csv

Output

Test cases are written to a CSV with columns: ID, Title, Preconditions, Steps, Expected Result, Priority, Type

When --trace is enabled, additional source traceability columns are appended: source.document, source.section, source.quote, source.derivation

By default the filename is suggested by the LLM based on the feature context; pass -o to override.

Step formats

--format	Steps column example
steps (default)	1. Open app (each step on its own line)
bdd	Given user has an active account / When user provides valid credentials / Then user is authenticated (each keyword on its own line)

BDD mode enforces business-focused language — steps describe domain intent and outcomes, not UI interactions (no "click", "tap", "navigate", etc.).

Quality enforcement

testsmith automatically validates generated test cases and fixes issues:

1. Code-level check — scans output for hedging language ("may", "likely", "e.g."), non-specific steps, precondition/step duplication, and duplicate test cases.
2. LLM-as-judge — if warnings are found, the output is sent to a second LLM call with a QA Reviewer persona that rewrites only the flagged fields with specific, deterministic values.
3. Re-check — the corrected output is validated again. Remaining warnings (if any) are displayed but do not block CSV output.

This two-layer approach (regex + LLM judge) catches issues that prompt-level self-check instructions alone cannot enforce reliably across all providers.

Claude Code skill

Testsmith ships with a Claude Code skill so Claude can run testsmith for you when you ask for test cases.

Install the skill

Copy the bundled skill to your Claude Code skills directory:

# macOS / Linux
cp -r skills/testsmith ~/.claude/skills/

# Windows (PowerShell)
Copy-Item -Recurse skills\testsmith $env:USERPROFILE\.claude\skills\

Or create a symlink so it stays in sync with the repo:

ln -s "$(pwd)/skills/testsmith" ~/.claude/skills/testsmith

Once installed, just ask Claude Code: "generate test cases for the login screen" — it will invoke testsmith automatically.

Contributing

git clone https://github.com/priyankshah217/testsmith.git
cd testsmith
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest tests/ -v