lingtest-cli 0.2.0

AI testing agent for PRDs, OpenAPI specifications, and Vibe Coding workflows

LingTest CLI

简体中文 | English

LingTest CLI is a local-first AI testing agent for Vibe Coding workflows. It analyzes PRDs, generates validated test plans from requirements and OpenAPI, runs deterministic HTTP tests, and diagnoses failures with structured evidence.

The deterministic core does not require an LLM or API key. Generated cases are plain JSON, so they can be reviewed and changed before any request is sent.

Status: 0.2.0 alpha. The file format and Python API may evolve before 1.0.

What's New in 0.2.0

• Independent OpenAI-compatible provider layer with no FastAPI, Redis, database, or worker dependency.
• Built-in provider presets for OpenAI, DeepSeek, and Qwen, plus custom base_url and model support.
• lingtest analyze for PRD ambiguity, contradiction, missing requirement, and testability-risk analysis.
• lingtest ai-generate for functional, boundary, exception, and API test plans.
• lingtest diagnose for evidence-grounded failure classification and advice.
• Pydantic validation for every AI result, with one controlled correction retry.
• Provider credentials read from environment variables and excluded from cases and reports.
• Expanded Chinese and English documentation, product design, and Vibe Coding testing article.

Features

• Read OpenAPI specifications in YAML or JSON.
• Discover GET, POST, PUT, PATCH, DELETE, HEAD, and OPTIONS operations.
• Generate values from example, default, enum, and schema types.
• Resolve path parameters and generate query, header, and JSON body values.
• Select the first documented 2xx response as the expected status.
• Run cases with HTTPX using a configurable base URL and timeout.
• Add runtime headers without storing credentials in generated cases.
• Write detailed JSON reports and optional JUnit XML.
• Return CI-friendly exit codes.
• Use the same core through a small Python API.
• Analyze PRDs for ambiguity, contradictions, missing requirements, and testability risks.
• Generate validated functional, boundary, exception, and API test plans with an LLM.
• Diagnose failed HTTP tests with evidence-grounded structured output.
• Connect to OpenAI, DeepSeek, Qwen, or a custom OpenAI-compatible provider.

Requirements

• Python 3.11 or newer
• An OpenAPI 3.x YAML or JSON document
• Network access to the API under test when running cases

Installation

Standard pip

Install into the active Python environment or virtual environment:

python -m pip install lingtest-cli
lingtest --version

Use this when LingTest should share an existing project environment or when pip is your standard package manager.

Isolated CLI with pipx (recommended)

pipx install lingtest-cli

pipx creates a dedicated environment while making the lingtest command available globally. This avoids dependency conflicts with application projects.

Isolated CLI with uv

uv tool install lingtest-cli

Add to a UV project

uv add --dev lingtest-cli
uv run lingtest --help

All installation methods provide the same lingtest command and Python API.

Quick Start

Generate cases:

lingtest generate openapi.yaml -o cases.json

Review cases.json, then run it:

lingtest run cases.json \
  --base-url http://localhost:8000 \
  --junit junit.xml \
  -o report.json

PowerShell authentication example:

lingtest run cases.json `
  --base-url https://api.example.com `
  -H "Authorization:Bearer $env:API_TOKEN" `
  --junit junit.xml

Use -H multiple times to add more headers:

lingtest run cases.json --base-url https://api.example.com \
  -H "Authorization:Bearer ${API_TOKEN}" \
  -H "X-Tenant-ID:demo"

Analyze requirements and generate an AI test plan:

export OPENAI_API_KEY="..."
lingtest analyze requirements.md -o analysis.json
lingtest ai-generate requirements.md -o ai-cases.json

Diagnose failed cases from a LingTest JSON report:

lingtest diagnose report.json -o diagnosis.json

Use --provider deepseek, --provider qwen, or a custom OpenAI-compatible endpoint with --provider custom --base-url URL --model MODEL.

Test Data Flow

flowchart LR
    PRD["PRD / Requirements"] --> Analyze["lingtest analyze"]
    Analyze --> Analysis["analysis.json<br/>ambiguities and risks"]

    PRD --> AIGenerate["lingtest ai-generate"]
    OpenAPI["OpenAPI YAML / JSON"] --> AIGenerate
    AIGenerate --> AIPlan["ai-cases.json<br/>validated test plan"]
    AIPlan --> Review["Human review"]

    OpenAPI --> Generate["lingtest generate"]
    Generate --> Cases["cases.json<br/>executable HTTP cases"]
    Review -.->|planned 0.3 conversion| Cases

    Cases --> Run["lingtest run"]
    Runtime["Base URL + runtime headers"] --> Run
    Run --> JSONReport["report.json<br/>original evidence"]
    Run --> JUnit["junit.xml<br/>CI result"]

    JSONReport --> Diagnose["lingtest diagnose"]
    Diagnose --> Diagnosis["diagnosis.json<br/>classification, evidence, advice"]

The solid path is implemented in 0.2.0. AI plans remain review artifacts; automatic conversion from an approved AI plan to executable HTTP cases is planned for 0.3. Diagnosis produces a separate file and never overwrites the original run report.

Commands

lingtest analyze

Finds ambiguities, contradictions, missing requirements, and testability risks in .txt, .md, .json, .yaml, and .yml documents.

lingtest ai-generate

Generates a validated plan containing functional, boundary, exception, and API test cases. AI cases are review artifacts and are never executed implicitly.

lingtest diagnose

Reads a LingTest JSON run report and classifies failed cases using the supplied execution evidence.

lingtest generate

lingtest generate SPEC [-o OUTPUT]

SPEC is an OpenAPI YAML or JSON file. The default output is lingtest-cases.json.

lingtest run

lingtest run CASES_FILE --base-url URL [OPTIONS]

Important options:

Option	Description	Default
--base-url	Base URL of the API under test. It can also be provided through LINGTEST_BASE_URL.	Required
-o, --output	JSON report path.	lingtest-report.json
--junit	Optional JUnit XML report path.	Disabled
--timeout	Per-request timeout in seconds.	10.0
-H, --header	Runtime HTTP header in NAME:VALUE form. Repeatable.	None

Exit codes:

Code	Meaning
0	All test cases passed.
1	One or more test cases failed or produced a request error.
2	The command input or case file is invalid.

Case File

Generated files are JSON arrays. A case looks like this:

{
  "id": "get_user",
  "title": "Get user",
  "method": "GET",
  "path": "/users/42",
  "expected_status": 200,
  "headers": {},
  "query": {
    "verbose": true
  },
  "body": null
}

Edit expected_status, request values, or titles before execution. Do not put long-lived credentials in this file; inject them with --header and environment variables at runtime.

Reports

The JSON report contains the start time, base URL, and a result for each case:

• case ID and title
• passed, failed, or error status
• request duration in milliseconds
• expected and actual HTTP status
• a concise error message

JUnit output can be uploaded to GitHub Actions, Jenkins, GitLab CI, Azure Pipelines, and other systems that support JUnit XML.

Python API

from lingtest import generate_cases, run_cases

cases = generate_cases("openapi.yaml")
report = run_cases(cases, "http://localhost:8000")

print(f"{report.passed} passed, {report.failed} failed")
for result in report.results:
    print(result.case_id, result.status, result.duration_ms)

The public data models are TestCase, TestResult, and RunReport.

CI Example

name: API tests

on: [push, pull_request]

jobs:
  lingtest:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v6
      - run: uv tool install lingtest-cli
      - run: lingtest generate openapi.yaml -o cases.json
      - run: lingtest run cases.json --base-url "${{ secrets.TEST_API_URL }}" --junit junit.xml

The target API must be reachable from the runner. Start the service in an earlier step or point the command at a deployed test environment.

AI Providers

Provider	Option	API key environment variable	Default model
OpenAI	--provider openai	OPENAI_API_KEY	gpt-4.1-mini
DeepSeek	--provider deepseek	DEEPSEEK_API_KEY	deepseek-chat
Qwen	--provider qwen	DASHSCOPE_API_KEY	qwen-plus
Custom	--provider custom	LINGTEST_API_KEY	Set with --model

Provider responses are parsed as JSON and validated with Pydantic. LingTest requests one correction when a response fails validation. Documents are sent to the configured provider; review its privacy policy before processing confidential requirements.

Current Limitations

• $ref resolution and advanced schema composition are not implemented yet.
• Assertions currently compare the HTTP status only.
• Cases run sequentially.
• OpenAPI security schemes are not applied automatically.
• PDF and DOCX document extraction are not implemented.
• AI-generated cases are review artifacts and are not automatically converted into executable HTTP cases.

See product-design.html for implemented scope, design decisions, and the roadmap.

The Chinese essay AI Agent 时代如何打造高质量软件 explains the Vibe Coding quality problem behind this project.

Development

Clone the repository and install all development dependencies:

uv sync --extra dev
uv run pytest
uv run ruff check .

Build and validate distributions:

uv build
uv run twine check dist/*

Run the development CLI:

uv run lingtest --version
uv run lingtest generate examples/openapi.yaml -o cases.json

Release

1. Update the version in pyproject.toml and src/lingtest/__init__.py.
2. Run tests, Ruff, uv build, and twine check.
3. Commit the release and create a matching Git tag.
4. Publish with UV_PUBLISH_TOKEN:

uv publish

Security

Only run generated cases against systems you are authorized to test. Runtime headers are not written to the report by version 0.2, but generated case files and reports should still be treated as project data.

Roadmap

• Better OpenAPI schema and $ref support
• Configuration files and named environments
• Filtering by tag, operation, path, and method
• JSON-path, schema, header, and latency assertions
• Authentication strategies
• Controlled concurrency and retries
• Static HTML reports
• Conversion of reviewed AI API cases into deterministic executable cases
• pytest export and a stable plugin API

License

MIT