IATest turns a business goal into an observed UI journey, independent replay and a reusable test plan. It also imports API contracts, runs bounded load and security checks, validates data and messages, and connects those tests into larger business workflows.

Use the local Web console or CLI. Keep goals, cases, plans, rules and measurement assets together. Review original assertions and evidence alongside any AI explanation.

Version 2.1.1 hardens console sessions, credential destinations and configuration updates, and adds explicit assessment counts and release quality gates. Read the upgrade notes and console security guide. The file asset layout introduced in 2.1.0 remains compatible.

Quick start

Requires Python 3.11+. Extra tools depend on the target you test.

python -m pip install "iatest-framework==2.1.1"
mkdir my-tests
cd my-tests
iatest init --project demo
iatest web --project demo

Open the one-time authorization link printed by iatest web. Enroll a model origin and credential reference locally with iatest llm-trust, then select that connection in Settings. See the connection setup. Then open Test workspace → UI & interaction:

1. Choose a browser, Android, iPhone/iPad or desktop target.
2. Describe the business goal and a concrete expected outcome.
3. Start AI goal exploration and review the independent replay results.
4. Open the saved case and plan in Test assets → UI tests.

For example:

Set the nickname to “Test User”, save it, then reload the page. The test passes only if the nickname still equals “Test User”.

No selectors or complete click script are required for this entry path. You supply the target and trusted expectation; IATest discovers the steps. Starting exploration uses the configured model and acts on the target application.

Browsers need an installed Chromium-family browser or a supported WebDriver backend. Native mobile and desktop tests need the platform tools and permissions in the environment guide.

Run a test without an AI model

Create records.json:

{
  "id": "unique-records",
  "kind": "data",
  "spec": {
    "records": [{"id": 1}, {"id": 2}],
    "checks": [{"id": "unique-id", "rule": "unique", "path": "$.id"}]
  }
}

iatest test import-plan records.json --project demo
iatest test run unique-records --project demo
iatest test reports --project demo

Saved deterministic plans run without model calls. AI is used when you request exploration, generation, visual assistance or evidence analysis.

For complete API, data and browser examples, including CI exit semantics, see deterministic execution.

How it works

Goal → observed actions → independent replay → reusable assets → evidence.

• Declare the outcome. Checks cite your goal and trusted rules before state-changing actions. Current screen contents do not become the expected answer.
• Observe and explore. Bounded native or browser tools locate controls and execute authorized actions. Manual step composition is also available.
• Replay independently. UI exploration performs at least two fresh-session replays. Stable success and stable business failure are recorded separately; incomplete cases remain candidates.
• Reuse the plan. Saved plans can run again, join workflows, execute on authorized nodes or become part of release verification.
• Keep the evidence. Assertions, screenshots, measurements, errors and cleanup results remain distinct from model interpretations.

The console separates Test workspace, Devices & environments, Test assets, Runs & reports, Project knowledge and Settings. The asset library treats UI, API, performance and security assets consistently, with filtering and links between goal cases and compiled plans.

Project assets follow projects/<project>/assets/<test-type>/: API, Web, desktop, Android, iOS, performance and security each have readable plan, case and rule files. Results and evidence live under reports/<test-type>/; SQLite provides rebuildable query views and internal state. Existing layouts migrate without deleting the originals. See project storage and migration.

What you can test

iatest test capabilities
iatest test doctor
iatest mobile doctor

Platform support has specific boundaries. Safari uses an explicit navigation-only mode; some Apple raw collectors remain unreliable; BLE digitizer reports are experimental; physical phones do not provide full OS isolation. See the capability matrix before selecting a backend.

Optional dependencies

# PostgreSQL/MySQL drivers and SQL authorization checks
python -m pip install "iatest-framework[database]==2.1.0"

# Pinyin support for the optional HID Chinese-input channel
python -m pip install "iatest-framework[hid]==2.1.0"

These extras do not install browsers, Android SDKs, Xcode or HID hardware.

Examples

The UI lab includes a deliberately broken save path for comparison. Demo services bind to loopback; replace their addresses and identities before testing your own environment. Examples are source-repository resources; installed users can start directly from the console or the JSON plan above.

FAQ

Do all tests need an AI model?

No. Saved plans and deterministic checks execute without a model. Exploration, generation and AI analysis require a compatible configured provider.

Is a finished task a passed test?

No. Task lifecycle and verdict are separate. Missing measurements, ambiguous targets, incomplete observation and unknown input delivery cannot pass. AI analysis does not replace the original verdict.

Is everything local?

The console, execution and asset storage run locally. Requested model operations send selected context, and explicitly enabled images, to your configured provider. Its data policy applies. Keep credentials in environment-variable references.

Do mobile tests need Appium or a jailbroken iPhone?

The framework owns its native drivers and does not require Appium. Standard iOS testing uses a signed XCTest Runner. Optional Android-to-iPhone Bluetooth HID and USB HID channels have their own setup and compatibility requirements.

Does installing the package complete device setup?

No. Browser drivers, mobile SDKs, signatures, native components, interactive desktops and hardware permissions are target-specific. Start with the environment and platform guides.

What changed in 2.0?

The Python package is now iatest; update imports and python -m ai_apitest to python -m iatest. The PyPI distribution remains iatest-framework and the command remains iatest. Examples and console filenames now describe the wider testing scope. See release and migration notes.

Documentation

• Documentation index · Getting started · Web console
• UI goal exploration · Browser · Desktop · Mobile
• Test plans & workflows · Performance & security · Data & protocols
• Architecture · Knowledge & agents · Capability boundaries

Development

git clone https://github.com/hinayoung23/IATest.git
cd IATest
python -m venv .venv
# Activate .venv using your shell's normal command.
python -m pip install -e ".[test,database,hid]"
python -m pytest -q

Browser and native integration tests require their documented environments; skipped tests are not device acceptance. Generated projects, credentials, recordings and build products are excluded from source control and release packages. See CONTRIBUTING for layout and release checks.

Licensed under Apache-2.0.