pytest-Jira-xray-plus

Start here: runnable quick start

Keep every parametrized pytest case in a local JSON file, and optionally put case details into one Jira/Xray Test Run comment. A case is one set of inputs; a Test Run is Xray's result for one Jira Test issue in a Test Execution. This package does not create native Xray iteration rows or datasets.

This guide covers 0.1.0, the stable release of local case reporting and the optional aggregate Xray comment fallback. Michael reported successful validation and authorized this release. That is user-reported evidence, not independently observed exhaustive validation of his server. Install the exact version below from PyPI. Native iteration rows and datasets are unsupported. Xray 8.5.3-j10's full API behavior and Automatic's effective type mapping were not independently verified.

This is an independent Apache-2.0 fork of pytest-jira-xray, based on upstream commit 6ac1117585da590001cdbc2bf261b78683c19dcb (0.9.4 line).

Quick start (no Jira required)

Use Python 3.12 or newer. The tested local baseline is Python 3.12.14, pytest 9.1.1, requests 2.34.2, pytest-xdist 3.8.0 and pytest-rerunfailures 16.7 on Linux. Metadata permits pytest >=9.1.1,<10 and requests >=2.34.2,<3; that is not a promise that every future version or OS has been tested. CI checks Python 3.12, 3.13 and 3.14; see its results before relying on those environments.

Create an empty folder and a fresh Python environment. The environment isolates this installation from other projects. These commands use a POSIX shell:

mkdir xray-demo
cd xray-demo
python3.12 -m venv .venv
. .venv/bin/activate
python -m pip install pytest-jira-xray-plus==0.1.0

On Windows, use py -3.12 -m venv .venv and activate with .venv\Scripts\Activate.ps1 in PowerShell or .venv\Scripts\activate.bat in Command Prompt. Subsequent python -m ... commands are the same. No Git checkout, Jira account, credentials, or configuration file is required. Installation downloads packages; the following test runs do not contact Jira.

Do not install pytest-jira-xray in this environment. Both distributions provide the same Python module (pytest_xray) and pytest plugin name (xray). They overwrite each other's files. If both were installed, start with a new environment; disabling one plugin does not repair overwritten files.

Create a file named test_example.py in xray-demo, next to .venv:

import pytest

@pytest.mark.xray("DEMO-1")
@pytest.mark.parametrize(
    "value, expected",
    [(1, 2), (2, 3)],
    ids=["one-plus-one", "two-plus-one"],
)
def test_add_one(value, expected):
    assert value + 1 == expected

parametrize runs the function twice. The readable ids identify each input set. DEMO-1 is a placeholder Jira Test key; it is not looked up locally. The marker is only needed for the optional Xray export later.

python -m pytest -q test_example.py --xray-case-report=cases.json

Expected: 2 passed, exit code 0, and cases.json in the current folder. It has two separate cases, each with its node ID (test path plus input ID), status: "passed", and an attempts list recording setup, call and teardown. Setup prepares fixtures, call runs the test, and teardown cleans up. Parameter values and failure text are excluded from this file by default. Inspect it with:

python -m json.tool cases.json

To see a failure, change (2, 3) to (2, 99) and rerun. Expect 1 failed, 1 passed, exit code 1, and two preserved case records, one failed and one passed. The file is replaced on each run: use different paths to keep history. cases.json is not Xray import JSON. Do not upload it to Xray.

Optional: preserve parameter details in an Xray comment

With the same example, run:

python -m pytest -q test_example.py --xray-case-report=cases.json \
  --jira-xray --xraypath=xray.json --allow-duplicate-ids \
  --xray-parameter-details

This writes xray.json beside cases.json. No upload occurs because --xraypath selects a file. Keep that option for all offline Xray experiments, including when --jira-xray comes from pytest.ini or PYTEST_ADDOPTS.

xray.json has one entry for DEMO-1: both cases share the same Jira key. Its comment contains both case identities, scalar input values, phase and outcome. With the original example its status is PASS; with the intentional failure it is FAIL. --allow-duplicate-ids permits this grouping; without it repeated Jira keys are rejected. Multiple keys can be given as @pytest.mark.xray("DEMO-1", "DEMO-2"), which copies the result to both Tests.

The fallback is opt-in because comments can expose data and grow large. Published 0.1.0a1 has no size limit. 0.1.0 uses a 32,000 UTF-8 byte limit per final comment, configurable with --xray-comment-limit (minimum 256). This is our conservative client policy, not a universal Jira/Xray server limit; a server may impose a different limit. The bound includes the truncation notice and any wiki formatting wrapper, but does not bound evidence or total request size.

Add --xray-full-report=full-xray.json to preserve the entire original Xray payload locally before shortening any comments. The main export (either xray.json or the upload) then contains a clearly marked shortened comment, and pytest prints how many comments were truncated. Without a full-report path, an oversized comment stops export with a nonzero exit rather than silently losing detail. If writing the archive fails, export also stops. Status and evidence are never truncated. Keep cases.json for structured cases/attempts; it remains unbounded and is independent of the full-payload archive.

full-xray.json contains full comments, values, failure text and evidence: protect it like the original test data. It is replaced each run; use a unique path to keep history. It must differ from cases.json and xray.json. Do not upload it to bypass the chosen bound. This extra file is written only when explicitly requested. Check xray.json before use and keep both local reports when exporting large suites. A successful import should show one aggregate Test Run and its comment, not separate native iteration rows. Successful use was reported by Michael; exact server rendering was not independently observed. Only parametrized reports get the extra parameter comment. Failure text is included in legacy comments even without this option.

Use synthetic inputs and safe explicit IDs. IDs can contain data even when values are excluded. --xray-case-redact-identities replaces names and IDs only in cases.json; it does not redact console output, xray.json, comments, captures or evidence. Non-scalar and non-finite values get a placeholder in 0.1.0; alpha 0.1.0a1 only filters non-scalar objects in comments. Indirect fixtures record their input parameters, not the fixture's return value.

Optional: send results to Jira/Xray Server or Data Center

Only do this in an authorized disposable test project first. The development automation did not access any workplace server. The target is Xray 8.5.3-j10; the custom Automatic type's apparent Generic mapping remains independently unverified. Ask the administrator to confirm the actual type; the label alone is insufficient.

Replace DEMO-1 in your test with an existing sandbox Test key. Obtain an existing sandbox Test Execution key, for example DEMO-100. The following Bash example reads an existing personal access token without putting it in shell history. Use your CI's protected environment variables for automation; do not commit credentials or put them in pytest options.

export XRAY_API_BASE_URL='https://jira.example.invalid'
read -r -s -p 'Existing sandbox Jira token: ' XRAY_API_KEY
printf '\n'
export XRAY_API_KEY
python -m pytest -q test_example.py --jira-xray --api-key-auth \
  --execution=DEMO-100 --allow-duplicate-ids --xray-parameter-details \
  --xray-case-report=cases.json
unset XRAY_API_KEY

The .invalid URL is a nonworking placeholder: replace it with the authorized sandbox base URL, including Jira's context path if any, but no REST suffix. This command uploads, because --xraypath is absent. It submits one JSON request to /rest/raven/2.0/import/execution. Do not use --cloud for Server/DC. On success pytest prints the returned Test Execution ID. Inspect the Test Run's comment in that execution. Permissions, custom statuses, proxy configuration, comment limits and API response shape need sandbox validation.

Supplying --execution updates that execution. Omitting it asks Xray to create an execution using the report metadata, subject to server requirements. Reimporting can replace existing Test Run details, including dataset/iteration data; use a disposable execution. The official Server/DC import API documents this replacement behavior. This package does not retry imports automatically: a timeout can occur after a server accepted a request. Check the execution before retrying.

Authentication choices (choose one):

Cloud uses the Xray API base URL, /api/v2/authenticate and /api/v2/import/execution; it is a different service from the Jira URL. --cloud changes status strings to PASSED/FAILED, and does not convert Generic tests into Manual tests or enable native iterations. Cloud behavior has only synthetic tests, not live validation.

TLS certificate verification is on by default. XRAY_API_VERIFY_SSL accepts True, or a path to a trusted CA certificate bundle for an internal CA. False disables verification and should not be used with real credentials.

Optional execution metadata is read from these environment variables:

All package options

See python -m pytest --help for pytest's own options.

No options enable native iterations. --collect-only does not create reports. Use different output paths for the two report formats. Local report identity aliases are assigned in arrival order and cannot be compared across sessions.

Results, retries and exact limitations

The local report deliberately remembers any failure, even if pytest's retry plugin eventually exits 0. The legacy fix introduced in 0.1.0a2 uses that same conservative retry rule. Legacy grouping still uses upstream's priority: ABORTED outranks FAIL, so skip plus failure produces ABORTED; failure remains in the comment. This differs from local failure-first grouping and is retained for migration.

Use python -m pip install pytest-xdist==3.8.0 pytest-rerunfailures==16.7 for the tested optional parallel/retry plugins. Add -n 2 to run on two workers, or --reruns=1 to retry once. Workers are separate processes; the main process writes one report. Completion order is not guaranteed. --dist=each observations of the same node are stored as separate worker attempts. Test phases and retries are observations, not additional parametrized cases.

Local files include the pytest exit status and only observed reports, not a full inventory of unexecuted tests. Collection errors and interruption produce partial files when pytest reaches session finish. Worker crashes may lack parameter metadata and Jira keys. Hard process termination may prevent any output. Subtests, repeat plugins, crash recovery identity, cross-session merging and per-iteration evidence are not supported as distinct case models. Do not assume one record per subtest or one stable attempt ID across restarted workers.

Release behavior (introduced in 0.1.0a2): requests have a 10-second connect and 60-second read-inactivity timeout (not a total deadline); redirects are rejected; authentication and response shapes are checked; response bodies are not dumped to shared temporary files. Failed publishing changes an otherwise successful pytest exit to 3; existing nonzero exits are preserved. Uploads are withheld for worker crashes, interrupted/collection-error sessions (including continue-on-error), fail-fast/maxfail stops, or empty marked results. Local Xray file writes are atomic and retain the previous file on failure. These guarantees are not in the published 0.1.0a1 wheel.

Troubleshooting and migration

From upstream, start a clean environment and replace the installed distribution with pytest-jira-xray-plus. Imports, markers and the xray entry point remain compatible. Review the narrower dependency baseline, explicitly enable case reporting and comments as needed, and compare an offline xray.json first. For alpha users, the 0.1.0a2 failure-exit, timeout and retry changes are intentional behavior changes: update CI expectations before upgrading to a future release.

Advanced inherited hooks allow editing the final payload with pytest_xray_results and attaching evidence through pytest_runtest_makereport; see the upstream-style reference. Hooks run trusted local Python and can alter the safety and correctness of a report.

Demonstrate the comment-size policy

For a no-network size-policy demonstration after installing 0.1.0, run the same test_example.py with:

python -m pytest -q test_example.py --jira-xray --xraypath=xray.json \
  --allow-duplicate-ids --xray-parameter-details --xray-comment-limit=256 \
  --xray-full-report=full-xray.json --xray-case-report=cases.json

With the original passing example, expect exit 0, two passing cases in cases.json, one full comment in full-xray.json, and one shorter marked comment in xray.json. The terminal reports one truncated comment. Neither file is sent. The deliberately small 256-byte limit demonstrates the behavior, not a recommended server setting.

Development and compatibility scope

To run the development checks, clone this repository, check out the reviewed commit, create a new environment, and run from the checkout:

python -m pip install -e . -r requirements-tests.txt build==1.6.1 setuptools==84.0.0 wheel==0.48.0 twine==7.0.0
python -m pytest -q
python tools/reproduce.py src --details
python tools/validate_offline.py
python tools/build_release.py release-dist
python -m twine check release-dist/*

Regression tests use synthetic data, blocked transport or a localhost fake server. The readiness record separates executed checks from user-reported validation and unverified claims. Stable scope is the documented local reporter and optional aggregate comment fallback, including its explicit status mappings and limits. It does not certify every deployment, OS or plugin combination. For another server configuration, validate comment behavior, effective Test type and accepted payload limits in an authorized sandbox. Native iteration support is outside this release and requires separate evidence for schema, stable identity, reordered/repeated imports and persisted UI/readback; the experimental probe is a review plan, not an enabled feature. The minimal sandbox check needs only synthetic Tests and an existing browser login; it requests no private credentials or data. No workplace access is needed or requested.

Publishing remains manual through the existing publish.yml / pypi environment. The workflow checks the reviewed main-branch commit and exact artifact hashes, then uses the existing pypi environment. The stable version does not extend support to native iterations or untested server behaviors. Report synthetic reproductions through GitHub issues.