jira-test-reporting 1.4.4

A utility to report pytest results to Jira and Slack

Description

This repository contains utility scripts for

• reporting automated test results Jira
• sending a slack notification with proper stats and jira url that shows failed tests in the current test run id (trid) specifically designed for use in CI/CD pipelines such as Bitbucket Pipelines.

Prerequisites

• Python: Version 3.12 or higher.
• Json Report: Calling project should generate json report. How to create pytest json report
• Jira Access: A Jira instance with API token authentication. How to create jira api token
• Slack Webhook: A Slack webhook URL for sending notifications. How to create slack incoming webhook
• Configuration File: A _env_configs/third_party.conf file with Jira and Slack settings.

Installation

pip install jira-test-reporting

Jira project preperation

Create new Jira project and configure issue type "Task" with following fields

The script uses the following custom fields in Jira tasks:

• Test Environment : Field Type - Dropdown. Important - Pre-populate the values
• Test Area : Field Type - Dropdown - Important - Pre-populate the values
• Test Type : Field Type - Labels
• Test Run : Field Type - Short Text
• Test Tags : Field Type - Labels
• Test Status : Field Type - Dropdown Important - Pre-populate the values
• TRID : Field Type - Short Text

Important Instructions

• In the pytest json report check block "nodeid": "api_tests/Test_Pilot/test_jira_reporting_scenarios.py::Test_JIRA_Reporting_Scenarios::test_jira_reporting_test_passed", -- api_tests should be pre-populated under the Test Type field options -- Test_Pilot should be pre-populated under the Test Area field options
• Similarly, in the pytest json report check block "outcome": "passed" -- Passed should be pre-populated under the Test Status field options. For this field, the values should be pre-populate in the title case.
• Also, make sure that in your jira project, the issue type "Task" has default fields Description and Status
• In the caller projetct, create _env_configs/third_party.conf file with the following structure:

  [DEFAULT]
  jira_field_id_test_env = customfield_10208
  jira_field_id_test_area = customfield_10236
  jira_field_id_test_type = customfield_10301
  jira_field_id_test_run_name = customfield_10205
  jira_field_id_test_tags = customfield_10202
  jira_field_id_test_status = customfield_10235
  jira_field_id_test_run_id = customfield_10269
  scm_url_variable = BITBUCKET_GIT_HTTP_ORIGIN
  scm_build_number_variable = BITBUCKET_BUILD_NUMBER
  

The script also uses the following default fields in Jira tasks:

• project - reflects jira_project_key
• summary - test_name
• description - failure or passing description
• status - reflects test_status as in jira_field_id_test_status

  The values for the fields above will be fetched directly from the json_report
  New jira tasks will be created for non-existing tests
  Existing tests will be updated

Examples

Jira Reported Tests Example

Slack Notification Example

API Test Results
──────────────
🚀 *Test Run:* Release-X
🌎 *Environment:* Staging
❌ *Failed:* 4
──────────────
🧪 *Total Tests:* 148
✅ *Passed:* 143
🔄 *Executed:* 147
⏸️ *Skipped:* 1
📈 Click to open Test Report in Jira
📡 FYA: @User1 @User2
Execution Date: May-23-2025

Usage

Standalone

1. Ensure parameters in _env_configs/third_party.conf has valid values
2. Export environment variables as follows

export jira_host_url=https://my-jira-team.atlassian.net
export jira_username=[e-mail]
export jira_password=XXXXXXXXXXXXXXXXXXXXX
export jira_project_key=TQER
export slack_dev_channel_webhook=https://hooks.slack.com/services/AAAAAA/BBBBBBB/CCCCCCCCC
export slack_prod_channel_webhook=https://hooks.slack.com/services/AAAAAA/BBBBBBB/CCCCCCCCC
export slack_test_webhook=https://hooks.slack.com/services/AAAAAA/BBBBBBB/CCCCCCCCC

3. Run the script with command-line arguments to process a pytest report:

python -m jira_test_reporting.test_results_processor --test-env=Dev --test-run=Release-X --report=test-reports/pytest_report.json --notify-slack=yes

CI-CD hooked example (this copies required files into your test_automation directory)

Assuming you have

• set Repository Variables (as in step #2 mentioned in the standalone setup above) in your scm tool. (How to: bitbcket, github)
• configured pipeline in your SCM tool or a shell script in the caller project to execute the tests.

#!/bin/bash
# -----------------------------------------------------------------------------------------
# # Test Execution
# -----------------------------------------------------------------------------------------
.. pip install -r requirements.txt > /dev/null 2>&1
.. test execution code here
.. pytest -s --tb=no --no-header api_tests --testenv="$TEST_ENV" --json-report -v --json-report-indent=4 --json-report-omit collectors setup teardown --json-report-file=./test-reports/pytest_report.json
# JUST ADD FOLLOWING CODE BLOCK to report the issues
# -----------------------------------------------------------------------------------------
# Report test results to Jira
# -----------------------------------------------------------------------------------------
echo "Reporting test results into Jira and notifying slack"
if [ -n "$TEST_RUN_NAME" ]; then
    python -m jira_test_reporting.test_results_processor --test-env="$TEST_ENV" --test-run="$TEST_RUN_NAME"
else
    python -m jira_test_reporting.test_results_processor --test-env="$TEST_ENV"
fi

Arguments

• --test-env: Test environment (default: Dev). Examples: --test-env=dev, --test-env=stage.
• --test-run: Test run identifier (default: Daily Run). Examples: --test-run=Release-X, --test-run="Regression Tests".
• --report: Test report file path (default: test-reports/pytest_report.json). Examples: --report=my-test-reports/my-pytest_report.json
• --notify-slack: Whether or not you want to send out a notification to slack (default: yes). Examples: --notify-slack=yes, --notify-slack=no
• --comments-cleanup: whether or not you want to clean up the comments on the tests if it has grown a big pile. Examples: --comments-cleanup=yes, --comments-cleanup=no

Troubleshooting

• Jira Connection Errors:
  • Verify jira_host_url, jira_username, and jira_password in _env_configs/third_party.conf.
  • Ensure the API token is valid and has “Create Issues” and “Edit Issues” permissions.
• Slack Notification Failure:
  • Check the webhook URL in the config file.
  • Ensure the Slack app is configured to allow incoming webhooks.
• Pytest Report Issues:
  • Confirm test-reports/pytest_report.json exists and contains valid JSON.
• Custom Field Errors:
  • Validate field IDs and allowed values in Jira Admin > Issues > Custom Fields.

Contributing

Please read CONTRIBUTE.md

License

This project is licensed under the MIT License. See LICENSE for details.