pytest-railflow-testrail-reporter 0.2.5

Generate json reports along with specified metadata defined in test markers.

pytest-railflow-testrail-reporter

pytest-railflow-testrail-reporter is a custom reporter which generates JSON report file containing some TestRail-related data.
The reporter helps you to integrate your pytest test with TestRail easily by producing JSON test report files which can be uploaded into TestRail by using powerful Railflow NPM CLI tool or native plugins for Jenkins and TeamCity.

Requirements

In order to use pytest-railflow-testrail-reporter plugin, following prerequisites should be met.

• Python 2.7, 3.4 or greater
• Pytest

Installation

Using Pip

To install the pytest-railflow-testrail-reporter plugin using pip,run the following command in terminal:

pip install pytest-railflow-testrail-reporter

This will install the plugin to the system.

Usage

To add Railflow markers to the tests '@pytest.mark.railflow' marker is used.

The plugin supports the following attributes:

Railflow configuration attributes description

Attribute Name	Description
title	Name of the test suite or test case
case_type	Case type in TestRail, e.g.: Automated, Compatibility, etc...
case_priority	Case type in TestRail, e.g.: Critical, High, etc...
case_fields	Values for custom case fields in TestRail in the following format: ['field1=value1','field2=value2']
result_fields	Values for result fields in TestRail in the following format: ['field1=value1','field2=value2']
jira_ids	Jira IDs.These values will be populated as a case field 'refs'. Should input as an array of strings, e.g.: ['jid1','jid2']
testrail_ids	IDs of test cases in TestRail. Should input as an array of integers, e.g.: [1,2,3]
smart_failure_assignment	Array of TestRail users to automatically assign failed test cases. Should input as a string array, e.g.: ['[e-mail]','[e-mail]']

These attributes can be either used in class level or function level.

Function level attributes	Class level attributes
jira_ids	case_fields
case_fields	result_fields
result_fields	case_type
testrail_ids	case_priority
case_type	smart_failure_assignment
case_priority	title
title

To run the test, execute the following command in the terminal from the test directory:

pytest --jsonfile output.json

Adding arbitrary attachments into test report

The testrail_add_screenshot fixture can be used to add arbitrary attachments into the test report:

def test_add(testrail_add_screenshot):
    a = 3
    b = 2
    c = a + b
    assert c == 5
    testrail_add_screenshot("screenshot path")

Adding test steps to test report

The testrail_add_test_step fixture can be used to add test steps into test report:

The fixture has the following parameters:

Order	Description	Is Mandatory
1	Step Name	Yes
2	Status Name	Yes
3	Actual Value	No
4	Expected Value	No

def test_multiply(testrail_add_test_step):
    res = 8 * 4
    testrail_add_test_step("multiply 8 and 4", "PASSED", res, "32")

Examples

Example 1 : pytest tests within a class

1. Install 'pytest-railflow-testrail-reporter' .

$ pip install pytest-railflow-testrail-reporter

2. Add pytest test with Railfllow marker on class level.

test_calculation.py

import pytest

@pytest.mark.railflow(
     jira_ids=["301219"],
    case_fields=[
        {
            "name": "Required text field",
            "value": "Class"
        }
    ],
    result_fields=[
        {
            "name": "Custom field",
            "value": "Result from class"
        }
    ],
    case_type="Automated",
    case_priority="Critical",
    smart_failure_assignment=["[e-mail], [e-mail]"]
)
class TestClass:

    def test_add(self):
        a = 3
        b = 2
        c = a + b
        assert c == 5

    def test_subtract(self):
        a = 3
        b = 2
        c = a - b
        assert c == 0

    @pytest.mark.railflow(
        title = "method title",
        jira_ids=["11111"],
        case_fields=[
            {
                "name": "Required text field",
                "value": "method"
            }
        ],
        result_fields=[
            {
                "name": "Custom fieLD",
                "value": "Result from method"
            }
        ],
        case_type="Compatibility",
        case_priority="High",
        smart_failure_assignment=["[e-mail]"]
    )
    @pytest.mark.parametrize("a,b,c", [(22, 11, 2), (64, 8, 8), (9, 3, 3)])
    def test_divide(self, a, b, c):
        assert a / b == c  

3. Run tests and generate report

pytest --jsonfile output.json test_calculation.py

4. View report file

Report file generated at the same directory where the test executed.

output.json

[
    {
        "class_name": "TestClass",
        "markers": "",
        "file_name": "test_sample2",
        "attachments": [],
        "tests": [
            {
                "test_name": "test_add",
                "details": null,
                "markers": "",
                "result": "PASSED",
                "duration": 0.00012004900054307655,
                "timestamp": "2022-03-09T15:53:53",
                "message": null
            },
            {
                "test_name": "test_subtract",
                "details": null,
                "markers": "",
                "result": "FAILED",
                "duration": 0.00017888799993670546,
                "timestamp": "2022-03-09T15:53:53",
                "message": "self = <test_sample2.TestClass object at 0x7f659f8af310>\n\n    def test_subtract(self):\n        a = 3\n        b = 2\n        c = a - b\n>       assert c == 0\nE       assert 1 == 0\n\ntest_sample2.py:33: AssertionError"
            },
            {
                "test_name": "test_divide",
                "details": null,
                "markers": "parametrize",
                "parameters": [
                    {
                        "name": "a",
                        "value": 22
                    },
                    {
                        "name": "b",
                        "value": 11
                    },
                    {
                        "name": "c",
                        "value": 2
                    }
                ],
                "result": "PASSED",
                "duration": 0.0001058529996953439,
                "timestamp": "2022-03-09T15:53:53",
                "message": null,
                "railflow_test_attributes": {
                    "title": "method title",
                    "jira_ids": [
                        "11111"
                    ],
                    "case_fields": [
                        {
                            "name": "Required text field",
                            "value": "method"
                        }
                    ],
                    "result_fields": [
                        {
                            "name": "Custom field",
                            "value": "Result from method"
                        }
                    ],
                    "case_type": "Compatibility",
                    "case_priority": "High",
                    "smart_failure_assignment": [
                        "[e-mail]"
                    ]
                }
            },
            {
                "test_name": "test_divide",
                "details": null,
                "markers": "parametrize",
                "parameters": [
                    {
                        "name": "a",
                        "value": 64
                    },
                    {
                        "name": "b",
                        "value": 8
                    },
                    {
                        "name": "c",
                        "value": 8
                    }
                ],
                "result": "PASSED",
                "duration": 0.00013493100050254725,
                "timestamp": "2022-03-09T15:53:53",
                "message": null,
                "railflow_test_attributes": {
                    "title": "method title",
                    "jira_ids": [
                        "11111"
                    ],
                    "case_fields": [
                        {
                            "name": "Required text field",
                            "value": "method"
                        }
                    ],
                    "result_fields": [
                        {
                            "name": "Custom field",
                            "value": "Result from method"
                        }
                    ],
                    "case_type": "Compatibility",
                    "case_priority": "High",
                    "smart_failure_assignment": [
                        "[e-mail]"
                    ]
                }
            },
            {
                "test_name": "test_divide",
                "details": null,
                "markers": "parametrize",
                "parameters": [
                    {
                        "name": "a",
                        "value": 9
                    },
                    {
                        "name": "b",
                        "value": 3
                    },
                    {
                        "name": "c",
                        "value": 3
                    }
                ],
                "result": "PASSED",
                "duration": 0.00020506200235104188,
                "timestamp": "2022-03-09T15:53:53",
                "message": null,
                "railflow_test_attributes": {
                    "title": "method title",
                    "jira_ids": [
                        "11111"
                    ],
                    "case_fields": [
                        {
                            "name": "Required text field",
                            "value": "method"
                        }
                    ],
                    "result_fields": [
                        {
                            "name": "Custom field",
                            "value": "Result from method"
                        }
                    ],
                    "case_type": "Compatibility",
                    "case_priority": "High",
                    "smart_failure_assignment": [
                        "[e-mail]"
                    ]
                }
            }
        ],
        "railflow_test_attributes": {
            "jira_ids": [
                "301219"
            ],
            "case_fields": [
                {
                    "name": "Required text field",
                    "value": "Class"
                }
            ],
            "result_fields": [
                {
                    "name": "Custom field",
                    "value": "Result from class"
                }
            ],
            "case_type": "Automated",
            "case_priority": "Critical",
            "smart_failure_assignment": [
                "[e-mail], [e-mail]"
            ]
        }
    }
]

5. Install Railflow CLI

npm install railflow

6. Run Railflow CLI and upload test results into TestRail

npx railflow -k ABCDE-12345-FGHIJ-67890 -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path section1/section2 -f pytest-railflow -r pytest_example/*.json -sm path

Where:

Key	Description	Example
-k, --key	(online activation) License key. Can be set with RAILFLOW_LICENSE environment variable	-k XXXXX-XXXXX-XXXXX-XXXXX
-url, --url	TestRail instance URL	-url https://example.testrail.io
-u, --username	TestRail username. Can be set with RAILFLOW_TR_USER environment variable	-u test-username
-p, --password	TestRail password or API Key. Can be set with RAILFLOW_TR_PASSWORD environment variable	-p XtpHXiPLEODyhF
-pr, --project	TestRail project name	-pr "example project"
-path, --test-path	TestRail test cases path	-path "Section1/subsection2/ShoppingCart
-f, --format	Report format: 'pytest-railflow' (case insensitive)	-f junit
-r, --report-files	The file path(s) to the test report file(s) generated during the build. User can pass multiple values separated with spaces. Ant-style patterns such as **/reports/*.xml can be used. E.g. use target/reports/*.xml to capture all XML files in target/reports directory.	-r target/surefire-reports/*.xml target/failsafe-reports/*.xml
-sm, --search-mode	Specifies the test case lookup algorithm. name: search for test case matching the name within the entire test suite. If test case found, update the test case. If test case not found, create a new test case within specified -path path: search for test case matching the name within the specified -path. If test case found, update the test case. If test case not found, create a new test case within specified -path	-sm path

Please see Railflow NPM documentation for the all the details about Railflow CLI options.

7. View results in TestRail

Test run

Test result details

Parameterized tests details

Example 2 : pytest tests without a class (using pytest splinter)

1. Install 'pytest-splinter' .

$ pip install pytest-splinter

Also, make sure that chromedriver or Firefox driver is installed in your system for running splinter.

For Firefox driver: [Splinter Documentation on Firefox Driver](https://splinter.readthedocs.io/en/latest/drivers/firefox.html)

For Chrome driver : [SPlinter Documentation on Chrome Driver](https://splinter.readthedocs.io/en/latest/drivers/chrome.html)

2. Add pytest test.

test_browser.py

import pytest

@pytest.mark.railflow(
    jira_ids=["301219"],
    case_fields=[
        {
            "name": "Required text field",
            "value": "Class"
        }
    ],
    result_fields=[
        {
            "name": "Custom field",
            "value": "Result from class"
        }
    ],
    case_type="Automated",
    case_priority="Critical",
    smart_failure_assignment=["[e-mail], [e-mail]"]
)
def test_google(browser):
    """Test using real browser."""
    url = "https://www.google.com"
    browser.visit(url)
    browser.fill('q', 'splinter - python acceptance testing for web applications')
    # Find and click the 'search' button
    button = browser.find_by_name('btnK')
    # Interact with elements
    button.click()
    assert browser.is_text_present('splinter.cobrateam.info'), "splinter.cobrateam.info wasn't found... We need to"
    ' improve our SEO techniques'

3. Run tests and generate report (using chrome driver) if '--splinter-webdriver' argument is not provided , it will use firefox as default web driver

pytest --splinter-webdriver chrome --jsonfile output.json test_browser.py

4. View report file

output.json

[
    {
        "class_name": null,
        "test_name": "test_google",
        "details": "Test using real browser.",
        "markers": "",
        "result": "FAILED",
        "duration": 8.737954783000077,
        "timestamp": "2022-03-06T10:51:51",
        "message": "browser = <splinter.driver.webdriver.chrome.WebDriver object at 0x7f55c6f50670>\n\n    @pytest.mark.railflow(\n         jira_ids=[\"301219\"],\n        case_fields=[\n            {\n                \"name\": \"ReQuired text field\",\n                \"value\": \"Class\"\n            }\n        ],\n        result_fields=[\n            {\n                \"name\": \"Custom fIeLD\",\n                \"value\": \"Result from class\"\n            }\n        ],\n        case_type=\"Railflow\",\n        case_priority=\"Critical\",\n        smart_failure_assignment=[\"[e-mail], [e-mail]\"]\n    )\n    def test_google(browser):\n        \"\"\"Test using real browser.\"\"\"\n        url = \"https://www.google.com\"\n        browser.visit(url)\n        browser.fill('q', 'splinter - python acceptance testing for web applications')\n        # Find and click the 'search' button\n        button = browser.find_by_name('btnK')\n        # Interact with elements\n        button.click()\n>       assert browser.is_text_present('splinter.cobrateam.info'), \"splinter.cobrateam.info wasn't found... We need to\"\nE       AssertionError: splinter.cobrateam.info wasn't found... We need to\nE       assert False\nE        +  where False = <bound method BaseWebDriver.is_text_present of <splinter.driver.webdriver.chrome.WebDriver object at 0x7f55c6f50670>>('splinter.cobrateam.info')\nE        +    where <bound method BaseWebDriver.is_text_present of <splinter.driver.webdriver.chrome.WebDriver object at 0x7f55c6f50670>> = <splinter.driver.webdriver.chrome.WebDriver object at 0x7f55c6f50670>.is_text_present\n\ntest_browser.py:30: AssertionError",
        "file_name": "test_browser",
        "attachments": [
            " /home/projects/pytest/example_tests/railflow_pytest_examples/railflow_pytest_examples.test_browser/test_google-browser.png"
        ],
        "railflow_test_attributes": {
            "jira_ids": [
                "301219"
            ],
            "case_fields": [
                {
                    "name": "Required text field",
                    "value": "Class"
                }
            ],
            "result_fields": [
                {
                    "name": "Custom field",
                    "value": "Result from class"
                }
            ],
            "case_type": "Railflow",
            "case_priority": "Critical",
            "smart_failure_assignment": [
                "[e-mail], [e-mail]"
            ]
        }
    }
]

5. Run Railflow CLI and upload test results into TestRail

npx railflow -k ABCDE-12345-FGHIJ-67890 -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path section1/section2 -f pytest-railflow -r pytest_example/*.json

Please check examples for more information and sample tests.

License

This software is licensed under the MIT license

See License file for more information.