pytest_robotframework 4.1.2

a pytest plugin that can run both python and robotframework tests while generating robot reports for them

pytest-robotframework

pytest-robotframework is a pytest plugin that creates robotframework reports for tests written in python and allows you to run robotframework tests with pytest.

install

pytest should automatically find and activate the plugin once you install it.

API documentation

click here

features

write robot tests in python

# you can use both robot and pytest features
from robot.api import logger
from pytest import Cache

from pytest_robotframework import keyword

@keyword  # make this function show as a keyword in the robot log
def foo():
    ...

@mark.slow  # markers get converted to robot tags
def test_foo():
    foo()

run .robot tests

to allow for gradual adoption, the plugin also runs regular robot tests as well:

*** Settings ***
test setup  foo

*** Test Cases ***
bar
    [Tags]  asdf  key:value
    no operation

*** Keywords ***
foo
    log  ran setup

which is roughly equivalent to the following python code:

# test_foo.py
from pytest import mark

@keyword
def foo():
    logger.info("ran setup")

@fixture(autouse=True)
def setup():
    foo()

@mark.asdf
@mark.key("value")
def test_bar():
    ...

setup/teardown

in pytest, setups and teardowns are defined using fixtures:

from pytest import fixture
from robot.api import logger

@fixture
def user():
    logger.info("logging in")
    user = ...
    yield user
    logger.info("logging off")

def test_something(user):
    ...

under the hood, pytest calls the fixture setup/teardown code as part of the pytest_runtest_setup and and pytest_runtest_teardown hooks, which appear in the robot log like so:

for more information, see the pytest documentation for fixtures and hook functions.

tags/markers

pytest markers are converted to tags in the robot log:

from pytest import mark

@mark.slow
def test_blazingly_fast_sorting_algorithm():
    [1,2,3].sort()

markers like skip, skipif and parameterize also work how you'd expect:

from pytest import mark

@mark.parametrize("test_input,expected", [(1, 8), (6, 6)])
def test_eval(test_input: int, expected: int):
    assert test_input == expected

robot suite variables

to set suite-level robot variables, call the set_variables function at the top of the test suite:

from robot.libraries.BuiltIn import BuiltIn
from pytest_robotframework import set_variables

set_variables(
    {
        "foo": "bar",
        "baz": ["a", "b"],
    }
)

def test_variables():
    assert BuiltIn().get_variable_value("$foo") == "bar"

set_variables is equivalent to the *** Variables *** section in a .robot file. all variables are prefixed with $. @ and & are not required since $ variables can store lists and dicts anyway

running tests in parallel

running tests in parallel using pytest-xdist is supported. when running with xdist, pytest-robotframework will run separate instances of robot for each test, then merge the robot output files together automatically using rebot.

config

since this is a pytest plugin, you should avoid using robot options that have pytest equivalents:

instead of...	use...	notes
robot --include tag_name	pytest -m tag_name
robot --exclude tag_name	pytest -m not tag_name
robot --skip tag_name	pytest -m "not tag_name"
robot --test "test name" ./test.robot	pytest ./test.robot::"Test Name"
robot --suite "suite name" ./folder	pytest ./folder
robot --dryrun	pytest --collect-only	not exactly the same. you should use a type checker on your python tests as a replacement for robot dryrun
robot --exitonfailure	pytest --maxfail=1
robot --rerunfailed	pytest --lf
robot --runemptysuite	pytest --suppress-no-test-exit-code	requires the pytest-custom-exit-code plugin
robot --help	pytest --help	all supported robot options will be listed in the robotframework section

specifying robot options directlty

there are multiple ways you can specify the robot arguments directly. however, arguments that have pytest equivalents cannot be set with robot as they would cause the plugin to behave incorrectly.

pytest cli arguments

most robot cli arguments can be passed to pytest by prefixing the argument names with --robot-. for example, here's how to change the log level:

before

robot --loglevel DEBUG:INFO foo.robot

after

pytest --robot-loglevel DEBUG:INFO test_foo.py

you can see a complete list of the available arguments using the pytest --help command. any robot arguments not present in that list are not supported because they are replaced by a pytest equivalent (see above).

pytest_robot_modify_options hook

you can specify a pytest_robot_modify_options hook in your conftest.py to programmatically modify the arguments. see the pytest_robotframework.hooks documentation for more information.

from pytest_robotframework import RobotOptions
from robot.api.interfaces import ListenerV3

class Foo(ListenerV3):
    ...

def pytest_robot_modify_options(options: RobotOptions, session: Session) -> None:
    if not session.config.option.collectonly:
        options["loglevel"] = "DEBUG:INFO"
        options["listener"].append(Foo()) # you can specify instances as listeners, prerebotmodifiers, etc.

note that not all arguments that the plugin passes to robot will be present in the args list. arguments required for the plugin to function (eg. the plugin's listeners and prerunmodifiers) cannot be viewed or modified with this hook

ROBOT_OPTIONS environment variable

ROBOT_OPTIONS="-d results --listener foo.Foo"

enabling pytest assertions in the robot log

by default, only failed assertions will appear in the log. to make passed assertions show up, you'll have to add enable_assertion_pass_hook = true to your pytest ini options:

# pyproject.toml
[tool.pytest.ini_options]
enable_assertion_pass_hook = true

hiding non-user facing assertions

you may have existing assert statements in your codebase that are not intended to be part of your tests (eg. for narrowing types/validating input data) and don't want them to show up in the robot log. there are two ways you can can hide individual assert statements from the log:

from pytest_robotframework import AssertOptions, hide_asserts_from_robot_log

def test_foo():
    # hide a single passing `assert` statement:
    assert foo == bar, AssertOptions(log_pass=False)

    # hide a group of passing `assert` statements:
    with hide_asserts_from_robot_log():
        assert foo == bar
        assert bar == baz

note that failing assert statements will still show in the log regardless.

you can also run pytest with the --no-assertions-in-robot-log argument to disable assert statements in the robot log by default, then use AssertOptions to explicitly enable individual assert statements:

from pytest_robotframework import AssertOptions

def test_foo():
    assert "foo" == "bar" # hidden from the robot log (when run with --no-assertions-in-robot-log)
    assert "bar" == "baz", AssertOptions(log_pass=True) # not hidden

customizing assertions

pytest-robotframework allows you to customize the message for the assert keyword which appears on both passing and failing assertions:

assert 1 == 1  # no custom description
assert 1 == 1, AssertOptions(description="custom description")

you can still pass a custom message to be displayed only when your assertion fails:

assert 1 == 2, "the values did not match"

however if you want to specify both a custom description and a failure message, you can use the fail_message argument:

assert 1 == 2, "failure message"
assert 1 == 2, AssertOptions(description="checking values", fail_message="failure message")

note that enable_assertion_pass_hook pytest option needs to be enabled for this to work.

limitations with tests written in python

there are some limitations when writing robotframework tests in python. pytest-robotframework includes solutions for these issues.

making keywords show in the robot log

by default when writing tests in python, the only keywords that you'll see in the robot log are Setup, Run Test and Teardown. this is because robot is not capable of recognizing keywords called outside of robot code. (see this issue)

this plugin has several workarounds for the problem:

@keyword decorator

if you want a function you wrote to show up as a keyword in the log, decorate it with the pytest_robotframework.keyword instead of robot.api.deco.keyword

from pytest_robotframework import keyword

@keyword
def foo():
    ...

pytest functions are patched by the plugin

most of the pytest functions are patched so that they show as keywords in the robot log

def test_foo():
    with pytest.raises(ZeroDivisionError):
        logger.info(1 / 0)

patching third party functions with keywordify

if you want a function from a third party module/robot library to be displayed as a keyword, you can patch it with the keywordify function:

# in your conftest.py

from pyest_robotframework import keywordify
import some_module

# patch a function from the module:
keywordify(some_module, "some_function")
# works on classes too:
keywordify(some_module.SomeClass, "some_method")

continuable failures don't work

keywords that raise ContinuableFailure don't work properly when called from python code. this includes builtin keywords such as Run Keyword And Continue On Failure.

use pytest.raises for expected failures instead:

from pytest import raises

with raises(SomeException):
    some_keyword_that_fails()

or if the exception is conditionally raised, use a try/except statement like you would in regular python code:

try:
    some_keyword_that_fails()
except SomeException:
    ... # ignore the exception, or re-raise it later

the keyword will still show as failed in the log (as long as it's decorated with pytest_robotframework.keyword), but it won't effect the status of the test unless the exception is re-raised.

why?

robotframework introduced TRY/EXCEPT statements in version 5.0, which they now recommend using instead of the old Run Keyword And Ignore Error/Run Keyword And Expect Error keywords.

however TRY/EXCEPT behaves differently to its python equivalent, as it allows for errors that do not actually raise an exception to be caught:

*** Test Cases ***
Foo
    TRY
        Run Keyword And Continue On Failure    Fail
        Log    this is executed
    EXCEPT
        Log    and so is this
    END

this means that if control flows like Run Keyword And Continue On Failure were supported, its failures would be impossible to catch:

from robot.api.logger import info
from robot.libraries.BuiltIn import BuiltIn

try:
    BuiltIn().run_keyword_and_continue_on_failure("fail")
    info("this is executed because an exception was not actually raised")
except:
    info("this is NOT executed, but the test will still fail")

IDE integration

vscode

vscode's builtin python plugin should discover both your python and robot tests by default, and show run buttons next to them:

if you use the robotframework-lsp extension, you'll see duplicated tests on .robot files because you have two extensions that can discover them. to fix this, set robot.testView.enabled to false in vscode's settings.

(note: at the time of writing, this option is not yet in the latest version of the extension, so for now you'll need to install this build)

pycharm

pycharm currently does not support pytest plugins for non-python files. see this issue

compatibility

dependency	version range	comments
python	>=3.8,<4.0	all versions of python will be supported until their end-of-life as described here
robotframework	>=6.1,<8.0	i will try to support at least the two most recent major versions. robot 6.0 is not supported as the parser API that the plugin relies on to support tests written in python was introduced in version 6.1
pytest	>=7.0,<9.0	may work on other versions, but things may break since this plugin relies on some internal pytest modules