Skip to main content

pytest plugin that let you play a YAML file describing some actions and assertions.

Project description

pytest-play

See Build Status on Travis CI Documentation Status https://codecov.io/gh/pytest-dev/pytest-play/branch/master/graph/badge.svg

pytest-play is a codeless, generic, pluggable and extensible automation tool, not necessarily test automation only, based on the fantastic pytest test framework that let you define and execute YAML files containing scripts or test scenarios through actions and assertions that can be implemented and managed even by non technical users:

  • automation (not necessarily test automation). You can build a set of actions on a single file (e.g, call a JSON based API endpoint, perform an action if a condition matches) or a test automation project with many test scenarios.

    For example you can create always fresh test data on demand supporting manual testing activities, build a live simulator and so on

  • codeless, or better almost codeless. If you have to write assertions against action results or some conditional expressions you need a very basic knowledge of Python or Javascript expressions with a smooth learning curve (something like variables['foo'] == 'bar')

  • generic. It is not yet again another automation tool for browser automation only, API only, etc. You can drive a browser, perform some API calls, make database queries and/or make assertions using the same tool for different technologies

    So there are several free or not free testing frameworks or automation tools and many times they address just one single area testing needs and they are not extensible: API testing only, UI testing only and so on. It could be fine if you are testing a web only application like a CMS but if you are dealing with a reactive IoT application you might something more, make cross actions or cross checks against different systems or build something of more complex upon pytest-play

  • powerful. It is not yet again another test automation tool, it only extends the pytest framework with another paradigm and inherits a lot of good stuff (test data decoupled by test implementation that let you write once and executed many times the same scenario thanks to native parametrization support, reporting, integration with test management tools, many useful command line options, browsers and remote Selenium grids integration, etc)

  • pluggable and extensible. Let’s say you need to interact with a system not yet supported by a pytest-play plugin, you can write by your own or pay someone for you. In addition there is a scaffolding tool that let you implement your own command: https://github.com/davidemoro/cookiecutter-play-plugin

  • easy to use. Why YAML? Easy to read, easy to write, simple and standard syntax, easy to be validated and no parentheses hell. Despite there are no recording tools (not yet) for browser interaction or API calls, the documentation based on very common patterns let you copy, paste and edit command by command with no pain

  • free software. It’s an open source project based on the large and friendly pytest community

  • easy to install. The only prerequisite is Docker thanks to the davidemoro/pytest-play Docker Hub container. Or better, with docker, no installation is required: you just need to type the following command docker run -i --rm -v $(pwd):/src davidemoro/pytest-play inside your project folder See https://hub.docker.com/r/davidemoro/pytest-play

See at the bottom of the page the third party plugins that extends pytest-play:

How it works

Depending on your needs and skills you can choose to use pytest-play programmatically writing some Python code or following a Python-less approach.

As said before with pytest-play you will be able to create codeless scripts or test scenarios with no or very little Python knowledge: a file test_XXX.yml (e.g., test_something.yml, where test_ and .yml matter) will be automatically recognized and executed without having to touch any *.py module.

You can run a single scenario with pytest test_XXX.yml or running the entire suite filtering by name or keyword markers.

Despite pytest-play was born with native support for JSON format, pytest-play>=2.0 versions will support YAML only for improved usability.

Python-less (pure YAML)

Here you can see the contents of a pytest-play project without any Python files inside containing a login scenario:

$ tree
.
├── env-ALPHA.yml    (OPTIONAL)
└── test_login.yml

and you might have some global variables in a settings file specific for a target environment:

$ cat env-ALPHA.yml
pytest-play:
  base_url: https://www.yoursite.com

The test scenario with action, assertions and optional metadata (play_selenium external plugin needed):

$ cat test_login.yml
---
markers:
  - login
test_data:
  - username: siteadmin
    password: siteadmin
  - username: editor
    password: editor
  - username: reader
    password: reader
---
- comment: visit base url
  type: get
  provider: selenium
  url: "$base_url"
- comment: click on login link
  locator:
    type: id
    value: personaltools-login
  type: clickElement
  provider: selenium
- comment: provide a username
  locator:
    type: id
    value: __ac_name
  text: "$username"
  type: setElementText
  provider: selenium
- comment: provide a password
  locator:
    type: id
    value: __ac_password
  text: "$password"
  type: setElementText
  provider: selenium
- comment: click on login submit button
  locator:
    type: css
    value: ".pattern-modal-buttons > input[name=submit]"
  type: clickElement
  provider: selenium
- comment: wait for page loaded
  locator:
    type: css
    value: ".icon-user"
  type: waitForElementVisible
  provider: selenium

The first optional YAML document contains some metadata with keywords aka markers so you can filter tests to be executed invoking pytest with marker expressions, decoupled test data, etc.

The same test_login.yml scenario will be executed 3 times with different decoupled test data test_data defined inside its first optional YAML document (the block between the 2 --- lines).

So write once and execute many times with different test data!

You can see a hello world example here:

As told before the metadata document is optional so you might have 1 or 2 documents in your YAML file. You can find more info about Metadata format.

Here you can see the same example without the metadata section for sake of completeness:

---
- comment: visit base url
  type: get
  provider: selenium
  url: "http://YOURSITE"
- comment: click on login link
  type: clickElement
  provider: selenium
  locator:
    type: id
    value: personaltools-login
- comment: provide a username
  type: setElementText
  provider: selenium
  locator:
    type: id
    value: __ac_name
  text: "YOURUSERNAME"
- comment: provide a password
  type: setElementText
  provider: selenium
  locator:
    type: id
    value: __ac_password
  text: "YOURPASSWORD"
- comment: click on login submit button
  type: clickElement
  provider: selenium
  locator:
    type: css
    value: ".pattern-modal-buttons > input[name=submit]"
- comment: wait for page loaded
  type: waitForElementVisible
  provider: selenium
  locator:
    type: css
    value: ".icon-user"

Programmatically

You can invoke pytest-play programmatically too.

You can define a test test_login.py like this:

def test_login(play):
    data = play.get_file_contents(
        'my', 'path', 'etc', 'login.yml')
    play.execute_raw(data, extra_variables={})

Or this programmatical approach might be used if you are implementing BDD based tests using pytest-bdd.

Core commands

pytest-play provides some core commands that let you:

  • write simple Python assertions, expressions and variables

  • reuse steps including other test scenario scripts

  • provide a default command template for some particular providers (eg: add by default HTTP authentication headers for all requests)

  • a generic wait until machinery. Useful for waiting for an observable asynchronous event will complete its flow before proceeding with the following commands that depends on the previous step completion

You can write restricted Python expressions and assertions based on the RestrictedPython package.

RestrictedPython is a tool that helps to define a subset of the Python language which allows to provide a program input into a trusted environment. RestrictedPython is not a sandbox system or a secured environment, but it helps to define a trusted environment and execute untrusted code inside of it.

See:

How to reuse steps

You can split your commands and reuse them using the include command avoiding duplication:

- provider: include
  type: include
  path: "/some-path/included-scenario.yml"

You can create a variable for the base folder where your test scripts live.

Default commands

Some commands require many verbose options you don’t want to repeat (eg: authentication headers for play_requests).

Instead of replicating all the headers information you can initialize a pytest-play with the provider name as key and as a value the default command you want to omit (this example neets the external plugin play_selenium):

- provider: python
  type: store_variable
  name: bearer
  expression: "'BEARER'"
- provider: python
  type: store_variable
  name: play_requests
  expression: "{'parameters': {'headers': {'Authorization': '$bearer'}}}"
- provider: play_requests
  type: GET
  comment: this is an authenticated request!
  url: "$base_url"

Store variables

You can store a pytest-play variables:

- provider: python
  type: store_variable
  expression: "1+1"
  name: foo

Make a Python assertion

You can make an assertion based on a Python expression:

- provider: python
  type: assert
  expression: variables['foo'] == 2

Sleep

Sleep for a given amount of seconds:

- provider: python
  type: sleep
  seconds: 2

Exec a Python expresssion

You can execute a Python expression:

- provider: python
  type: exec
  expression: "1+1"

Wait until condition

The wait_until_not command waits until the wait expression is False (this example contains a SQL query so the external plugin called play_sql is needed plus the appropriate SQL driver depending on database type):

- provider: python
  type: wait_until_not
  expression: variables['expected_id'] is not None and variables['expected_id'][0] == $id
  timeout: 5
  poll: 0.1
  subcommands:
  - provider: play_sql
    type: sql
    database_url: postgresql://$db_user:$db_pwd@$db_host/$db_name
    query: SELECT id FROM table WHERE id=$id ORDER BY id DESC;
    variable: expected_id
    expression: results.first()

assuming that the subcommand updates the execution results updating a pytest-play variable (eg: expected_id) where tipically the $id value comes from a previously executed command that causes an asynchrounous update on a relational database soon or later (eg: a play_requests command making a HTTP POST call or a MQTT message coming from a simulated IoT device with play_mqtt).

The wait command will try (and retry) to execute the subcommand with a poll frequency poll (default: 0.1 seconds) until the provided timeout expressed in seconds expires or an exception occurs.

You can use the opposite command named wait_until that waits until the wait expression is not False.

Loop commands

You can repeat a group of subcommands using a variable as a counter. Assuming you have defined a countdown variable with 10 value, the wait until command will repeat the group of commands for 10 times:

play.execute_command({
    'provider': 'python',
    'type': 'wait_until',
    'expression': 'variables["countdown"] == 0',
    'timeout': 0,
    'poll': 0,
    'sub_commands': [{
        'provider': 'python',
        'type': 'store_variable',
        'name': 'countdown',
        'expression': 'variables["countdown"] - 1'
    }]
})

or:

- provider: python
  type: wait_until
  expression: variables['countdown'] == 0
  timeout: 0
  poll: 0
  sub_commands:
  - provider: python
    type: store_variable
    name: countdown
    expression: variables['countdown'] - 1

Conditional commands (Python)

You can skip any command evaluating a Python based skip condition like the following:

- provider: include
  type: include
  path: "/some-path/assertions.yml"
  skip_condition: variables['cassandra_assertions'] is True

Browser based commands

The pytest-play core no more includes browser based commands. Moved to play_selenium external plugin.

pytest-play is pluggable and extensible

pytest-play has a pluggable architecture and you can extend it.

For example you might want to support your own commands, support non UI commands like making raw POST/GET/etc calls, simulate IoT devices activities, provide easy interaction with complex UI widgets like calendar widgets, send commands to a device using the serial port implementing a binary protocol and so on.

How to register a new command provider

Let’s suppose you want to extend pytest-play with the following command:

command = {'type': 'print', 'provider': 'newprovider', 'message': 'Hello, World!'}

You just have to implement a command provider:

from pytest_play.providers import BaseProvider

class NewProvider(BaseProvider):

    def this_is_not_a_command(self):
        """ Commands should be command_ prefixed """

    def command_print(self, command):
        print(command['message'])

    def command_yetAnotherCommand(self, command):
        print(command)

and register your new provider in your setup.py adding an entrypoint:

entry_points={
    'playcommands': [
        'print = your_package.providers:NewProvider',
    ],
},

You can define new providers also for non UI commands. For example publish MQTT messages simulating IoT device activities for integration tests.

If you want you can generate a new command provider thanks to:

Metadata format

You can also add some scenario metadata placing another YAML document on top of the scenario defined on the test_XXX.yml with the following format:

---
markers:
  - marker1
  - marker2
test_data:
  - username: foo
  - username: bar
---
# omitted scenario steps in this example...

Option details:

  • markers, you can decorate your scenario with one or more markers. You can use them in pytest command line for filtering scenarios to be executed thanks to marker expressions like -m "marker1 and not slow"

  • test_data, enables parametrization of your decoupletd test data and let you execute the same scenario many times. For example the example above will be executed twice (one time with “foo” username and another time with “bar”)

New options will be added in the next feature (e.g., skip scenarios, xfail, xpass, etc).

Articles and talks

Articles:

Talks:

Third party pytest-play plugins

  • play_selenium, pytest-play plugin driving browsers using Selenium/Splinter under the hood. Selenium grid compatible and implicit auto wait actions for more robust scenarios with less controls.

  • play_requests, pytest-play plugin driving the famous Python requests library for making HTTP calls.

  • play_sql, pytest-play support for SQL expressions and assertions

  • play_cassandra, pytest-play support for Cassandra expressions and assertions

  • play_dynamodb, pytest-play support for AWS DynamoDB queries and assertions

  • play_websocket, pytest-play support for websockets

  • play_mqtt, pytest-play plugin for MQTT support. Thanks to play_mqtt you can test the integration between a mocked IoT device that sends commands on MQTT and a reactive web application with UI checks.

    You can also build a simulator that generates messages for you.

Feel free to add your own public plugins with a pull request!

Twitter

pytest-play tweets happens here:

Changelog

2.0.2 (2019-02-06)

Documentation:

  • more examples

  • fix documentation bug on README (example based on selenium with missing provider: selenium)

2.0.1 (2019-01-30)

Documentation:

  • Mention davidemoro/pytest-play docker container in README. You can use pytest-play with a docker command like that now docker run -i --rm -v $(pwd):/src davidemoro/pytest-play

Bugfix:

  • Fix error locking pipenv due to pytest-play requirement constraint not existing (RestrictedPython>=4.0.b2 -> RestrictedPython>=4.0b2)

2.0.0 (2019-01-25)

Breaking changes:

  • Renamed fixture from play_json to play (#5)

  • Drop json support, adopt yaml only format for scenarios (#5)

  • Drop .ini file for metadata, if you need them you can add a YAML document on top of the scenario .yml file. You no more need multiple files for decorating your scenarios now (#65)

  • play.execute no more accepts raw data string), consumes a list of commands. Introduced play.execute_raw accepting raw data string.

  • play.execute_command accepts a Python dictionary only now (not a string)

  • Selenium provider removed from pytest-play core, implemented on a separate package play_selenium. Starting from now you have to add to your selenium commands provider: selenium

  • engine’s parametrizer_class attribute no more available ( use parametrizer.Parametrizer by default now)

Bug fix:

  • Fix invalid markup on PyPI (#55)

  • Fix invalid escape sequences (#62).

Documentation and trivial changes:

  • Add examples folder

1.4.2 (2018-05-17)

  • Configuration change on Github. Use the same branching policy adopted by pytest (master becomes main branch, see #56)

  • Fixed skipped test and added new tests (deselect scenarios with keyword and marker expressions)

  • Fix #58: you no more get a TypeError if you try to launch pytest-play in autodiscovery mode

  • Fix #55: restructured text lint on README.rst (bad visualization on pypi)

  • Updated README (articles and talks links)

  • Added a DeprecationWarning for play_json fixture. pytest-play will be based on yaml instead of json in version >=2.0.0. See https://github.com/pytest-dev/pytest-play/issues/5

1.4.1 (2018-04-06)

  • Documentation improvements

  • Add bzt/Taurus/BlazeMeter compatibility

1.4.0 (2018-04-05)

  • Small documentation improvements

  • Now test_XXX.json files are automatically collected and executed

  • You can run a test scenario using the pytest CLI pytest test_YYY.json

  • Introduced json test scenario ini file with markers definition. For a given test_YYY.json scenario you can add a test_YYY.ini ini file:

    [pytest]
    markers =
        marker1
        marker2

    and filter scenarios using marker expressions pytest -m marker1

  • Enabled parametrization of arguments for a plain json scenario in scenario ini file:

    [pytest]
    test_data =
       {"username": "foo"}
       {"username": "bar"}

    and your json scenario will be executed twice

  • pytest-play loads some variables based on the contents of the optional pytest-play section in your pytest-variables file now. So if your variables file contains the following values:

    pytest-play:
      foo: bar
      date_format: YYYYMMDD

    you will be able to use expressions $foo, $date_format, variables['foo'] or variables['date_format']

1.3.2 (2018-02-05)

  • Add sorted in python expressions

1.3.1 (2018-01-31)

  • Add more tests

  • Documentation update

  • play_json fixture no more assumes that you have some pytest-variables settings. No more mandatory

  • fix include scenario bug that occurs only on Windows (slash vs backslash and JSON decoding issues)

1.3.0 (2018-01-22)

  • documentation improvements

  • supports teardown callbacks

1.2.0 (2018-01-22)

  • implement python based commands in pytest-play and deprecates play_python. So this feature is a drop-in replacement for the play-python plugin.

    You should no more install play_python since now.

  • update documentation

  • deprecate selenium commands (they will be implemented on a separate plugin and dropped in pytest-play >= 2.0.0). All your previous scripts will work fine, this warning is just for people directly importing the provider for some reason.

  • implement skip conditions. You can omit the execution of any command evaluating a Python based skip condition

1.1.0 (2018-01-16)

  • Documentation updated (add new pytest play plugins)

  • Support default payloads for command providers. Useful for HTTP authentication headers, common database settings

1.0.0 (2018-01-10)

  • execute command accepts kwargs now

  • execute command returns the command value now

  • complete refactor of include provider (no backwards compatibility)

  • add play_json.get_file_contents and removed data_getter fixture (no backwards compatibility)

0.3.1 (2018-01-04)

  • play engine now logs commands to be executed and errors

0.3.0 (2018-01-04)

  • you are able to update variables when executing commands

  • you can extend pytest-play with new pluggable commands coming from third party packages thanks to setuptools entrypoints

0.2.0 (2018-01-02)

  • no more open browser by default pytest-play is a generic test engine and it could be used for non UI tests too.

    So there is no need to open the browser for non UI tests (eg: API tests)

0.1.0 (2017-12-22)

  • implement reusable steps (include scenario)

  • minor documentation changes

0.0.1 (2017-12-20)

  • First release

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pytest-play-2.0.2.tar.gz (29.8 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page