A testing framework where tests are defined in YAML files
Project description
YAML Testing Framework
A simple, low-code framework for unit testing in Python with tests are defined in YAML files.
Features
Requirements
Python 3.7+
Installation
pip install yaml-testing-framework
Example
Create the files
test_entrypoint.py
- uses pytest to collect and run tests
from types import SimpleNamespace as sns
import pytest
@pytest.mark.parametrize(argnames='test', argvalues=pytest.yaml_tests)
def test_(test: sns) -> None:
assert test.expected == test.output
checks.py
- contains logic for verifying the output from a function
from typing import Any
def check_equal(
expected: Any,
output: Any,
) -> dict:
passed = expected == output
return dict(
passed=passed,
output=output,
expected=expected, )
add.py
- contains the function to test
def main(a, b):
return a + b
add_test.yaml
- contains tests for the function
resources:
- &CHECKS
resource: ./checks.py
tests:
- function: main
description: Returns the result of adding two integers
tests:
- description: Add one negative and one positive integer
arguments:
a: 1
b: -1
checks:
- method: equals
<< : *CHECKS
expected: 0
- description: Add two positive integers
arguments:
a: 1
b: 2
checks:
- method: equals
<< : *CHECKS
expected: 3
- description: Add integer and string
arguments:
a: 1
b: '1'
checks:
- method: equals
<< : *CHECKS
field: __class__.__name__
expected: TypeError
Execute the following command in your command line to run the tests.
pytest --project-path=./add.py
Configuration
The app can be configured within the pytest settings of a configuration file,
such as a pytest.ini
, or console command when invoking pytest. The
configurations are
Field | Type | Description | Default |
---|---|---|---|
project-path | str | Location of a directory containing files or an an individual module or YAML file to test. | . |
exclude-files | str or list | Patterns found in the paths of files to exclude | [] |
exclude-methods | str or list | Patterns found in the names of methods to exclude | [] |
include-files | str or list | Patterns found in the paths of files to include | [] |
include-methods | str or list | Patterns found in the names of methods to include | [] |
yaml-suffix | str | Suffix in the names of YAML files containing tests | _test |
setup | str | Defines resources to setup or teardown during tests | '' |
Configuration file
# ./pytest.ini
[pytest]
project-path = .
exclude_files =
exclude
patterns
exclude_methods =
exclude
patterns
include_files =
include
patterns
include_methods =
include
patterns
yaml_suffix = _test
setup = '''
- name: api
phase: global
method: run_server
arguments:
host: localhost
port: 1234
resource: ./api.py
timeout: -1
'''
Console command
pytest --project-path=. --exclude_files exclude patterns --exclude_methods exclude patterns --include_files include patterns --include_method include patterns --yaml-suffix _test --setup '- name: api\nphase: global\nmethod: run_server\narguments:\n host: localhost\n port: 1234\nresource: ./api.py\ntimeout: -1\n'
YAML Test Files
Tests are defined in YAML files with the top level keys picked up by the app being:
configurations
- Configurations to be used locally for each test in the YAML filestests
- Configurations used for multiple of individual tests.
Expanding Tests
Using the app we can define configurations for tests at various levels (configurations, tests, nested tests), expand those configurations out to individual tests. This allows us to reuse configurations and reduce the duplication of content across a YAML file; similar to anchors in YAML, which we can take advantage, along with the other features available in YAML.
Example
This is an abstract example of the expanding configurations done by the app, where the configurations for tests are comprised of:
config_a
- a listconfig_b
- an objectconfig_c
- a stringconfig_d
- null
In this example, we set these configurations at various levels, globally, tests, and nested tests; and the expanded results are three individual tests containing various values for each configuration.
config_a:
- A
config_b:
b: B
config_c: C
tests:
- config_a:
- B
- config_b:
c: C
tests:
- config_a:
- C
config_c: C0
- config_d: D
tests:
- config_a:
- B
config_b:
b: B0
tests: # Expanded
- config_a: # test 1
- A
- B # Appended item
config_b:
b: B
config_c: C
config_d: null # Standard test config not defined
- config_a: # test 2
- A
- C # Appended item
config_b:
b: B
c: C # Added key/value
config_c: C0 # Replace string
config_d: null
- config_a: # test 3
- A
config_b:
b: B0 # Updated key/value pair
c: C
config_c: C
config_d: D # Standard test config defined
Levels (Phases)
The testing process can be broken up into following four ordered phases:
- global
- module
- function
- check
These phases are synonymous with the levels configurations are be defined at. Configurations described at higher levels are expanded to lower levels, and objects created at higher phases are available at lower phases.
Schema
Details for configurations or fields of an actual test are defined below. These fields can be defined globally or at different test levels.
Field | Type | Description | Expand Action |
---|---|---|---|
function | str | Name of function to test | replace |
environment | dict | Environment variables used by functions in a module | update |
description | str or list | Additional details about the module, function, or test | append |
patches | dict or list | Objects in a module to patch for tests | append |
cast_arguments | dict or list | Convert function arguments to other data types | append |
cast_output | dict or list | Convert function output to other data types | append |
checks | dict or list | Verifies the output of functions | append |
spies | list or str | List of methods to spy on | append |
setup | list or str | Defines objects to setup or teardown during test phases | append |
tests | dict or list | Nested configurations that get expanded into individual tests | append |
Checks
Methods
We can define "check" methods or reusable functions used to compare expected and actual output from a function being tested. These methods can have any of the fields in the table below as parameters.
Parameter | Type | Description |
---|---|---|
output | Any | Output from a method to check |
expected | Any | Value to verify the output against |
setup_ | dict | Access objects created or setup to facilitate testing |
spies_ | dict | Access spies placed on methods |
module | ModuleType | Module of the method being tested |
Check methods should return a dictionary (or object with dict attribute)
containing the fields expected
, output
, and passed
; where passed
is a
boolean indicating whether the check passed of failed.
Example
Here we define a method that verifies the type of output. If the type matches the expected value the check passes.
./checks.py
from typing import Any
def check_type(
output: Any,
expected: str,
) -> dict:
passed = expected == type(output).__name__
return dict(
output=output,
expected=expected,
passed=passed, )
Schema
Checks are defined in YAML test files under the key checks
, and a
single check has the following fields:
Field | Type | Description | Default |
---|---|---|---|
method | str | Function or method used to verify the result of test | pass_through |
expected | Any | The expected output of the function | null |
field | str | Sets the output to a dot-delimited route to an attribute or key within the output. | null |
cast_output | dict or list | Converts output or an attribute or key in the output before processing an check method | null |
resource | str | Location of a module containing a resource to use during testing | '' |
And single test can have multiple checks
checks:
- method: check_type
expected: str
field: null
resource: ./checks.py
cast_output:
- method: __builtins__.str
resource: ./resource.py
Cast arguments and output
We can convert arguments passed to functions and output from functions to other data types. To do this we define cast objects and list them under the keys cast_arguments
and cast_output
for tests or cast_output
for checks.
Schema
The following fields make up a cast object:
Field | Description | Default |
---|---|---|
method | Dot-delimited route to a function or object to cast a value to | null |
field | Dot-delimited route to a field, attribute, or key of an object. When set the specified field of the object is cast | null |
unpack | Boolean indicating whether to unpack an object when casting | False |
resource | Location of a module containing a resource to use during testing | '' |
cast_arguments:
- method: __builtins__.str # Cast arguments as a string
field: null
unpack: false
resource: ./resource.py
- method: SimpleNamespace # Cast an output field as a SimpleNamespace
field: field.key
unpack: true
resource: ./resource.py
cast_output:
- method: do_nothing # Call a casting method
field: null
unpack: false
resource: ./resource.py
Patches
We can patch objects in the module to test before running tests, and since tests are run in individual threads we can different patches for the same object without interference between tests.
Methods
There are four patch methods:
value
- A value to return when the patched object is used.callable
- A value to return when the patched object is called as function.side_effect_list
- A list of values to call based off of the number of times the object is called. Returns the item at indexn - 1
of the list for thenth
call of the object. Reverts to index 0 when number of calls exceeds the length of the list.side_effect_dict
- A dictionary of key, values for to patch an object with. When the patched object is called with a key, the key's associated value is returned
Schema
Patches are defined at a list of objects in YAML test files under the key
patches
, and a single patch object has the following fields:
Field | Type | Description | Default |
---|---|---|---|
method | str | One of the four patch methods defined above | null |
value | Any | The value the patched object should return when called or used | null |
route | str | The dot-delimited route to the object we wish to patch, in the module to test | null |
callable_route | str | Dot-delimited route to a function if the patch method is callable |
'' |
resource | str | Location of a module containing a resource to use during testing | '' |
patches:
# Value patch
- method: value
value: 1
route: num_a
resource: ./resource.py
# Callable patch
- method: callable
callable_route: hello_world
resource: ./resource.py
Environment
For modules containing a global variable CONFIG
, we can perform tests using different environment variables by the variables as adding key/value pairs under the key environment
in YAML files. The environment variables are accessible from CONFIG.environment.[route]
, where [route]
is the dot-delimited route to the variable within the module.
Example
environment:
NAME_A: a
NAME_C: c
tests:
- environment:
NAME_A: A
NAME_B: b
Spies
We can spy on methods to verify that methods are called when the function being tested is executed. To do this we list the dot delimited routes to the methods to spy on under the key spies
in YAML test files. Spies can be defined at the global, configuration, and test levels; and are combined into one. Spies are saved to the attribute spies_
in the module of the function being tests, and are accessible from an check method.
Example
In this example, the methods to spy on are listed under the spies
key at the individual test level, and an check method to verify spies is defined as a function.
./app_test.py
resources:
- &CHECKS
resource: ./checks.py
spies:
- method_a
tests:
- spies:
- method_b
checks:
- method: check_spies
<< : *CHECKS
expected:
method_a:
called: True
called_with: []
method_b:
called: False
called_with: None
./checks.py
def check_spies(
spies_: dict,
expected: dict,
) -> sns:
spies = {}
for key, value for expected.items():
spy = spies_.get(key, {})
if spy != value
continue
spies[key] = spy
passed = expected == spies
return dict(
output=spies,
expected=expected,
passed=passed, )
Setup
We can setup (create) or teardown (destroy) resources to use during tests. These
resources can be defined as a list of objects under the setup
key in YAML test
files or under configurations options. The resources are created at the start of
the different phases.
Schema
A single setup object has the following fields:
Field | Type | Description | Default |
---|---|---|---|
name | str | The name of the resource | null |
phase | Any | The phase the resource should be created before and destroyed after | null |
method | str | The method to call to setup or teardown the resource | null |
arguments | dict | Arguments to pass to the method | {} |
unpack | bool | Indicates whether arguments should be unpacked or packed when passed to method | false |
timeout | int | Number of seconds to keep the resource alive for: - -1 to keep the process running- 0 to get immediate value from resource- Any other integer to keep process |
0 |
Example
In this example, we define a setup resource for the module phase. The resource stays alive during the phase, or for all of the tests for the module. The method, run_server
, starts an api; and we have define a method and check to verify the response from the api.
./checks.py
def check_equals(
output: str,
expected: str,
) -> dict:
passed = output == expected
return dict(
passed=passed,
output=output,
expected=expected, )
./app.py
import requests
def get_json_response(url: str) -> str:
return requests.get(url).json()
./app_test.py
resources:
- &CHECKS
resource: ./checks.py
- &RESOURCE
resource: ./resource.py
- &URL
url: localhost:1234/hello_world
setup:
- name: api
phase: module
<< : *RESOURCE
method: run_server
arguments:
<< : *URL
unpack: true
timeout: -1
tests:
- function: get_json_response
arguments:
<< : *URL
checks:
- method: check_equals
<< : *CHECKS
expected: '{"hello": "world"}'
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for yaml_testing_framework-0.0.6a8.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | d471cf2c9dc57807da165f34d93fc82adc7db89e78cf2d9f55a70aac8f7cd933 |
|
MD5 | c9cac4270be0144082ba600b3707eab4 |
|
BLAKE2b-256 | b2d5bfedb82199f2da9c53e72f5d2b34d62945678a0903a406ee2c89db7668a6 |
Hashes for yaml_testing_framework-0.0.6a8-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 80787f4462de3f0d25acb55ee1784ac3d8fcfb1d588651cd4f161c05357d8aa6 |
|
MD5 | db2cc17dff16bcff0af7575bef621427 |
|
BLAKE2b-256 | 9f54a129c2984ab5477b16a9c6019b222b7d03069d068c8bed63450213fd66d3 |