yaml-testing-framework 0.0.6

A testing framework where tests are defined in YAML files

YAML Testing Framework

A simple, low-code framework for unit testing in Python with tests are defined in YAML files.

Features

Requirements

Python 3.7+

Installation

pip install yaml-testing-framework

Example

Create the files

• test_entrypoint.py - uses pytest to collect and run tests

from types import SimpleNamespace as sns

import pytest


@pytest.mark.parametrize(argnames='test', argvalues=pytest.yaml_tests)
def test_(test: sns) -> None:
  assert test.expected == test.output

• checks.py - contains logic for verifying the output from a function

from typing import Any


 def check_equal(
  expected: Any,
  output: Any,
) -> dict:
  passed = expected == output
  return dict(
    passed=passed,
    output=output,
    expected=expected, )

• add.py - contains the function to test

  def main(a, b):
    return a + b

• add_test.yaml - contains tests for the function

resources:
- &CHECKS
  resource: ./checks.py


tests:
- function: main
  description: Returns the result of adding two integers
  tests:
  - description: Add one negative and one positive integer
    arguments:
      a: 1
      b: -1
    checks:
    - method: equals
      << : *CHECKS
      expected: 0
  - description: Add two positive integers
    arguments:
      a: 1
      b: 2
    checks:
    - method: equals
      << : *CHECKS
      expected: 3
  - description: Add integer and string
    arguments:
      a: 1
      b: '1'
    checks:
    - method: equals
      << : *CHECKS
      field: __class__.__name__
      expected: TypeError

Execute the following command in your command line to run the tests.

pytest --project-path=./add.py

Configuration

The app can be configured within the pytest settings of a configuration file, such as a pytest.ini, or console command when invoking pytest. The configurations are

Field	Type	Description	Default
project-path	str	Location of a directory containing files or an an individual module or YAML file to test.	.
exclude-files	str or list	Patterns found in the paths of files to exclude	[]
exclude-methods	str or list	Patterns found in the names of methods to exclude	[]
include-files	str or list	Patterns found in the paths of files to include	[]
include-methods	str or list	Patterns found in the names of methods to include	[]
yaml-suffix	str	Suffix in the names of YAML files containing tests	_test
setup	str	Defines resources to setup or teardown during tests	''

Configuration file

# ./pytest.ini


[pytest]
project-path = .
exclude_files =
  exclude
  patterns
exclude_methods =
  exclude
  patterns
include_files =
  include
  patterns
include_methods =
  include
  patterns
yaml_suffix = _test
setup = '''
  - name: api
    phase: global
    method: run_server
    arguments:
      host: localhost
      port: 1234
    resource: ./api.py
    timeout: -1
  '''

Console command

pytest --project-path=. --exclude_files exclude patterns --exclude_methods exclude patterns --include_files include patterns --include_method include patterns --yaml-suffix _test --setup '- name: api\nphase: global\nmethod: run_server\narguments:\n  host: localhost\n  port: 1234\nresource: ./api.py\ntimeout: -1\n'

YAML Test Files

Tests are defined in YAML files with the top level keys picked up by the app being:

• configurations - Configurations to be used locally for each test in the YAML files
• tests - Configurations used for multiple of individual tests.

Expanding Tests

Using the app we can define configurations for tests at various levels (configurations, tests, nested tests), expand those configurations out to individual tests. This allows us to reuse configurations and reduce the duplication of content across a YAML file; similar to anchors in YAML, which we can take advantage, along with the other features available in YAML.

Example

This is an abstract example of the expanding configurations done by the app, where the configurations for tests are comprised of:

• config_a - a list
• config_b - an object
• config_c - a string
• config_d - null

In this example, we set these configurations at various levels, globally, tests, and nested tests; and the expanded results are three individual tests containing various values for each configuration.

config_a:
- A
config_b:
  b: B
config_c: C


tests:
- config_a:
  - B
- config_b:
    c: C
  tests:
  - config_a:
    - C
    config_c: C0
  - config_d: D
    tests:
    - config_a:
      - B
      config_b:
        b: B0

tests:  # Expanded
- config_a: # test 1
  - A
  - B  # Appended item
  config_b:
    b: B
  config_c: C
  config_d: null  # Standard test config not defined
- config_a: # test 2
  - A
  - C  # Appended item
  config_b:
    b: B
    c: C  # Added key/value
  config_c: C0  # Replace string
  config_d: null
- config_a: # test 3
  - A
  config_b:
    b: B0  # Updated key/value pair
    c: C
  config_c: C
  config_d: D  # Standard test config defined

Levels (Phases)

The testing process can be broken up into following four ordered phases:

1. global
2. module
3. function
4. check

These phases are synonymous with the levels configurations are be defined at. Configurations described at higher levels are expanded to lower levels, and objects created at higher phases are available at lower phases.

Schema

Details for configurations or fields of an actual test are defined below. These fields can be defined globally or at different test levels.

Field	Type	Description	Expand Action
function	str	Name of function to test	replace
environment	dict	Environment variables used by functions in a module	update
description	str or list	Additional details about the module, function, or test	append
patches	dict or list	Objects in a module to patch for tests	append
cast_arguments	dict or list	Convert function arguments to other data types	append
cast_output	dict or list	Convert function output to other data types	append
checks	dict or list	Verifies the output of functions	append
spies	list or str	List of methods to spy on	append
setup	list or str	Defines objects to setup or teardown during test phases	append
tests	dict or list	Nested configurations that get expanded into individual tests	append

Checks

Methods

We can define "check" methods or reusable functions used to compare expected and actual output from a function being tested. These methods can have any of the fields in the table below as parameters.

Parameter	Type	Description
output	Any	Output from a method to check
expected	Any	Value to verify the output against
setup_	dict	Access objects created or setup to facilitate testing
spies_	dict	Access spies placed on methods
module	ModuleType	Module of the method being tested

Check methods should return a dictionary (or object with dict attribute) containing the fields expected, output, and passed; where passed is a boolean indicating whether the check passed of failed.

Example

Here we define a method that verifies the type of output. If the type matches the expected value the check passes.

• ./checks.py

from typing import Any


def check_type(
  output: Any,
  expected: str,
) -> dict:
  passed = expected == type(output).__name__
  return dict(
    output=output,
    expected=expected,
    passed=passed, )

Schema

Checks are defined in YAML test files under the key checks, and a single check has the following fields:

Field	Type	Description	Default
method	str	Function or method used to verify the result of test	pass_through
expected	Any	The expected output of the function	null
field	str	Sets the output to a dot-delimited route to an attribute or key within the output.	null
cast_output	dict or list	Converts output or an attribute or key in the output before processing an check method	null
resource	str	Location of a module containing a resource to use during testing	''

And single test can have multiple checks

checks:
- method: check_type
  expected: str
  field: null
  resource: ./checks.py
  cast_output:
  - method: __builtins__.str
    resource: ./resource.py

Cast arguments and output

We can convert arguments passed to functions and output from functions to other data types. To do this we define cast objects and list them under the keys cast_arguments and cast_output for tests or cast_output for checks.

Schema

The following fields make up a cast object:

Field	Description	Default
method	Dot-delimited route to a function or object to cast a value to	null
field	Dot-delimited route to a field, attribute, or key of an object. When set the specified field of the object is cast	null
unpack	Boolean indicating whether to unpack an object when casting	False
resource	Location of a module containing a resource to use during testing	''

cast_arguments:
- method: __builtins__.str  # Cast arguments as a string
  field: null
  unpack: false
  resource: ./resource.py
- method: SimpleNamespace  # Cast an output field as a SimpleNamespace
  field: field.key
  unpack: true
  resource: ./resource.py

cast_output:
- method: do_nothing  # Call a casting method
  field: null
  unpack: false
  resource: ./resource.py

Patches

We can patch objects in the module to test before running tests, and since tests are run in individual threads we can different patches for the same object without interference between tests.

Methods

There are four patch methods:

• value - A value to return when the patched object is used.
• callable - A value to return when the patched object is called as function.
• side_effect_list - A list of values to call based off of the number of times the object is called. Returns the item at index n - 1 of the list for the nth call of the object. Reverts to index 0 when number of calls exceeds the length of the list.
• side_effect_dict - A dictionary of key, values for to patch an object with. When the patched object is called with a key, the key's associated value is returned

Schema

Patches are defined at a list of objects in YAML test files under the key patches, and a single patch object has the following fields:

Field	Type	Description	Default
method	str	One of the four patch methods defined above	null
value	Any	The value the patched object should return when called or used	null
route	str	The dot-delimited route to the object we wish to patch, in the module to test	null
callable_route	str	Dot-delimited route to a function if the patch method is callable	''
resource	str	Location of a module containing a resource to use during testing	''

patches:
# Value patch
- method: value
  value: 1
  route: num_a
  resource: ./resource.py
# Callable patch
- method: callable
  callable_route: hello_world
  resource: ./resource.py

Environment

For modules containing a global variable CONFIG, we can perform tests using different environment variables by the variables as adding key/value pairs under the key environment in YAML files. The environment variables are accessible from CONFIG.environment.[route], where [route] is the dot-delimited route to the variable within the module.

Example

environment:
  NAME_A: a
  NAME_C: c
  

tests:
- environment:
    NAME_A: A
    NAME_B: b

Spies

We can spy on methods to verify that methods are called when the function being tested is executed. To do this we list the dot delimited routes to the methods to spy on under the key spies in YAML test files. Spies can be defined at the global, configuration, and test levels; and are combined into one. Spies are saved to the attribute spies_ in the module of the function being tests, and are accessible from an check method.

Example

In this example, the methods to spy on are listed under the spies key at the individual test level, and an check method to verify spies is defined as a function.

• ./app_test.py

resources:
- &CHECKS
  resource: ./checks.py


spies:
- method_a


tests:
- spies:
  - method_b
  checks:
  - method: check_spies
    << : *CHECKS
    expected:
      method_a:
        called: True
        called_with: []
      method_b:
        called: False
        called_with: None

• ./checks.py

def check_spies(
  spies_: dict,
  expected: dict,
) -> sns:
  spies = {}
  for key, value for expected.items():
    spy = spies_.get(key, {})
      if spy != value
        continue
    spies[key] = spy
  passed = expected == spies
  return dict(
    output=spies,
    expected=expected,
    passed=passed, )

Setup

We can setup (create) or teardown (destroy) resources to use during tests. These resources can be defined as a list of objects under the setup key in YAML test files or under configurations options. The resources are created at the start of the different phases.

Schema

A single setup object has the following fields:

Field	Type	Description	Default
name	str	The name of the resource	null
phase	Any	The phase the resource should be created before and destroyed after	null
method	str	The method to call to setup or teardown the resource	null
arguments	dict	Arguments to pass to the method	{}
unpack	bool	Indicates whether arguments should be unpacked or packed when passed to method	false
timeout	int	Number of seconds to keep the resource alive for: - -1 to keep the process running - 0 to get immediate value from resource - Any other integer to keep process	0

Example

In this example, we define a setup resource for the module phase. The resource stays alive during the phase, or for all of the tests for the module. The method, run_server, starts an api; and we have define a method and check to verify the response from the api.

• ./checks.py

def check_equals(
  output: str,
  expected: str,
) -> dict:
  passed = output == expected
  return dict(
    passed=passed,
    output=output,
    expected=expected, )

• ./app.py

import requests


def get_json_response(url: str) -> str:
  return requests.get(url).json()

• ./app_test.py

resources:
- &CHECKS
  resource: ./checks.py
- &RESOURCE
  resource: ./resource.py
- &URL
  url: localhost:1234/hello_world


setup:
- name: api
  phase: module
  << : *RESOURCE
  method: run_server
  arguments:
    << : *URL
  unpack: true
  timeout: -1


tests:
- function: get_json_response
  arguments:
    << : *URL
  checks:
  - method: check_equals
    << : *CHECKS
    expected: '{"hello": "world"}'