Vampytest

Vampytest is a testing framework that allows, but is not limited to, writing relative import based tests.

Table of contents

• Introduction
• Installation
• Writing tests
• Running tests
• Return codes
• Features
  • Assertions
    • Equality assertions
    • Boolean assertions
    • Container assertions
    • Identity assertions
    • Exception assertions
    • Type assertions
    • Reversed assertions
  • Parameterized tests
  • Returning and raising tests
  • Named test cases
  • Skipping tests
  • Reversing test results
  • Garbage collection
  • Capturing output
• Advanced features
  • Testing environments

Introduction

Here are several advantages why you would want to use a relative import supporting testing framework such as vampytest:

• Simplicity and ease of use

  Relative import based tests simplifies the process of writing and executing tests. Relative imports provide a straightforward and intuitive way to import and utilize testing functionalities without the need for complex configuration or setup. Developers can quickly get started with writing tests and running them, making the testing process more accessible and efficient.

• Seamless integration with codebase

  By using relative imports the testing framework seamlessly integrates with the codebase being tested. It allows the developers to import and test modules or components in a manner that reflects the same import structure used in the actual code. This integration enhances readability and maintainability.

• Encourages modular and isolated testing

  Relative import based testing promotes modular and isolated testing practices. It's encouraging the testing of individual units in isolation. This isolation makes it easier to pinpoint issues, debug problems, and maintain the codebase. It also supports the principles of unit testing, where individual units of code are tested independently for their expected behavior.

• Enhanced collaboration and teamwork

  Relative import based testing facilitate collaboration among team members. By utilizing the same import structure as the codebase, it allows multiple developers to work on tests concurrently without conflicts. It makes coordinating testing efforts easier within a team.

• Easy test refactoring

  When refactoring or restructuring the codebase, relative import based testing can make it easier to refactor tests as well. Since the test code closely follows the structure of the codebase, the required changes in test imports are often straightforward, resulting in less effort and potential errors during refactoring.

• Ease of migration and adoption

  If a codebase already follows a relative import structure, using a testing framework that supports them too simplifies the adoption and integration of testing practices. Developers can directly use relative imports in their test code, resulting in a smoother transition to the testing framework.

Installation

To install vampytest, you can use pip. Open your terminal and run the following command:

python3 -m pip install vampytest

Once installed you are ready to run your tests with vampytest.

Writing tests

1. Create a directory for your test suite in your project. The directory should be called tests or have test_ or tests_ prefix.
2. Create a new python file. The file should have test_ prefix for example: test_example.py.
3. In the file import the necessary libraries including vampytest.
4. Write test functions within the file. Each test function should start with the prefix test_.

import vampytest


def test_addition():
    vampytest.assert_eq(2 + 2, 4)

def test_subtraction():
    vampytest.assert_eq(5 - 3, 2)

To check whether the actual behavior matches the expected behavior use assert_... functionalities such as assert_eq. Using python builtin options such as assert works as well, but the report will not be as detailed.

Running tests

To run the tests, navigate to the directory where your project is located using the terminal. Then enter the following command:

vampytest

If the project has a setup file, it will detect which are your project's directories and import its files beforehand. If you do not have a file like that, you might want to navigate into your project or pass the target path where the tests should be loaded from:

vampytest *directory*

By navigating into a specific directory it is possible to limit the test lookup only to the directories / files under it. The same can be achieved by passing the path to the location.

vampytest *directory/sub_directory/etc*

Note: To test vampytest itself vampytest command wont work. Use python3 -m vampytest instead.

Return codes

By reading the return code of the vampytest call it is possible to determine how the testing went without actually reading the output.

Features

Assertions

Vampytest provides a rich set of assertion methods to validate expected behavior and outcomes. These assertions include checking for equality, inequality, truthiness, exceptions, containment, and more.

Here is a list of available assertions:

Here are them in examples:

Equality assertions

# Asserts a == b
vampytest.assert_eq(a, b)

# Asserts c != d
vampytest.assert_ne(c, d)

Boolean assertions

# Asserts bool(e)
vampytest.assert_true(e)

# Asserts not bool(f)
vampytest.assert_false(f)

Container assertions

# Asserts g in h
vampytest.assert_in(g, h)

# Asserts i not in j
vampytest.assert_not_in(i, j)

Identity assertions

# Asserts k is l
vampytest.assert_is(k, l)

# Asserts m is not n
vampytest.assert_is_not(m, n)

Exception assertions

# Asserts that the code inside the context manager raises the defined exception
with vampytest.assert_raises(ValueError):
    raise ValueError

# Asserts that the code inside the context manager raises an equal exception as the defined one.
with vampytest.assert_raises(ValueError('aya')):
    raise ValueError('aya')

# Also asserts whether the given condition is satisfied
with vampytest.assert_raises(ValueError, where = lambda e: 'aya' in repr(e)):
    raise ValueError('ayaya')

Type assertions

# Asserts isinstance(o, str)
vampytest.assert_instance(o, str)

# Asserts q is None or isinstance(q, str)
vampytest.assert_instance(q, str, nullable = True)

# Asserts type(r) is str
vampytest.assert_instance(r, str, accept_subtypes = False)

# Asserts isinstance(s, type) and issubclass(s, str)
vampytest.assert_subtype(s, str)

# Asserts t is None or isinstance(t, type) and issubclass(t, str)
vampytest.assert_subtype(t, str, nullable = True)

Reversed assertions

# Assert not u == v
vampytest.assert_eq(u, v, reverse = True)

# Asserts not isinstance(w, str)
vampytest.assert_instance(w, str, reverse = True)

Parameterized tests

Vampytest allows you to write parameterized tests that run the same test function with different sets of inputs. This allows for more comprehensive testing with fewer lines of code.

import vampytest


@vampytest.call_with(2, 2)
@vampytest.call_with(3, 3)
def test_values_equal(value_0, value_1):
    vampytest.assert_eq(value_0, value_1)

Multiple inputs can also be parameterised with one decorator. This can be useful when the input is more complex, and you might want to define a generator for it.

import vampytest


def input_generator():
    a = object()
    yield a, a

    b = 'apple'
    yield b ,b

    c = int
    yield c, c


@vampytest.call_from(input_generator())
def test_values_identical(value_0, value_1):
    vampytest.assert_is(value_0, value_1)

Returning and raising tests

With vampytest, it is possible to assert the output of tests. This assertion can be done using the returning and raising decorators.

import vampytest


@vampytest.returning(True)
def test_values_equal():
    return 2 == 2


@vampytest.raising(ValueError)
def test_convert_to_int():
    return int('apple')

These decorators pair-up well with parameterised tests.

import vampytest


@vampytest.returning(True)
@vampytest.call_with(2, 2)
@vampytest.call_with(3, 3)
def test_values_equal(value_0, value_1):
    return value_0 == value_1


@vampytest.raising(ValueError)
@vampytest.call_with('apple')
@vampytest.call_with('peach')
def test_convert_to_int(value):
    return int(value)

It is also possible to define the expected output for each set of input.

import vampytest


@vampytest.call_with('apple').raising(ValueError)
@vampytest.call_with('peach').raising(ValueError)
@vampytest.call_with('12').returning(12)
def test_convert_to_int(value):
    return int(value)

For cases when the returned value could be easily calculated from the input parameters the returning_transformed option might be the solution.

import operator
import vampytest


@vampytest.call_with(2, 1).returning_transformed(operator.add)
@vampytest.call_with(3, 1).returning_transformed(operator.add)
@vampytest.call_with(4, 3).returning(0)
def test_sum_if_lt_5(value_0, value_1):
    output = value_0 + value_1
    if output >= 5:
        output = 0

    return output

When using call_from, aside from the already mentioned raising, returning, returning_transformed options, we'll also have returning_last and raising_last as available. As their name implies they take the last input parameter and expect it to be either raised or returned.

import vampytest


@vampytest.call_from(['apple', 'peach']).raising(ValueError)
@vampytest.call_from(['12', '12']).returning(12)
@vampytest.call_from(['6', '42']).returning_transformed(int)
def test_convert_to_int(fruit):
    return int(fruit)


def input_and_return_generator():
    yield {'a': 'b'}, 'a', 'b'
    yield {'b': 'c'}, 'b', 'c'


def input_and_exception_generator():
    yield None, None, TypeError
    yield {}, 'a', KeyError
    yield {}, {}, TypeError


@vampytest.call_from(input_and_return_generator()).returning_last()
@vampytest.call_from(input_and_exception_generator()).raising_last()
def test_get_item_fails(container, key):
    return container[key]

Named test cases

More test cases mean more confusion. To avoid this each test case can also be named.

import vampytest

THRESHOLD = 10

@vampytest.named('over threshold').call_with(THRESHOLD + 1).returning(True)
@vampytest.named('at threshold').call_with(THRESHOLD).returning(True)
@vampytest.named('under threshold').call_with(THRESHOLD - 1).returning(False)
def test_over_or_equal_to_threshold(value):
    return value >= THRESHOLD

This is particularly true when using generator(s) to produce the test cases. When there are many scenarios it becomes cumbersome to find out which one exactly failed even tho the calling parameters are listed to help debugging. Comments are also nice in the code, but they still do not point at the exact location. As a guardian angel, the named test cases descend to save the day.

import vampytest

THRESHOLD = 10

def _iter_options():
    yield (
        'over threshold',
        THRESHOLD + 1,
        True,
    )

    yield (
        'at threshold',
        THRESHOLD,
        True,
    )

    yield (
        'under threshold',
        THRESHOLD - 1,
        False,
    )


@vampytest.call_from(_iter_options()).named_first().returning_last()
def test_over_or_equal_to_threshold(value):
    return value >= THRESHOLD

Skipping tests

Vampytest provides decorators to skip or mark tests as skipped in certain conditions.

import vampytest


class MyType:
    def a():
        return 1


@vampytest.skip()
def test_repr():
    instance = MyType()
    vampytest.assert_instance(repr(instance), str)


@vampytest.skip_if(not hasattr(MyType, 'b'))
def test_repr():
    instance = MyType()
    vampytest.assert_eq(instance.b(), 2)

Reversing test results

If you have a test where you expect the assertions to fail they can be marked with the reverse decorator. These tests will show up as passing when the tests' assertions fail. On the other hand they will show up as failing if the assertions pass.

import vampytest


@vampytest.reverse()
def test_addition():
    vampytest.assert_eq(9 + 10, 21)

Garbage collection

By default, garbage collection is not explicitly called between each test case since it could easily increase the time required to run the tests by 10000% on large projects. By using with_gc it is possible to explicitly call garbage collection before or after a test.

import vampytest


@vampytest.with_gc(after = True, before = True)
def test_addition():
    vampytest.assert_eq(2 + 2, 4)

Capturing output

Capturing stdout and stderr in tests is useful in scenarios where you need to verify the content, format, or structure of the output generated by your code. This technique allows you to intercept and store the output that would normally be printed to the console during the execution of your code, enabling you to inspect and assert against it in your tests.

import vampytest


def test_print():
    capture = vampytest.capture_output()
    with capture:
        print('apple')

    vampytest.assert_eq(capture.get_value(), 'apple\n')

Vampytest is capturing the stdout and stderr by default. If a test fails the captured output will show up in its report. If the test passes its captured output will only show up if all tests passed. This is to help the developer focus on the failing tests firsts. This feature can be useful to help debug failing tests and to catch warnings and forgotten print calls.

Advanced features

Testing environments

Vampytest supports tests to be run in a specific environment to ensure that the code behaves correctly and consistently.

Environments can be set to be used on a specific scope using the set_global_environment, set_directory_environment set_file_environment functions. They can also be set to apply to just a specific test using the in_environment decorator.

from vampytest import DefaultEnvironment, ResultState, in_environment, returning
from vampytest.core.environment.constants import ENVIRONMENT_TYPE_GENERATOR


# We create an environment that unpacks generators into a list
class GenerativeReturnTestEnvironment(DefaultEnvironment):
    __slots__ = ()

    # Define that this environment is only applicable for generators and will propagate the tests' results like that
    identifier = ENVIRONMENT_TYPE_GENERATOR

    # Run is called to run the test with the given parameters and we except it to return a `ResultState`
    def run(self, test, positional_parameters, keyword_parameters):
        try:
            returned_value = [*test(*positional_parameters, **keyword_parameters)]
        except BaseException as raised_exception:
            return ResultState().with_raise(raised_exception)

        return ResultState().with_return(returned_value)

    # Shutdown is called when we do not need this environment anymore. Can be useful when using global environments.
    def shutdown(self):
        pass


@in_environment(GenerativeReturnTestEnvironment())
@returning([1, 2])
def test_generator_in_environment():
    yield 1
    yield 2

Vampytest only defines 2 environments by default: default and scarletio coroutine.

• default environment applies to normal non-generator non-coroutine tests.
• scarletio coroutine environment applies to coroutine tests. Vampytest assumes that every coroutine test is meant to run on a scarletio event loop.

For generators and coroutine generators there is no environment defined by default.

Scarletio based projects might use the same event loop for their whole lifecycle. To use their event loop in the tests use a global environment for it:

import sys

if 'vampytest' in sys.modules:
    from vampytest import ScarletioCoroutineEnvironment, assert_is, set_global_environment

    # import event loop
    from somewhere import EVENT_LOOP

    set_global_environment(ScarletioCoroutineEnvironment(event_loop = EVENT_LOOP))

Test whether the event loops indeed match:

import vampytest
from scarletio import get_event_loop

# import event loop
from somewhere import EVENT_LOOP


# Test whether we are indeed on the correct event loop
async def test_event_loop_same():
    vampytest.assert_is(EVENT_LOOP, get_event_loop())