Lightweight testing helpers
Project description
coveo-testing
A set of test/pytest helpers to facilitate common routines.
Content in a nutshell:
-
Reusable pytest markers (UnitTest, IntegrationTest)
-
Unique ID generation for tests
-
Multiline logging assertions with includes, excludes, levels and comprehensive assertion output
-
Refactorable
unittest.mock.patch('this.module')
module references -
Human-readable (but still customizable) display for parametrized tests
This project is used as the test base for all other projects in this repository.
Therefore, it cannot depend on any of them.
More complex use cases may be implemented in the coveo-testing-extras
project.
That's also where you can depend on projects that depend on coveo-testing
.
pytest markers and auto-registration
This enables code completion on markers.
Three markers are already provided: [UnitTest, Integration, Interactive]
Here's how you can create additional markers:
# /test_some_module/markers.py
import pytest
DockerTest = pytest.mark.docker_test
CloudTest = pytest.mark.cloud_test
ALL_MARKERS = [DockerTest, CloudTest]
You can then import these markers and decorate your test functions accordingly:
# /test_some_module/test_something.py
from coveo_testing.markers import UnitTest, Integration, Interactive
from test_some_module.markers import CloudTest, DockerTest
@UnitTest
def test_unit() -> None:
... # designed to be fast and lightweight, most likely parametrized
@Integration
def test_integration() -> None:
... # combines multiple features to achieve a test
@CloudTest
def test_in_the_cloud() -> None:
... # this could be a post-deployment test, for instance.
@DockerTest
@Integration
def test_through_docker() -> None:
... # will run whenever docker tests or integration tests are requested
@Interactive
def test_interactive() -> None:
... # these tests rely on eye-validations, special developer setups, etc
Pytest will issue a warning when markers are not registered.
To register coveo-testing's markers along with your custom markers, use the provided register_markers
method:
# /test_some_module/conftest.py
from _pytest.config import Config
from coveo_testing.markers import register_markers
from test_some_module.markers import ALL_MARKERS
def pytest_configure(config: Config) -> None:
"""This pytest hook is ran once, before collecting tests."""
register_markers(config, *ALL_MARKERS)
Human-readable unique ID generation
The generated ID has this format:
friendly-name.timestamp.pid.host.executor.sequence
-
friendly-name:
- provided by you, for your own benefit
-
timestamp:
- format "%m%d%H%M%S" (month, day, hour, minutes, seconds)
- computed once, when TestId is imported
-
pid:
- the pid of the python process
-
host:
- the network name of the machine
-
executor:
- the content of the
EXECUTOR_NUMBER
environment variable - returns 'default' when not defined
- historically, this variable comes from jenkins
- conceptually, it can be used to help distribute (and identify) tests and executors
- the content of the
-
sequence:
- Thread-safe
- Each
friendly-name
has an isolatedsequence
that starts at 0 - Incremented on each new instance
- Enables support for parallel parametrized tests
from coveo_testing.temporary_resource.unique_id import TestId, unique_test_id
# the friendly name is the only thing you need to specify
test_id = TestId('friendly-name')
str(test_id)
'friendly-name.0202152243.18836.WORKSTATION.default.0'
# you can pass the instance around to share the ID
str(test_id)
'friendly-name.0202152243.18836.WORKSTATION.default.0'
# create new instances to increment the sequence number
test_id = TestId('friendly-name')
str(test_id)
'friendly-name.0202152243.18836.WORKSTATION.default.1'
# use it in parallel parameterized tests
import pytest
@pytest.mark.parametrize('param', (True, False))
def test_param(param: bool, unique_test_id: TestId) -> None:
# in this case, the friendly name is the function name and
# the sequence will increase on each parameter
# test_param.0202152243.18836.WORKSTATION.default.0
# test_param.0202152243.18836.WORKSTATION.default.1
...
multiline logging assertions
Maybe pytest's caplog
is enough for your needs, or maybe you need more options.
This tool uses in
and not in
to match strings in a case-sensitive way.
import logging
from coveo_testing.logging import assert_logging
with assert_logging(
logging.getLogger('logger-name'),
present=['evidence1', 'evidence2'],
absent=[...],
level=logging.WARN):
...
Human-readable (but still customizable) display for parametrized tests
If you're like me, you typed @pytest.mark.parametrize
wrong a couple of times!
Enable IDE completion by using this one instead:
from coveo_testing.parametrize import parametrize
@parametrize('var', (True, False))
def test_var(var: bool) -> None:
...
It has one difference vs the pytest one, and it's the way it formats the "parameter name" for each iteration of the test.
Pytest will skip a lot of types and will simply name your test "var0", "var1" and so on.
Using this @parametrize
instead, the variable's content will be inspected:
from typing import Any
from coveo_testing.parametrize import parametrize
import pytest
class StrMe:
def __init__(self, var: Any) -> None:
self.var = var
def __str__(self) -> str:
return f"Value: {self.var}"
@parametrize('var', [['list', 'display'], [StrMe('hello')]])
def test_param(var: bool) -> None:
...
@pytest.mark.parametrize('var', [['list', 'display'], [StrMe('hello')]])
def test_param_from_pytest(var: bool) -> None:
...
If you run pytest --collect-only
you will obtain the following:
<Function test_param[list-display]>
<Function test_param[Value: hello]>
<Function test_param_from_pytest[var0]>
<Function test_param_from_pytest[var1]>
Refactorable mock targets
Demo
Consider this common piece of code:
from unittest.mock import patch, MagicMock
@patch("mymodule.clients.APIClient._do_request")
def test(api_client_mock: MagicMock) -> None:
...
Because the mock target is a string, it makes it difficult to move things around without breaking the tests. You need a
tool that can extract the string representation of a python objet. This is what ref
was built for:
from unittest.mock import patch, MagicMock
from coveo_testing.mocks import ref
from mymodule.clients import APIClient
@patch(*ref(APIClient._do_request))
def test(api_client_mock: MagicMock) -> None:
...
🚀 This way, you can rename or move mymodule
, clients
, APIClient
or even _do_request
, and your IDE should find
these and adjust them just like any other reference in your project.
Let's examine a more complex example:
from unittest.mock import patch, MagicMock
from mymodule.tasks import process
@patch("mymodule.tasks.get_api_client")
def test(get_api_client_mock: MagicMock) -> None:
assert process() is None # pretend this tests the process function
The interesting thing in this example is that we're mocking get_api_client
in the tasks
module.
Let's take a look at the tasks
module:
from typing import Optional
from mymodule.clients import get_api_client
def process() -> Optional[bool]:
client = get_api_client()
return ...
As we can see, get_api_client
is defined in another module.
The test needs to patch the function in the tasks module since that's the context it will be called from.
Unfortunately, inspecting get_api_client
from the tasks
module at runtime leads us back to mymodule.clients
.
This single complexity means that hardcoding the context mymodule.tasks
and symbol get_api_client
into a string
for the patch is the straightforward solution.
But with ref
, you specify the context separately:
from unittest.mock import patch, MagicMock
from coveo_testing.mocks import ref
from mymodule.clients import get_api_client
from mymodule.tasks import process
@patch(*ref(get_api_client, context=process))
def test(get_api_client_mock: MagicMock) -> None:
assert process() is None # pretend this tests the process function
🚀 By giving a context to ref
, the symbol get_api_client
will be resolved from the context of process
, which is the
mymodule.tasks
module. The result is mymodule.tasks.get_api_client
.
If either objects (get_api_client
or process
) are moved or renamed using a refactoring tool, the mock will still
point to the correct name and context.
🚀 And a nice bonus is that your IDE can jump to get_api_client
's definition from the test file now!
It should be noted that this isn't just some string manipulation. ref
will import and inspect modules and objects
to make sure that they're correct. Here's a more complex case with a renamed symbol:
The module:
from typing import Optional
from mymodule.clients import get_api_client as client_factory # it got renamed! 😱
def process() -> Optional[bool]:
client = client_factory()
return ...
The test:
from unittest.mock import patch, MagicMock
from coveo_testing.mocks import ref
from mymodule.clients import get_api_client
from mymodule.tasks import process
@patch(*ref(get_api_client, context=process))
def test(get_api_client_mock: MagicMock) -> None:
assert process() is None # pretend this tests the process function
Notice how the test and patch did not change despite the renamed symbol?
🚀 This is because ref
will find get_api_client
as client_factory
when inspecting mymodule.tasks
module,
and return mymodule.tasks.client_factory
.
We can also use ref with patch.object()
in order to patch a single instance. Consider the following code:
from unittest.mock import patch
from mymodule.clients import APIClient
def test() -> None:
client = APIClient()
with patch.object(client, "_do_request"):
...
🚀 By specifying obj=True
to ref
, you will obtain a Tuple[instance, attribute_to_patch_as_a_string]
that you
can unpack to patch.object()
:
from unittest.mock import patch
from coveo_testing.mocks import ref
from mymodule.clients import APIClient
def test() -> None:
client = APIClient()
with patch.object(*ref(client._do_request, obj=True)):
...
Please refer to the docstring of ref
for argument usage information.
Common Mock Recipes
Mock something globally without context
Option 1: by leveraging the import mechanism
To mock something globally without regards for the context, it has to be accessed through a dot .
by the context.
For instance, consider this test:
from http.client import HTTPResponse
from unittest.mock import patch, MagicMock
from coveo_testing.mocks import ref
from mymodule.tasks import process
@patch(*ref(HTTPResponse.close))
def test(http_response_close_mock: MagicMock) -> None:
assert process()
The target is HTTPResponse.close
, which lives in the http.client
module.
The contextof the test is the process
function, which lives in the mymodule.tasks
module.
Let's take a look at mymodule.tasks
's source code:
from http import client
def process() -> bool:
_ = client.HTTPResponse(...) # of course this is fake, but serves the example
return ...
Since mymodule.tasks
reaches HTTPResponse
through a dot (i.e.: client.HTTPResponse
), we can patch HTTPResponse
without using mymodule.tasks
as the context.
However, if mymodule.tasks
was written like this:
from http.client import HTTPResponse
def process() -> bool:
_ = HTTPResponse(...)
return ...
Then the patch would not affect the object used by the process
function anymore. However, it would affect any other
module that uses the dot to reach HTTPResponse
since the patch was still applied globally.
Option 2: By wrapping a hidden function
Another approach to mocking things globally is to hide a function behind another, and mock the hidden function. This allows modules to use whatever import style they want, and the mocks become straightforward to setup.
Pretend this is mymodule.clients
:
class APIClient:
...
def get_api_client() -> APIClient:
return _get_api_client()
def _get_api_client() -> APIClient:
return APIClient()
And this is mymodule.tasks
:
from mymodule.clients import get_api_client
def process() -> bool:
return get_api_client() is not None
So you know this works globally, because no one will (should?) import the private one except the test:
from unittest.mock import patch, MagicMock
from coveo_testing.mocks import ref
from mymodule.tasks import process
from mymodule.clients import _get_api_client
@patch(*ref(_get_api_client))
def test(api_client_mock: MagicMock) -> None:
assert process()
Mock something for a given context
When you want to mock something for a given module, you must provide a hint to ref
as the context
argument.
The hint may be a module, or a function/class defined within that module. "Defined" here means that "def" or "class" was used in that module. If the hint was imported into the module, it will not work:
mymodule.tasks
:
from mymodule.clients import get_api_client
def process() -> bool:
client = get_api_client()
return ...
The test, showing 3 different methods that work:
from unittest.mock import patch, MagicMock
from coveo_testing.mocks import ref
from mymodule.clients import get_api_client
from mymodule.tasks import process
# you can pass the module as the context
import mymodule
@patch(*ref(get_api_client, context=mymodule.tasks))
def test(get_api_client_mock: MagicMock) -> None:
assert process()
# you can pass the module as the context, version 2
from mymodule import tasks
@patch(*ref(get_api_client, context=tasks))
def test(get_api_client_mock: MagicMock) -> None:
assert process()
# you can also pass a function or a class defined in the `tasks` module
from mymodule.tasks import process
@patch(*ref(get_api_client, context=process))
def test(get_api_client_mock: MagicMock) -> None:
assert process()
The 3rd method is encouraged: provide the function or class that is actually using the get_api_client
import.
In our example, that's the process
function.
If process
was ever moved to a different module, it would carry the get_api_client
import, and the mock would
be automatically adjusted to target process
's new module without changes.
Mock something for the current context
Sometimes, the test file is the context. When that happens, just pass __name__
as the context:
from unittest.mock import patch
from coveo_testing.mocks import ref
from mymodule.clients import get_api_client, APIClient
def _prepare_test() -> APIClient:
client = get_api_client()
...
return client
@patch(*ref(get_api_client, context=__name__))
def test() -> None:
client = _prepare_test()
...
Mock a method on a class
Since a method cannot be imported and can only be accessed through the use of a dot .
on a class or instance,
you can always patch methods globally:
with patch(*ref(MyClass.fn)): ...
This is because no module can import fn
; it has to go through an import of MyClass
.
Mock a method on one instance of a class
Simply add obj=True
and use patch.object()
:
with patch.object(*ref(instance.fn, obj=True)): ...
Mock an attribute on a class/instance/module/function/object/etc
ref
cannot help with this task:
- You cannot refer an attribute that exists (you would pass the value, not a reference)
- You cannot refer an attribute that doesn't exist (because it doesn't exist!)
For this, there's no going around hardcoding the attribute name in a string:
class MyClass:
def __init__(self) -> None:
self.a = 1
def test_attr() -> None:
instance = MyClass()
with patch.object(instance, "a", new=2):
assert instance.a == 2
assert MyClass().a == 1
This sometimes work when patching instances.
The example works because a
is a simple attribute that lives in instance.__dict__
and patch.object
knows
about that.
But if you tried to patch MyClass
instead of instance
, mock.patch
would complain that there's no
such thing as a
over there.
Thus, patching an attribute globally will most likely result in a lot of wasted time, and should be avoided.
There's no way to make the example work with ref
because there's no way to refer instance.a
without actually
getting the value of a
, unless we hardcode a string, which defeats the purpose of ref
completely.
Mock a property
You can only patch a property globally, through its class:
class MyClass:
@property
def get(self) -> bool:
return False
from unittest.mock import PropertyMock, patch, MagicMock
from coveo_testing.mocks import ref
from mymodule import MyClass
@patch(*ref(MyClass.get), new_callable=PropertyMock, return_value=True)
def test(my_class_get_mock: MagicMock) -> None:
assert MyClass().get == True
my_class_get_mock.assert_called_once()
You cannot patch a property on an instance, this is a limitation of unittest.mock
because of the way
properties work.
If you try, mock.patch.object()
will complain that the property is read only.
Mock a classmethod or staticmethod on a specific instance
When inspecting these special methods on an instance, ref
ends up finding the class instead of the instance.
Therefore, ref
is unable to return a Tuple[instance, function_name]
.
It would return Tuple[class, function_name]
, resulting in a global patch. 😱
But ref
will detect this mistake, and will raise a helpful exception if it cannot return an instance when you
specified obj=True
.
For this particular scenario, the workaround is to provide the instance as the context:
from unittest.mock import patch
from coveo_testing.mocks import ref
class MyClass:
@staticmethod
def get() -> bool:
return False
def test() -> None:
instance = MyClass()
with patch.object(*ref(instance.get, context=instance, obj=True)) as fn_mock:
assert instance.get == True
assert MyClass().get == False # new instances are not affected by the object mock
fn_mock.assert_called_once()
Some may prefer a more semantically-correct version by specifying the target through the class instead of the instance. In the end, these are all equivalent:
with patch.object(instance, "get"):
...
with patch.object(*ref(instance.get, context=instance, obj=True)):
...
with patch.object(*ref(MockClass.get, context=instance, obj=True)):
...
In this case, the version without ref is much shorter and arguably more pleasant for the eye, but get
can no longer
be renamed without altering the tests.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for coveo_testing-2.0.8-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6ed6b02d76c9793cef1aecd136aaa0e81940364fd208b716b1e1e4f7b6f9f18c |
|
MD5 | 717cb0ab700c15e59481135960a37324 |
|
BLAKE2b-256 | 8c5801e5458ac076558fa66f35578cdd4538f4aa91fd5c402e1f8be9954d05d5 |