A pytest plugin that parameterizes tests from data files.
Project description
pytest-parameterize-from-files
Making Pytest Parameterization Easy and Scalable
pytest-parameterize-from-files
is a
pytest
plugin that parameterizes tests using data
loaded from files using the hook
pytest_generate_tests()
.
Introduction
pytest
has a feature called parameterization that allows you to run
the same test function repeatedly using different inputs to test
multiple cases. However, managing the test data for parameterization can
be a problem. Sometimes the input data is very large or there are many
test cases, so that it is impractical to put all of the data into the
source code of the test.
This plug-in loads the data for the test parameterization from separate data files that are automatically matched up against the test functions. Each function can have one or more data files associated with it, and each file can contain multiple test cases. The data files can be in JSON or YAML format.
An additional issue with the basic pytest parameterization API is how the user must provide the parameters. First all of the test case fixture names in a list, followed by a list of lists with the values the fixtures will take on, and then an optional list of test case id's. Since the labels, values, and test case id's are in separate lists it can be difficult to keep track of which fixture corresponds to which value if you have many of them, and also which group of values corresponds to which test id. The file structure uses a dict to keep the test case id's, fixture names, and data values together in a way that is easier on the human brain.
Features
- Loads data for tests from files into fixtures
- Multiple test data sets may be in one file
- There may be multiple data files for each test
- Fixtures may refer to fixtures in other files
- Intuitive and sane data file structure
Compatibility
This package is a plug-in for pytest
and works with Python 3.9 and up.
- Tested with
pytest
version 7.4.x, should work with any version 6.2.5 or higher - Tested with CPython 3.9-3.12 and PyPy 3.9-3.10
While this code currently has a classifier of "Development Status :: 4 - Beta", it is solid and well-tested. I will likely promote it to "Development Status :: 5 - Production/Stable" after a little more real-world usage.
Installation
You can install pytest-parameterize-from-files
from PyPI by using
pip
:
$ pip install pytest-parameterize-from-files
Usage
To use this plugin you need to make only two changes to your tests:
- Create the data file(s) with the proper names and formats
- Call
pytest
with the flag--param-from-files
An example command line would be:
$ pytest --param-from-files
You can then access the data from the files via the fixtures defined in those files. The most common usage is to manage test case inputs and expected results. This allows the developer to change and add test cases without making changes to the test code.
Just as with pytest's basic parameterization, the test function must have all of the fixtures in its parameter list. Otherwise, an exception will be raised.
The unit tests for this package are good examples of possible ways to
use this package. Look in the files in the tests/
directory and the
corresponding files in the tests/pytester_example_files
directory.
Data File Matching
A data file will be loaded if it matches all of the following criteria:
- The filename starts with
data_
, followed by the name of the test function with the prefixtest_
removed. The remainder of the filename may be any value, and is usually used to identify the tests contained in the file. - The filename must end in
.json
,.yaml
, or.yml
. - The file is contained in a folder at or below the file that contains the test.
For example, for a test function
test_foo(...)
the files
data_foo_part_1.json
data_foo_part_2.yaml
subfolder/data_foo.yaml
would all be loaded.
Caution: Beware of situations where the name of one test is an
extended version of another. E.g., if you have two tests named
test_foo()
and test_foo_bar()
, a data file with the name
data_foo_bar.yaml
will be loaded for both tests. To prevent this,
split the two test functions into two separate files in two different
directories or change the name of one of the test functions. See
test_load_file_extended_name.py
and test_load_separate_subdirs.py
in
the unit test files for this package for concrete examples of what might
happen and how to avoid it.
Data File Structure
Each data file may contain one or more sets of test data, in either yaml
or json format. The top level is a dict whose keys are the test id's.
Each test id is a dict whose keys are fixture names and whose values are
the test data. The test data may be anything, including container types
such as lists or dicts. An example input file data_foo_bar.yaml
might
contain:
test1:
input_data_1: 17
input_data_2: 3
expected_result: 51
test2:
input_data_1:
- abc
input_data_2: 3
expected_result:
- abc
- abc
- abc
This would parameterize into two test cases labeled test1
and test2
,
each with three fixtures, input_data_1
, input_data_2
, and
expected_result
.
Test Case Merging and Conflicts
If the same test case id is present in two different files, the fixtures
from the two files will be merged as long as a fixture with the same
name is not defined more than once for any particular test case id. For
example, for a test function named test_foo()
with two data files:
File data_foo_1.yaml
;
test_case_one:
fixture_one: 17
File data_foo_2.yaml
;
test_case_one:
fixture_two: 170
The function will be passed two fixtures fixture_one=17
and
fixture_two=170
for a test case with id=test_case_one
.
However, if the fixture names are the same there will be a conflict and the code that merges the test cases will raise an exception.
Loading Fixtures by Reference
An additional powerful feature is the ability to load the value for a fixture from another data file. You can have fixture data loaded from another file by setting the fixture value to a specially formatted string. It must be prefixed with two underscores and be of the format:
__<Filename>:<test case id>:<fixture name>
For instance a data file data_other_check_3.yaml
might reference the
data file data_foo_2.yaml
from the previous section:
check_functionality:
input_data_1: 42
other_data: __data_foo_2.yaml:test_case_one:fixture_two
This would result in two fixture values being sent into the test
function, input_data_1 = 42
and other_data = 170
, for a test case
with id = check_functionality
. (Note that there is nothing preventing
an infinite self-referential loop although that is something that should
be avoided).
Reporting Issues
If you encounter any problems, please file an issue including a detailed description and (if possible) an example of the problem.
Contributing
Since this project is a pytest plug-in, it really does require
test-driven development. If you want to contribute a bug fix or new
feature, please first create a test case that demonstrates what your new
code is supposed to do. Note that you need to set things up using the
pytester
fixture, rather than testing directly.
This project uses hatch for its environments and build system, as well as pre-commit and ruff for formatting and linting. Before you send in a pull request, please:
- Set up
pre-commit
and use it to runruff
andmdformat
with the settings included in thepyproject.toml
and.pre-commit-config.yaml
files - Run tests using the command
hatch run test:test
, which will run all of the tests against CPython 3.9-3.12 and PyPy 3.9-3.10 - Check test coverage with
hatch run cov
License
Distributed under the terms of the MIT
license,
pytest-parameterize-from-files
is free and open source software.
Colophon
Inspired by the pytest plug-ins pytest-datadir
, pytest-datafixtures
,
and pytest-xpara
.
- I wanted to load data from files, but
pytest-datadir
andpytest-datafixtures
required code in the test specifically to read in the file. - I liked the way that
pytest-xpara
loaded data into a fixture, but didn't like that it would only work with one file and that I had to specify the file on the command line.
This pytest
plugin was developed using a skeleton generated by
cookiecutter
along with the
cookiecutter-pytest-plugin
template, then extensively modified to bring it up to modern standards.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pytest_parameterize_from_files-0.10.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9ca2c560f695909ea13ff14cc186a1c8a20454d36b66aa0c8afaf1a6b82e1c49 |
|
MD5 | 781088758ebbd5fe2daffb6cc2dc3021 |
|
BLAKE2b-256 | 9dbfa61694df4ba99d1ecf45c5c75cb0e67f32f8330624f13cb5f76d5b142a73 |
Hashes for pytest_parameterize_from_files-0.10-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3f5f32ffd8c3426475a3e7008314cd1034769d4468ab5eb93212c3a845a90e74 |
|
MD5 | 875da84c8120900b58e6e68762ff38b7 |
|
BLAKE2b-256 | 55f35ffc7f9d35c79dec8cec72da04150e11fe39e1ef94c41c8ef34f6937560d |