Buildkite Test Collector for Python
The official Python adapter for Buildkite Test Engine which collects information about your tests.
Supported Python versions: >=3.10
⚒ Supported test frameworks: pytest.
📦 Supported CI systems: Buildkite, GitHub Actions, CircleCI, and others via the BUILDKITE_ANALYTICS_* environment variables.
👉 Installing
-
Create a test suite, and copy the API token that it gives you.
-
Add
buildkite-test-collectorto your project dependencies
Using uv:
uv add --dev buildkite-test-collector
Or add it to your pyproject.toml:
[project.optional-dependencies]
dev = [
"buildkite-test-collector"
]
- Set up your API token
Add the BUILDKITE_ANALYTICS_TOKEN environment variable to your build system's environment.
- Run your tests
Run your tests like normal. Note that we attempt to detect the presence of several common CI environments, however if this fails you can set the CI environment variable to any value and it will work.
uv run pytest
- Verify that it works
If all is well, you should see the test run in the Test Engine section of the Buildkite dashboard.
🏷️ Filtering Tests by Tags
You can filter which tests to run based on execution tags using the --tag-filters option.
First, mark your tests with execution tags:
import pytest
@pytest.mark.execution_tag("color", "red")
def test_red_feature():
assert True
@pytest.mark.execution_tag("color", "blue")
def test_blue_feature():
assert True
Then filter tests by tag using the --tag-filters option with key:value format:
# Run only tests tagged with color:red
pytest --tag-filters "color:red"
# Run only tests tagged with color:blue
pytest --tag-filters "color:blue"
Note: The --tag-filters option performs exact key:value matching. Only tests with the specified tag will be selected.
🔁 Running batches with bktec pool exec
The package includes buildkite-pytest, a persistent runner for
bktec pool exec. It
starts pytest once and runs every batch bktec assigns in the same pytest
process. Each test file is imported and collected the first time a batch needs
it, and session-scoped fixtures are set up once per job. Module- and
class-scoped fixtures are torn down after each batch.
BUILDKITE_TEST_ENGINE_TEST_RUNNER=pytest bktec pool exec -- buildkite-pytest [pytest options]
bktec chooses which tests run and retries failures, so buildkite-pytest
rejects test arguments and options that select, retry or stop tests, such as
-k, -m, -x, --maxfail, --lf, --sw, -n (pytest-xdist),
--reruns (pytest-rerunfailures), --json and --tag-filters. Run it from
pytest's rootdir. Each batch's test executions are uploaded to Test Engine
using BUILDKITE_ANALYTICS_TOKEN.
🎢 Tracing
Buildkite Test Engine has support for tracing potentially slow operations within your tests, and can collect span data of four types: http, sql, sleep and annotations. This is documented as part of our public JSON API so anyone can instrument any code to send this data.
This library supports the ability to transmit tracing information to your Test Engine output by using the new spans pytest fixture. See the SpanCollector documentation for more information.
You may also need to manually capture the data you wish to trace for your use case. For examples of how we've done this in our Ruby test collector, see:
- Instrumenting Rails to capture sql data
- Monkey patching to various libraries to capture http requests
- Monkey patching for sleep
Note: the Ruby test collector is the only Test Engine collector that automatically captures and transmits span data. This Python collector can transmit information, but data capture must be done manually at this time.
🔜 Roadmap
See the GitHub 'enhancement' issues for planned features. Pull requests are always welcome, and we’ll give you feedback and guidance if you choose to contribute 💚
⚒ Developing
After cloning the repository, install uv if you haven't already:
curl -LsSf https://astral.sh/uv/install.sh | sh
Then install the dependencies:
uv sync --all-extras
And run the tests:
uv run pytest
Useful resources for developing collectors include the Buildkite Test Engine docs and the RSpec and Minitest collectors.
👩💻 Contributing
Bug reports and pull requests are welcome on GitHub at https://github.com/buildkite/test-collector-python
🚀 Releasing
See the monorepo's collector release guide.
📜 License
The package is available as open source under the terms of the MIT License.
🤙 Thanks
Thanks to the folks at Alembic for building and maintaining this package.
5 maintainers
Metadata
5 maintainers
Release files for buildkite-test-collector 1.9.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| buildkite_test_collector-1.9.1.tar.gz | 41.7 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| buildkite_test_collector-1.9.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 69.5 kB
Release files / buildkite_test_collector-1.9.1.tar.gz
| Download URL | buildkite_test_collector-1.9.1.tar.gz |
|---|---|
| Size | 41.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
586a40cc97540d3c45f5796d5e3b4f15d3174405425c9f19e403fca829e395c9
|
|
BLAKE2b-256 checksum How to use checksums |
9fe4d05fab2d5aeb05b30904efd5a3a76b241fe556e1bf652d38e13b44668f32
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / buildkite_test_collector-1.9.1-py3-none-any.whl
| Download URL | buildkite_test_collector-1.9.1-py3-none-any.whl |
|---|---|
| Size | 27.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b22561c37c68589b6413107049117b4e41af0f1f1dc08d64d7f9fb598f4aadad
|
|
BLAKE2b-256 checksum How to use checksums |
572ab0df5888d82cb636608312a0c17946562a2dace03a4b06f50ed389b262d9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Security for buildkite-test-collector 1.9.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
