bigtesty 1.1.0

BigTesty is an integration testing framework for BigQuery, running the tests on a real and short-lived infrastructure.

Documentation: tosun-si.github.io/bigtesty

• The real BigQuery engine, no emulator: the SQL behaves exactly as in production.
• Given / When / Then test files in YAML or JSON: load test data, run any SQL (queries, DML, MERGE, scripts, procedures), then assert on query results or table contents.
• Isolated and ephemeral: each scenario of each run gets its own datasets, created and destroyed by BigTesty. Several developers and CI pipelines can run the tests at the same time.
• Production SQL unchanged: the references to your datasets are rewritten to the isolated datasets.

Installation

Python 3.11+ is required.

# as a dev dependency of a uv project, the version is locked in uv.lock
uv add --dev bigtesty
# or as a standalone CLI
uv tool install bigtesty

BigTesty uses Application Default Credentials:

gcloud auth application-default login

Quick start

A BigTesty project is a bigtesty.yaml config file declaring the datasets and tables, and test files named *.bigtesty.yaml (or .yml / .json):

bigtesty.yaml
schema/monitoring/job_failure.json
ddl/monitoring/v_failures_count_by_feature.sql
tests/monitoring/
├── find_failures.bigtesty.yaml
├── merge_failures_summary.bigtesty.yaml
├── assertion_functions/failures_assertions.py
├── given/input_failures_feature_name.json
├── queries/find_failures_by_feature_name.sql
├── then/expected_failures_feature_name.json
└── when/merge_failures_summary.sql

bigtesty.yaml: the datasets and tables created for each scenario.

project: my-project        # or --project / GOOGLE_CLOUD_PROJECT
location: EU
tests: tests

datasets:
  - id: monitoring
    tables:
      - id: job_failure
        schema_file: schema/monitoring/job_failure.json   # BigQuery JSON schema
        partitioning: { type: DAY, field: dwhCreationDate }
        clustering: [featureName, jobName]
      - id: job_failure_summary
        schema:                                            # or inline
          - { name: featureName, type: STRING }
          - { name: failuresCount, type: INT64 }
          - { name: lastFailureDate, type: TIMESTAMP }
    ddl_files:                                             # views, functions, procedures...
      - ddl/monitoring/*.sql

tests/monitoring/merge_failures_summary.bigtesty.yaml: one or several scenarios.

scenarios:
  - name: merge_updates_existing_and_inserts_new_features
    given:
      - table: monitoring.job_failure
        rows:
          - { featureName: featurePSG, jobName: job1, dwhCreationDate: "2026-01-10 10:00:00" }
          - { featureName: featurePSG, jobName: job2, dwhCreationDate: "2026-01-11 10:00:00" }
          - { featureName: featureOM, jobName: job3, dwhCreationDate: "2026-01-12 10:00:00" }
      - table: monitoring.job_failure_summary
        rows:                                   # or rows_file, a JSON or YAML file
          - { featureName: featurePSG, failuresCount: 5, lastFailureDate: "2025-12-31 23:00:00" }
    when:
      sql_file: when/merge_failures_summary.sql
    then:
      - table: monitoring.job_failure_summary
        expected:
          - { featureName: featurePSG, failuresCount: 7, lastFailureDate: "2026-01-11 10:00:00" }
          - { featureName: featureOM, failuresCount: 1, lastFailureDate: "2026-01-12 10:00:00" }
      - query: SELECT featureName FROM monitoring.job_failure_summary GROUP BY featureName HAVING COUNT(*) > 1
        row_count: 0

Run the tests:

bigtesty test

PASSED  monitoring/find_failures.bigtesty.yaml::find_failures_by_feature_name  (4.7s, 10.0 MiB billed)
PASSED  monitoring/find_failures.bigtesty.yaml::find_failures_by_feature_name_with_functions  (4.1s, 10.0 MiB billed)
PASSED  monitoring/find_failures.bigtesty.yaml::find_failures_by_job_name  (3.8s, 10.0 MiB billed)
FAILED  monitoring/merge_failures_summary.bigtesty.yaml::merge_updates_existing_and_inserts_new_features  (8.2s, 40.0 MiB billed)

________ monitoring/merge_failures_summary.bigtesty.yaml::merge_updates_existing_and_inserts_new_features ________
then[0] / rows:
    expected 2 rows, got 2, diff from expected to actual:
    Value of root[0]['failuresCount'] changed from 8 to 7.

3 passed, 1 failed, 70.0 MiB billed (run zivarf)

The bytes billed are the ones of the query jobs (DDL, when and then), BigQuery bills at least 10 MiB per query. The test data is loaded with load jobs, which are free.

Runnable examples, from the basics to a uv project with its CI, are in the examples folder.

Test files

A scenario has 3 blocks:

• given: the rows loaded into the tables, inline (rows) or from a JSON / YAML file (rows_file). The rows are loaded with load jobs: they are immediately available for DML statements.
• when (optional): the SQL under test (sql or sql_file), executed as a BigQuery script: queries, DML, MERGE, CALL of a procedure, multi-statements...
• then: one or several assertions on a query, a query_file or a whole table:
  • expected / expected_file: the expected rows,
  • row_count: the expected number of rows,
  • functions: custom Python assertion functions, called with (actual_rows, expected_rows).

Options of the then blocks:

The actual and expected values are compared with the types of the query result: a TIMESTAMP written "2026-01-11 10:00:00", "2026-01-11T10:00:00Z" or "2026-01-11T12:00:00+02:00" is the same value, a NUMERIC can be written 1.5 or "1.50", a DATE "2026-01-11"...

In YAML, yes / no / on / off and the dates stay strings: only true / false are booleans.

IDE validation

JSON Schemas are published for the config and the test files. With the VS Code YAML extension or JetBrains IDEs:

# yaml-language-server: $schema=https://raw.githubusercontent.com/tosun-si/bigtesty/main/schemas/bigtesty-test.schema.json

Isolation and SQL rewrite

For each run, BigTesty creates one dataset per configured dataset and per scenario, e.g. monitoring_hlnyks_0, labelled with bigtesty_run_id. The references to the configured datasets in the SQL (DDL files, when and then) are rewritten to these datasets, the formatting of the SQL is kept:

• monitoring.job_failure, `monitoring.job_failure`, `monitoring`.`job_failure`
• my-project.monitoring.job_failure, when my-project is the test project or listed in project_aliases (e.g. the production project hardcoded in the SQL).

The comments, string literals and references to other datasets or projects are not touched. Don't alias a table with the name of a configured dataset (FROM x AS monitoring), it would be rewritten.

The datasets are destroyed at the end of the run (cleanup: always by default, or on_success / never to analyse the result in BigQuery). As a safety net, their tables expire after table_expiration_hours (6 by default), and bigtesty gc deletes the datasets left by killed runs.

CLI

bigtesty test                      # run all the scenarios
bigtesty test -k merge             # only the scenarios whose ID contains "merge"
bigtesty test --cleanup never -v   # keep the datasets, show the executed SQL
bigtesty test --update            # rewrite the expected files that don't match from the actual rows
bigtesty validate                  # validate the config and test files, no call to BigQuery
bigtesty gc --older-than-hours 6   # delete the datasets left by killed runs
bigtesty infra create -o json      # create the infra for the tests of another language
bigtesty infra destroy --run-id x  # and delete it
bigtesty schema -o schemas         # write the JSON Schemas

The exit code is 0 when all the scenarios pass, 1 when a scenario fails and 2 for an invalid config.

Run with pytest

BigTesty is also a pytest plugin, installed with the package: pytest collects the *.bigtesty.yaml / .json files, one test per scenario. The pytest features come for free: -k, --lf, JUnit XML reports for the CI, IDE integration...

pytest tests --bigtesty-project my-project
pytest tests -k merge --junitxml=report.xml

tests/monitoring/find_failures.bigtesty.yaml::find_failures_by_feature_name PASSED              [ 25%]
tests/monitoring/find_failures.bigtesty.yaml::find_failures_by_feature_name_with_functions PASSED [ 50%]
tests/monitoring/find_failures.bigtesty.yaml::find_failures_by_job_name PASSED                  [ 75%]
tests/monitoring/merge_failures_summary.bigtesty.yaml::merge_updates_existing_and_inserts_new_features PASSED [100%]

The ephemeral infra is created once for the selected scenarios, and the scenarios run in parallel on the first BigTesty test executed: this test shows the duration of the whole run. pytest-xdist is not needed.

The plugin does nothing in a project without BigTesty test files.

Use the ephemeral infra in Python tests

To test Python code reading or writing BigQuery, BigTestyInfra creates the datasets and tables of a bigtesty.yaml once in a setup and deletes them once in a teardown, around as many tests as needed, with pytest or unittest:

import pytest

from bigtesty.testing import BigTestyInfra


@pytest.fixture(scope="class")   # the scope is the lifecycle of the infra
def infra():
    with BigTestyInfra.from_config("bigtesty.yaml") as infra:
        infra.load("monitoring.job_failure", [{"featureName": "featurePSG", "jobName": "job1"}])
        build_failures_summary(infra.client, infra.dataset_id("monitoring"))   # the code under test
        yield infra


class TestFailuresSummary:
    def test_one_row_per_feature(self, infra):
        rows = infra.query("SELECT featureName, failuresCount FROM monitoring.job_failure_summary")
        assert rows == [{"featureName": "featurePSG", "failuresCount": 1}]

infra.query rewrites the configured datasets as in the test files, infra.reset() empties the tables between the scenarios sharing an infra, and anything callable from Python (a job, a service, dbt...) can run on the infra. The plugin also provides a session-scoped bigtesty_infra fixture. See the Python tests page and the 05-python-tests example.

The tests of the other languages (Java / JUnit 5, Go...) get the same infra with bigtesty infra create --output json in their setup and bigtesty infra destroy in their teardown, with the CLI or the Docker image, Testcontainers included: see the Other languages page.

Run with Docker

docker run -it \
   -e GOOGLE_CLOUD_PROJECT=$PROJECT_ID \
   -v $(pwd)/examples/01-queries:/usr/local/src/app/project \
   -v $HOME/.config/gcloud:/usr/local/src/app/.config/gcloud \
   groupbees/bigtesty test --config project/bigtesty.yaml

Mounting the local gcloud configuration authenticates the container with your Application Default Credentials, without a long-lived service account key.

Run in CI

The 04-uv-project example runs the tests with uv and pytest in GitHub Actions, GitLab CI and Cloud Build.

Migrating from 0.x

BigTesty 1.0 replaces Pulumi by the BigQuery client and introduces a new config and test file format: tables.json becomes a bigtesty.yaml config file, the definition_*.json files become *.bigtesty.yaml / *.bigtesty.json test files, and the --iac-backend-url and folder options of the CLI are replaced by --config. Python 3.11+ is required.

The migration guide gives the mapping of every option and field, with before / after examples.

Documentation

The full documentation is on the BigTesty website.