Tool for testing semantically equivalent samples in multiple languages and environments
Project description
Sample Tester
Version: 0.7.7
PRE-RELEASE: This surface is not guaranteed to be stable. Breaking changes may still be applied.
Setup
-
Activate your preferred virtual environment.
. PATH/TO/YOUR/VENV/bin/activate
-
Install the necessary requirements. This needs to be done first even if you pip-install the sample-tester.
pip install pyyaml
-
Install this sample-tester package in ONE of the following two ways:
- Install from PyPI (recommended). This will put the command
sample-tester
in your path.
pip install sample-tester
- Clone this GitHub repo (if you're going to be modifying it). The top-level directory includes a
sample-tester
command.
git clone https://github.com/googleapis/sample-tester.git
- Install from PyPI (recommended). This will put the command
-
To run against Google APIs, ensure you have credentials set up:
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/your/creds.json
Setting up the test
Set up the test plan as in ./examples/lang_region/language.test.yaml
. That sample test has three equivalent representations of the same test, one with absolute artifact paths in the imperative style, the second with canonical artifact paths in the imperative style, and the third with canonical artifact paths in the declarative style. (See NOTES below). Some features that may not be obvious from that test file:
- You can have any number of test suites.
- Each test suite can have
setup
,teardown
, andcases
sections. - The
cases
section is a list of test cases. For each test case,setup
is executed before running the test case andteardown
is executed after. setup
,teardown
and eachcases[...].spec
is a list of directives and arguments. The directives can be any of the following YAML directives:log
: print the arguments, printf styleuuid
: return a uuid (if called from yaml, assign it to the variable names as an argument)shell
: run in the shell the command specified in the argumentcall
: call the artifact named in the argument, error if the call failscall_may_fail
: call the artifact named in the argument, do not error even if the call failsassert_contains
: require the given variable to contain a stringassert_not_contains
: require the given variable to not contain a stringassert_success
: require that the exit code of the lastcall_may_fail
was 0. If the preceding call was a just acall
, it would have already failed on a non-zero exit code.assert_failure
: require that the exit code of the lastcall_may_fail
orcall
was NOT 0. Note, though, that if we're executing this after just acall
, it must have succeeded so this assertion will fail.env
: assign the value of an environment variable to a testcase variablecode
: execute the argument as a chunk of Python code. The other directives above are available as Python calls with the names above. In addition, the following functions are available inside Pythoncode
only:fail
: mark the test as having failed, but continue executingabort
: mark the test as having failed and stop executingassert_that
: if the condition in the first argument is false, abort the test
- In the usual case, you will be using the "manifest" convention. Thus, you will need one or more manifest files (
*.manifest.yaml
) listing the path and identifiers for each sample. Seeconvention/manifest/sample.manifest.yaml
for an explanation of the structure of the*.manifest.yaml
files.
Manifest File
A manifest file is a YAML file that associates each artifact (sample) of interest on disk with a series of tags that can be used to uniquely identify that artifact. Right now only version "1" of the manifest file format is supported, but we have the version as the first field in the file to support different schemas in the future.
The fundamental unit in a manifest is the "item", which is a collection of label name/value pairs; each unit should correspond to exactly one artifact on disk. Some labels are of special interest to the sample test runner, such as those named language
, path
, bin
, and region_tag
. These four are interpreted, respectively, as the programming language of the given artifact, the path to that artifact on disk, the binary used to execute the artifact (if the artifact is not itself executable), and the unique region tag by which to quickly identify the artifact for the given language. In particular, artifacts with the same region_tag
but different language
s are taken to represent the same conceptual sample, but implemented in the different programming languages; this allows a test specification to refer to the region_tag
s only and the runner will then run that test for each of the language
s available.
Since a lot of the artifacts will share part or all of some labels (for example, the initial directory components, or the binary used to execute them), "items" are grouped into "sets". Each set may define its own label name/value pairs. These pairs are applied to each of the items inside the set as follows:
- If the item does not define a given label name, then the label name/value pair in its set is applied to the item.
- If the item does define a given label name, then the corresponding label value specified in the set is prepended to the corresponding value specified in the item. This is particularly useful in the case of paths: the set may define the common path for all of its items, and each item specifies its unique trailing directories and filename.
See ./sample.manifest.yaml
for a concrete, commented example.
Running tests
The common usage is:
./sampletester TEST.yaml [TEST.yaml ...] [MANIFEST.manifest.yaml ...]
[--envs=REGEX] [--suites=REGEX] [--cases=REGEX]
[--fail-fast]
[--convention CONVENTION:ARG,ARGS]
where:
- there can be any number of
TEST.yaml
test plan files - there can be any number of
MANIFEST.manifest.yaml
manifest files --envs
,--suites
, and--cases
are Python-style regular expressions (beware shell-escapes!) to select which environments, suites, and cases to run, based on their names. All the environemnts, suites, or cases will be selected to run by default if the corresponding flag is not set. Note that if an environment is not selected, its suites are not selected regardless of--suites
; if a suite is not selected, its testcases are not selected regardless of--cases
.--fail-fast
makes execution stop as soon as a failing test case is encountered, without executing any remaining test cases.--convention
(or-c
) can be used to specify both a particular convention to resolve the sample IDs to disk artifacts, and a list of comma-separated arguments to that convention. The default convention istag:region_tag
, which uses theregion_tag
key in the manifest files. To use, say, thetarget
key in the manifest, simply pass-c tag:target
.
For example, my own invocation to run a test on the fake samples under testdata/
is
./sample-tester examples/convention-tag/language.test.yaml examples/convention-tag/language.manifest.yaml
Output
In all cases, sample-tester
exits with a non-zero code if there were any errors in the flags, test config, or test execution.
In addition, by default sampletester
prints the status of test cases to stdout. This output is controlled by the following flags:
--verbosity
(-v
): controls how much output to show for passing tests. The default is a "summary" view, but "quiet" (no output) and "detailed" (full case output) options are available.--suppress_failures
(-f
): Overrides the default behavior of showing output for failing test cases, regardless of the--verbosity
setting
The flag --xunit=FILE
outputs a test summary in xUnit format to FILE
(use -
for stdout).
Advanced usage
If you want to change the convention in use, you can pass the --convention=NAME
flag. You can provide any number of paths to the convention. (In the default case, the paths are just manifest files, but these could be anything).
./sampletester TEST.yaml [--convention=CONVENTION] [TEST.yaml ...] [USERPATH ...]
If you want to define your own convention, just add a package or module (whose name will become the convention name) under convention/
. Have that package or module export a function test_environments
which returns an array of instance of child classes of testenv.Base
.
Development
During development, run the devcheck
script to run all tests and examples and verify they work as expected.
Disclaimer
This is not an official Google product.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.