Tests a command line program by executing it in a temporary sandbox directory and inspecting its result.
Project description
Tests a command line program by executing it in a temporary sandbox directory and inspecting its result.
Or tests properties of existing files, directories etc.
Supports individual test cases and test suites.
Exactly has a Wiki. It also has a built in help system, which can, among other things, generate this Reference manual.
TEST CASES
A test case is written as a plain text file.
Testing stdin, stdout, stderr, exit code
The following checks that your new my-contacts-program reads a contact list from stdin, and is able to find the email of a person:
[setup] stdin = -file some-test-contacts.txt [act] my-contacts-program get-email-of --name 'Pablo Gauss' [assert] exit-code == 0 stdout equals <<EOF pablo@gauss.org EOF stderr empty
If the file ‘contacts.case’ contains this test case, then Exactly can execute it:
> exactly contacts.case PASS
“PASS” means that all assertions were satisfied.
This test assumes that
the system under test - my-contacts-program - is is found in the same directory as the test case file
the file “some-test-contacts.txt” (that is referenced from the test case) is found in the same directory as the test case file
The home and act-home instructions can be used to change the directories where Exactly looks for files referenced from the test case.
Testing side effects on files and directories
When the execution of a test case starts, the current directory is set to a temporary directory. This gives the test case a sandbox where it can create and manipulate files.
The sandbox - and all files within it - are removed when the execution ends.
The following tests a program that classifies files as either good or bad, by moving them to the appropriate directory:
[setup] dir input dir output/good dir output/bad file input/a.txt = <<EOF GOOD contents EOF file input/b.txt = <<EOF bad contents EOF [act] classify-files-by-moving-to-appropriate-dir GOOD input/ output/ [assert] dir-contents input empty exists -file output/good/a.txt dir-contents output/good num-files == 1 exists -file output/bad/b.txt dir-contents output/bad num-files == 1
file and dir makes files in the current directory (by default).
Using predefined source files
The home directory structure is directories containing predefined files involved in a test case:
- act-home
Location of the program file being tested
- home
Location of arbitrary test resources
Both of them defaults to the directory that contains the test case file, but can be changed via [conf].
There are options for making paths relative to them, and also to the temporary sandbox directories.
-rel-home refers to the home directory, and -rel-act to the temporary directory that is the current directory at the start of the execution:
[conf]
act-home = ../bin/
home = data/
[setup]
copy -rel-home input.txt -rel-act actual.txt
[act]
filter-lines "text to find" actual.txt
[assert]
contents -rel-act actual.txt
equals
-file -rel-home expected.txt
These “relativity” options have defaults designed to minimize the need for them. The following case does the same thing as the one above:
[conf]
act-home = ../bin/
home = data/
[setup]
copy input.txt actual.txt
[act]
filter-lines "text to find" actual.txt
[assert]
contents actual.txt
equals
-file expected.txt
Testing and transforming the contents of files
Use contents to test the contents of a file, or a transformed version of it, by applying a “string transformer”.
Such a “string transformer” may be given a name using the def instruction to make the test easier to read.
The following case tests that “timing lines” are output as part of a log file “log.txt”.
The challenge is that the (fictive) log file contains non-timing lines that the test is not interested in, and that timing lines contains a time stamp of the form “NN:NN”, who’s exact value also is not interesting.
A “string transformer” is used to extract all timing lines and to replace “NN:NN” time stamps with the constant string TIMESTAMP:
[setup]
def line-matcher IS_TIMING_LINE = regex ^timing
def string-transformer REPLACE_TIMESTAMPS = replace [0-9]{2}:[0-9]{2} TIMESTAMP
def string-transformer GET_TIMING_LINES = select IS_TIMING_LINE | REPLACE_TIMESTAMPS
[act]
program-that-writes-log-file
[assert]
contents log.txt
-transformed-by GET_TIMING_LINES
equals <<EOF
timing TIMESTAMP begin
timing TIMESTAMP preprocessing
timing TIMESTAMP validation
timing TIMESTAMP execution
timing TIMESTAMP end
EOF
The -transformed-by option does not modify the tested file, it just applies the assertion to a transformed version of it.
Using external helper programs
External programs can with help with setup and assertions etc.
Exactly can run executable files, shell commands and programs in the OS PATH, using run, $, %.
The following case shows some examples, but doesn’t make sense tough:
[setup]
run my-setup-helper-program first "second arg"
run % mysql -uu -pp -hlocalhost -Dd --batch --execute "create table my_table(id int)"
$ touch file
file root-files.txt = -stdout-from $ ls /
[act]
$ echo ${PATH}
[assert]
run my-assert-helper-program
$ test -f root-files.txt
stdout -from
% echo 'Interesting output'
equals
<<EOF
Interesting output
EOF
[cleanup]
run % mysql -uu -pp -hlocalhost -Dd --batch --execute "drop table my_table"
A program executed in [assert] becomes an assertion that depends on the exit code.
Program values can be defined for reuse using def and run using @:
[setup]
def program RUN_MYSQL = % mysql -uu -pp -hlocalhost -Dd
def program EXECUTE_SQL = @ RUN_MYSQL --skip-column-names --batch --execute
run @ EXECUTE_SQL "create table my_table(id int)"
[act]
system-under-test
[assert]
stdout -from
@ EXECUTE_SQL "select * from my_table"
! empty
[cleanup]
run @ EXECUTE_SQL "drop table my_table"
Testing source code files
The actor instruction can specify an interpreter to test a source code file:
[conf] actor = -file python [act] my-python-program.py 'an argument' second third [assert] stdout equals <<EOF Argument: an argument Argument: second Argument: third EOF
Print output from the tested program
If --act is used, the output of the “act” phase (the “action to check”) will become the output of exactly - stdout, stderr and exit code
[setup] dir a-dir file a-file [act] $ ls [assert] stdout num-lines == 314
> exactly --act my-test.case a-dir a-file
The test case is executed in a temporary sandbox, as usual, but assertions are ignored.
Testing existing OS environment - tests without [act]
A test case does not need to have an [act] phase.
For example, to just check that the names of some SQL files are correct:
[assert]
def path SQL_DIR = -rel-here sql
exists -dir @[SQL_DIR]@
'sql/ must only contain sql files'
dir-contents @[SQL_DIR]@
-selection ! name *.sql
empty
ORGANIZING TESTS
File inclusion
Test case contents can be included from external files:
[setup] including my-dir-symbols.def including my-common-setup-and-cleanup.xly
Test suites
Tests can be grouped in suites:
first.case second.case
or:
[cases] helloworld.case *.case **/*.case [suites] sub-suite.suite *.suite pkg/suite.suite **/*.suite
If the file my-suite.suite contains this text, then Exactly can run it:
> exactly suite my-suite.suite ... OK
The result of a suite can be reported as simple progress information, or JUnit XML.
Suites can contain test case functionality that is common to all cases in the suite. For example:
[cases] *.case [conf] act-home = ../bin/ [setup] def string CONF_FILE = my.conf file @[CONF_FILE]@ = <<EOF common = configuration EOF
The common functionality is included in each test case.
MORE EXAMPLES
The examples/ directory of the source distribution contains more examples.
INSTALLING
Exactly is written in Python and does not require any external libraries.
Exactly requires Python >= 3.5 (not tested on earlier version of Python 3).
Use pip or pip3 to install:
> pip install exactly
or:
> pip3 install exactly
The program can also be run from a source distribution:
> python3 src/default-main-program-runner.py
DEVELOPMENT STATUS
Current version is fully functional, but some syntax and semantics is inconsistent:
Some instructions allow arguments to span multiple lines, some do not.
Most instructions interpret symbol references in arguments, some do not.
Support for escapes characters in strings is missing.
Incompatible changes to syntax and semantics may occur in every release until v 1.0.
Comments are welcome!
Future development
More functionality is needed, smaller and larger. Including (but not limited to):
More string transformers
Possibility to use “program” values in more places, e.g. in [act]
Improved string character escaping
Separate sets of environment variables for “action to check” and other processes
Possibility to set stdin for processes other than the “action to check”
Variables - corresponding to symbol definitions - but for variable values
Macros and functions
Python library for running cases and suites from within Python as a DSEL
THANKS
The Python IDE PyCharm from JetBrains has greatly helped the development of this software.
DEDICATION
Aron Karlén
Tommy Karlsson
Götabergsgatan 10, lägenhet 4
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for exactly-0.9.1.1-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 04b1d8b2aca13688a4ac5deaebca96745ee072e73a9509664a813f8d3991d362 |
|
| MD5 | e6c04fc27a4721ee71a711bedf9df372 |
|
| BLAKE2b-256 | 1ce62dbafa09b2f39d830ee0f22580eeb02f7d3482832d4ffc4ad42f44738e97 |
