scruf 0.4

Functional testing for command line applications

Scruf is a functional testing framework for command line applications. It is heavily inspired by cram. This is currently early in development and while I aim to maintain the current interface breaking changes may occur.

usage: scruf [-h] [--no-cleanup] [-s SHELL] [-i INDENT] [-e ENV_FILE]
             FILE [FILE ...]

positional arguments:
  FILE                  File(s) to be tested

optional arguments:
  -h, --help            show this help message and exit
  --no-cleanup          Avoid cleaning up temporary test directory
  -s SHELL, --shell SHELL
                        Path to shell to be used to run tests with. Default is
                        '/bin/sh'
  -i INDENT, --indent INDENT
                        String to be used for detecting indentation when
                        parsing tests. Default is 4 spaces, use a literal '\t'
                        to denote a tab character
  -e ENV_FILE, --env-file ENV_FILE
                        Name of config file to read environment variables from
  --version             show program's version number and exit
  --strict              Whether tests should be run in strict mode. In this
                        mode a test that does not check each line of output is
                        considered to have failed

Example test:

# in my_test.scf
# the following line is a test description
Verify 'cat' reads file contents

# Run some commands to setup environment
$ echo 'some text' > test.txt
$ echo '1234' >> test.txt

# The command to test (note indentation)
    $ cat test.txt
    some text
    [RE] ^\d+$

Running:

scruf my_test.scf
# Testing: my_test.scf
1..1
ok 1 - Verify 'cat' reads file contents

Test Structure

This is a summary of the structure of tests expected by scruf, these tests contain:

• A optional description lines, followed by

• Optional setup command to run, followed by

• A test test command to run, followed by

• Optional output specification to compare against the result of running the command.

There are examples, run as part of the test suite, present under examples/

Comments

Any line beginning with a # is treated as a comment, and is ignored.

Test Descriptions

Non-comment lines beginning with anything other than a space of $ are interpreted as descriptions. These are used as descriptions in the test output

For example:

# This is a comment
This would be be a test description

Setup Command

Lines beginning with a $ are interpreted as setup commands, these are commands to be run before the actual test and whose output is not used as part of a test. If one of these commands fails (i.e. returns a non-zero value) the test is marked as failed. e.g.:

# This is a comment
This would be a test description
# Add some content to 'my_file'
$ echo 'some content' > my_file

Test Commands

Commands are specified by a line indented (by default with 4 spaces, but this can be configured with the --indent option) followed by $. Building on our example:

# This is a comment
This would be a test description
# Add some content to 'my_file'
$ echo 'some content' > my_file
     $ cat my_file

Commands can continued over more than one lines, such continuations are marked by an indented line beginning with >, e.g.:

Concatentate two files
    $ cat first_file
    > second_file

This would be equivalent to:

Concatenate two files
    $ cat first_file second_file

Testing Output

Output is defined by a indented line beginning with anything other than $ or >. The simplest comparison that can be made on output is a direct comparison on contents, our complete test is then:

# This is a comment
This would be a test description
# Add some content to 'my_file'
$ echo 'some content' > my_file
     $ cat my_file
     some content

Stream Specification

The output stream, standard out or standard error, can be specified by prefixing an output line with 1: or 2: respectively, for example:

'printf' prints to stdout
    $ printf "Off I go\n"
    1: Off I go

Output can be redirected
    $ printf "Error!\n" >&2
    2: Error!

Regex Comparisons

Output can be compared against a provided regex using the flag: [RE], for example:

Printing numbers
    $ echo '1234'
    [RE] ^\d+$

To combine this with stream specification simply specify the stream before the regex flag:

Numbers to stderr
    $ echo '1234' >&2
    2:[RE] ^\d+$

Comparisons Without End-Of-Line Characters

An output line without a line ending can be tested using the No End of Line flag: [NEoL], for example:

'printf' doesn't add newlines
    $ printf "Hello, world"
    [NEoL] Hello, world

Exit Code Comparisons

Line’s which contain only an integer contained in [] are used to test the exit code of a command, for example:

# Intentionally disregard output
'echo' exits with 0 on success
    $ echo "Everything is ok"
    [0]

'exit' sets the exit code
    $ exit 1
    [1]

Combining Comparisons

Any combination of comparisons can be used within a single test:

Printf with varied output
    $ printf "Lots of interesting output\n12345\nIn this test"
    1: Lots of interesting output
    [RE] ^[0-9]+$
    [NEoL] In this test
    [0]

Test Environment

Environment variables can be passed when testing either by passing them down:

$ FOO=12 BAR=13 scruf some_test.scf

Or by storing them in a config file under they key: [Scruf Env]

# in test.cfg
[Scruf Env]
FOO=12
BAR=13

Then calling scruf with -e/--env-file:

$ scruf --env-file my.cfg some_test.scf

Strict Testing

By default scruf ignores any lines remaining in the output once a test is finished. If you would like to ensure all lines of output have been tested (e.g. to verify there are no more lines of output than expected) you can pass the --strict flag. In strict mode any test that doesn’t explicitly test each line of output is marked as a failure.

Scruf Output

By default scruf will output results following TAP (Test Anything Protocol). Other formats, e.g. JUnit should be coming in the future.