testy-command-line 0.8.1

A Testing Script/Module for Command-line Programs

Table of Contents

1. Testy Synopsis
   1. Rationale
   2. Non-Features
   3. Basic Usage
   4. Example Run
2. User Guide
   1. Terminology
   2. Test File Format
   3. Specifying Tests to Run and Showing Failure Results
   4. Running Tests Sequentially and in Parallel
   5. Multi-Segment Tests
   6. Markdown Output Format
   7. Options for Tests
   8. Other Test File Formats
   9. Generating Expected Test Output from Programs
3. License
4. Planned and Completed Additions

Testy Synopsis

A script to run tests for interactive terminal programs. Tests are specified in Emacs Org-like files.

The project homepage is on Github here: https://github.com/kauffman77/testy

Rationale

I teach university courses which do a fair bit of C programming (and some assembly). There isn't a standard testing infrastructure for C programs that is suitable for education so testy is my attempt to do so. The program strives for the following goals:

• Write tests in plain text files that focus on input/output behavior and provide a place document the intent and purpose of individual tests.
• Specify program tests that require interactive input in a fashion similar to how an interactive session would appear.
• Make it easy to use Valgrind to check for memory errors as the program runs and trigger a failure if things go wrong.
• Present test results in an accessible fashion showing output differences and Valgrind traces to speed debugging.
• Provide some flexibility on tests such as specifying timeout options, assigning varying points to tests, and running tests in parallel to speed up the results.
• The testing infrastructure is self-contained in a single file so can be "installed" by copying the testy script where it is needed. Rely primarily on the Python standard library, not on 3rd party packages that require elaborate installs.

It is general purpose and I've used it to help test C, Assembly, Rust, Makefiles, and even some MPI programs at various stages.

Non-Features

Testy aims to provide a convenient wrapper/infrastructure around some sort of Unit testing framework. It tests whole program runs and allows easy checking of the program output against expected errors and detection of memory problems in programs. For those in education settings it also provides a means to assign points to passing tests.

The following are not goals for it Testy so are not features.

• It does not make writing unit or integration tests easier. To test individual C functions, I typically write a separate C files that is compiled with the code it will test and use Testy as a way to run the individual or batches of those tests. Using testy to wrap around a framework like JUnit for Java, OUnit for OCaml, etc. is straight-forward BUT you can also just write some simple input/output tests and use Testy to more cleanly document and summarize results.
• It is not a build system. While one could compile and run programs with the Testy, I use Makefiles extensively to first build and then run tests. Usually a make test will first compile code and then invoke testy testfile.org; if the build fails, no tests are run.

Basic Usage

Below is the basic usage on the command line when a file called testsfile.org is available.

usage:    testy <testfile.org> [test# test# ...]
          testy --help

>> testy testsfile.org              # runs all tests in file
>> testy testsfile.org 3 5 7        # runs tests 3,5,7 in file
>> testy testsfile.org 5            # runs only test 5 and shows failures to stdout
>> testy -o md testsfile.org        # generate the results in Markdown format instead of Org
>> SHOW=fail testy testsfile.org    # runs tests and prints all failures to stdout

Example Run

Running a test is done from the command line and will default to running all tests in a provided test file. Output shows each test with a pass/fail and failures have results files indicating what went wrong. Below is an example from the examples/ directory:

>> cd examples/
>> ../testy bash_tests.org
=================
== bash_tests.org
== Running 2 / 2 tests
1) Output Tests : ok
2) Failure Demo : FAIL -> see test-results/test-02-result.org
=================
RESULTS: 1 / 2 tests passed

Inspecting the failure file indicated under the freshly created directory test-results/ shows the following output (plain text but easier easier to read in emacs org-mode):

* (TEST 2) Failure Demo : FAIL
** COMMENTS
This test will fail and produce output associated to show the
side-by-side diff that primarily reports failures.

** PROGRAM: bash -v
To run this individual test in GDB use the command:
  gdb --args bash -v
but any input to the program must be typed within the debugger

** FAILURE MESSAGES
- Output Differenes: Expected/Actual do not match, check Diff Sections for details

** SIDE-BY-SIDE DIFF of Expected vs Actual
. lines match; | lines differ; < expected line missing; > extra line in actual

#+BEGIN_SRC sdiff
===EXPECT===                            ===ACTUAL===
>> echo "Matching Line"               . >> echo "Matching Line"
Matching Line                         . Matching Line
>> echo "Mismatching Line"            . >> echo "Mismatching Line"
Misma______ Li__                      | Mismatching Line
>> echo "Extra line in ACTUAL"        . >> echo "Extra line in ACTUAL"
>> echo "Extra line in EXPECT"        | Extra line in ACTUAL
This is the extra line                | >> echo "Extra line in EXPECT"
Extra line in EXPECT                  . Extra line in EXPECT
>> printf "Matches fine\nAnd again\n" . >> printf "Matches fine\nAnd again\n"
Matches fine                          . Matches fine
And again                             . And again

#+END_SRC

** LINE-BY-LINE DIFF of Expected vs Actual
#+BEGIN_SRC text
EXPECT   4) Misma______ Li__
ACTUAL   4) Mismatching Line

EXPECT   6) >> echo "Extra line in EXPECT"
ACTUAL   6) Extra line in ACTUAL

EXPECT   7) This is the extra line
ACTUAL   7) >> echo "Extra line in EXPECT"

#+END_SRC

** VALGRIND Not in Use
** SUMMARY
Test FAILED for the following reasons
- Output Differenes: Expected/Actual do not match, check Diff Sections for details

User Guide

Terminology

I'm no expert on software testing theory so I don't expect these terms to be universal but they are the ones I settled on for testy. The source code has a class hierarchy that reflects theses terms.

• Session: program input/output
  • a run of a program with input and expected output
  • appear in .org test files as #+BEGIN_SRC: / #+END_SRC:
• Segment: a single session and options/description
  • combination of some description, options, and a single session
  • appear in .org files as some descriptive text, #+TESTY: directives, then a session
  • some segments are separated using sub headings like ** Segment Title
• Test: one or more segments and options/description
  • a sequence of segments (possibly only 1) run in order up to the first failure
  • appear in .org files as top-level headings like * Test Title
• Suite: one or more tests and global options
  • a collection of tests specified in a single file; may have global options for all tests
  • appear as .org files with global options specified at the top of the file

Test File Format

Tests are specified in org-like files. This is to make the test-writing experience akin to writing a text file and encourage documenting the intent and purpose of the tests.

Org files are similar to Markdown but have a wider set of uses and deep support in Emacs (this document is also written as a Org file). The basic structure of Org files is an outline with each heading denoting a test.

• Headings start with a line that looks like

  * Test Title Here
  

  with the * character denoting a top-level heading

• Text that appears after the title line is commentary on the test

• Tags or "directives" in org-mode start with #+. The most important tag in the format is the #+BEGIN_SRC and #+END_SRC pair which denote a test session that shows prompts with input and expected output together.

• The other common tag is the #+TESTY: tag which specifies options / directives for tests like the program invocation to run (global or local to a test), the prompt used, timeouts, whether to use Valgrind to check for memory problems, how to name test results files, etc.

• Comments in org files look like

  # this is a comment line
  

  which is a "hash space" at the beginning of the line. Comments will not affect tests nor appear in any results files.

A good example of this structure is in the <examples_bc_tests_small.md> file which has two tests along with descriptive comments in it. Below are the contents of that file.

#+TITLE: Tests of the bc program
# the title to display when running the tests

# the lines below set some global options for all tests which may be
# overridden in invididual tests.

#+TESTY: PREFIX="bctests" 
# a prefix for the results files that will appear in the test-results/
# directory; the default prefix is "test" but when multiple test files
# are present such as for multiple problems, it's handy to distinguis
# them. 

#+TESTY: PROGRAM="bc -iq" 
# the default program to run, in this case the standard interactive
# calculator program "bc"; the -iq options force an interactive
# setting (-i) and silence the welcome message copyright when starting
# the program (-q).

* (FIRST TEST) Addition and Multiplication 
# The above line indicates the start of a test with its title.

Some add/multiply tests
# This line is a comment on the intent of the test.

# Below is a "session" which will run the program `bc -iq` and feed in
# the input given on ">>" lines and check that the output matches the
# other lines. The "text" designator has no effect in testy and can be
# left off or chosen to make Emacs Org-Mode display code blocks with
# syntax highlighting.

#+BEGIN_SRC text
>> 1+1
2
>> 3+4
7
>> 9*2+3
21
#+END_SRC

# Below is a second test with similar features to the first.

* (SECOND TEST) No -q option; likely fail
# test title above and comments below

The 'program' for this test is changed to ~bc -i~; since the ~-q~
option is omitted, the startup is not "quiet" and so the tests output
should include the startup message for ~bc~. This test will fail
unless you just happen to have the exact version of bc reported below.

#+TESTY: program="bc -i"
# This line overrides the program to run; instead of `bc -iq`, the
# above program will run which shows the welcome message. The test
# session is below.

#+BEGIN_SRC sh
bc 1.07.1
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006, 2008, 2012-2017 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'. 
>> 1-1
0
>> 6-3
3
>> 9-3
6
>> 10-8
2
#+END_SRC

Below is how that code renders in Emacs with my selection of fonts/colors for syntax highlighting. NOTE: When viewing on Github, keep in mind that the site will render .org files as HTML so headings will stand out and comments may be omitted. Looking at the raw version gives the full picture for the sample test files.

Specifying Tests to Run and Showing Failure Results

If only a subset of tests is to be run, these can be passed by number as additional command line arguments.

>> ../testy bc_tests_full.org                   # run all tests
==============================================
== bc_tests_full.org : Tests of the bc program
== Running 6 / 6 tests
1) Addition and Multiplication           : ok
2) Subtraction, will fail                : FAIL -> see test-results/bc-02-result.org
3) Combined                              : ok
4) No -q option                          : FAIL -> see test-results/bc-04-result.org
5) Test output includes input, will fail : FAIL -> see test-results/bc-05-result.org
6) bash tests for bc                     : ok
==============================================
RESULTS: 3 / 6 tests passed

>> ../testy bc_tests_full.org 2 4 6             # run only tests 2 4 6
==============================================
== bc_tests_full.org : Tests of the bc program
== Running 3 / 6 tests
2) Subtraction, will fail : FAIL -> see test-results/bc-02-result.org
4) No -q option           : FAIL -> see test-results/bc-04-result.org
6) bash tests for bc      : ok
==============================================
RESULTS: 1 / 3 tests passed

A common activity is to just run a single test due to trying resolve a failure. The default when running a single test is to print out the failure results in the console. This can be disabled by running with show=none.

>> ../testy bc_tests_full.org 1                 # run a single test which passes
==============================================
== bc_tests_full.org : Tests of the bc program
== Running 1 / 6 tests
1) Addition and Multiplication : ok
==============================================
RESULTS: 1 / 1 tests passed

>> ../testy bc_tests_full.org 2                 # run a single test which fails
==============================================
== bc_tests_full.org : Tests of the bc program
== Running 1 / 6 tests
2) Subtraction, will fail : FAIL -> see test-results/bc-02-result.org
==============================================
RESULTS: 0 / 1 tests passed

---- Single Test Failed -----                   # info on the failed test is shown
* (TEST 2) Subtraction, will fail : FAIL
** COMMENTS
Some subtraction tests.  This test will fail due to a typo in the
expected output where ~9-3~ is reported as 5.

** PROGRAM: bc -iq
To run this individual test in GDB use the command:
  gdb --args bc -iq
but any input to the program must be typed within the debugger

** FAILURE MESSAGES
- Output Differenes: Expected/Actual do not match, check Diff Sections for details

** SIDE-BY-SIDE DIFF of Expected vs Actual
. lines match; | lines differ; < expected line missing; > extra line in actual

#+BEGIN_SRC sdiff
===EXPECT===   ===ACTUAL===
>> 1-1       . >> 1-1
0            . 0
>> 6-3       . >> 6-3
3            . 3
>> 9-3       . >> 9-3
5            | 6
>> 10-8      . >> 10-8
2            . 2

#+END_SRC

** LINE-BY-LINE DIFF of Expected vs Actual
#+BEGIN_SRC text
EXPECT   6) 5
ACTUAL   6) 6

#+END_SRC

** VALGRIND Not in Use
** SUMMARY
Test FAILED for the following reasons
- Output Differenes: Expected/Actual do not match, check Diff Sections for details

>> show=none ../testy bc_tests_full.org 2       # run a single test without showing the results
==============================================
== bc_tests_full.org : Tests of the bc program
== Running 1 / 6 tests
2) Subtraction, will fail : FAIL -> see test-results/bc-02-result.org
==============================================
RESULTS: 0 / 1 tests passed

Running Tests Sequentially and in Parallel

By default tests are run sequentially in the order that they appear on the command line (if numbers are indicated) or in the order they appear in the test file. The default behavior is to run "serially" using a single processor/core.

Tests can be run in parallel on multiple cores by setting the PARALLEL environment variable to an appropriate value..

>> PARALLEL=False testy tests.org  # run serially, single core
>> PARALLEL=True  testy tests.org  # run with max cores reported by OS
>> PARALLEL=max   testy tests.org  # same as above
>> PARALLEL=2     testy tests.org  # run with 2 cores
>> export PARALLEL=4               # set environment variable in bash
>> testy tests.org                 # run with 4 cores as per environment variable

Internally, testy uses the standard Python multiproc library to run tests in parallel for true parallelism of test runs (not that green, faux parallelism of the threads package). This means running tests in parallel should speed up considerably and benefit from multiple cores though there might be slightly higher memory utilization as the python process is fork()'d to get the parallelism.

When writing tests, it's best practice NOT to have dependencies between then that require a specific order of tests. If a test requires several steps with distinct runs/sessions, write it as a single test possibly employing 2.5 as way to get the sequencing. This will prevent problems when running in parallel. The segments of a test are always run in sequence from beginning to end.

Multi-Segment Tests

Each test can have multiple segments; each segment is a description, some options, and a test session for a program run. Segments are run in order and if a segment fails, the test terminates in failure and subsequent segments for that test are not run.

A demonstrative example for this is the file <examples/multi-segment-tests.md> which shows several examples of how each test can be a sequence of segments each with its own program session. The first test is shown below and shows how to include multiple segments that will be run in sequence in the test.

* Two Segment Test, Passing

This is the FIRST SEGMENT which uses BASH to create some files.

#+BEGIN_SRC sh
>> echo 'Creating fileA'
Creating fileA
>> echo 'Hello world' >  test-results/fileA.txt
>> echo 'Goodbye now' >> test-results/fileA.txt
>> echo 'Creating fileB'
Creating fileB
>> seq 10 > test-results/fileB.txt
>> echo 'Done'
Done
#+END_SRC

This is the SECOND SEGMENT which uses BASH to counts words in the
files created in the first segment. If for some reason the first
segment fails, the subsequent segment won't run. This test should have
all segments complete and thus the test will pass.

#+BEGIN_SRC sh
>> echo 'Counting fileA'
Counting fileA
>> wc test-results/fileA.txt
 2 4 24 test-results/fileA.txt
>> echo 'Counting fileB'
Counting fileB
>> wc test-results/fileB.txt
10 10 21 test-results/fileB.txt
>> echo 'Counting both files'
Counting both files
>> wc test-results/file[AB].txt
 2  4 24 test-results/fileA.txt
10 10 21 test-results/fileB.txt
12 14 45 total
#+END_SRC

The multi-segment example file has additional details in it including:

• demo of a multi-segment test which fails midway
• organization of test segments via org sub-headings
• using different programs in different segments via directives

Markdown Output Format

Many folks are more inclined towards the (lesser) Markdown format for output rather than the default Org format. Markdown output is enabled via -o md on the command line and will generate .md files rather than .org files.

>> ../testy -o md bash_tests.org                # run tests with md-formatted results files
=================
== bash_tests.org
== Running 2 / 2 tests
1) Output Tests : ok
2) Failure Demo : FAIL -> see test-results/test-02-result.md
=================
RESULTS: 1 / 2 tests passed

>> cat test-results/test-02-result.md           # show the results file which is in markdown format
(TEST 2) Failure Demo : FAIL
============================

COMMENTS
--------
This test will fail and produce output associated to show the
side-by-side diff that primarily reports failures.

PROGRAM: bash -v
----------------
To run this individual test in GDB use the command:
  gdb --args bash -v
but any input to the program must be typed within the debugger

FAILURE MESSAGES
----------------
- Output Differenes: Expected/Actual do not match, check Diff Sections for details

SIDE-BY-SIDE DIFF of Expected vs Actual
---------------------------------------
. lines match; | lines differ; < expected line missing; > extra line in actual

```sdiff
===EXPECT===                            ===ACTUAL===
>> echo "Matching Line"               . >> echo "Matching Line"
Matching Line                         . Matching Line
>> echo "Mismatching Line"            . >> echo "Mismatching Line"
Misma______ Li__                      | Mismatching Line
>> echo "Extra line in ACTUAL"        . >> echo "Extra line in ACTUAL"
>> echo "Extra line in EXPECT"        | Extra line in ACTUAL
This is the extra line                | >> echo "Extra line in EXPECT"
Extra line in EXPECT                  . Extra line in EXPECT
>> printf "Matches fine\nAnd again\n" . >> printf "Matches fine\nAnd again\n"
Matches fine                          . Matches fine
And again                             . And again

```

LINE-BY-LINE DIFF of Expected vs Actual
---------------------------------------
```
EXPECT   4) Misma______ Li__
ACTUAL   4) Mismatching Line

EXPECT   6) >> echo "Extra line in EXPECT"
ACTUAL   6) Extra line in ACTUAL

EXPECT   7) This is the extra line
ACTUAL   7) >> echo "Extra line in EXPECT"

```

VALGRIND Not in Use
-------------------
SUMMARY
-------
Test FAILED for the following reasons
- Output Differenes: Expected/Actual do not match, check Diff Sections for details

Options for Tests

There are variety of options that can be placed in test files that set the default for the entire suite or for an individual test or segment. The table below surveys these. There are others that are possible and the general philosophy is to make most internal parts of the Suite, Test, and Segment available as tweak able options through #+TESTY: directives.

SYNTAX / DEFAULT	EFFECT	SCOPE
GENERAL OPTIONS	see examples/options.html
#+TESTY: program='bash -v'	set the program to run in a session	Suite,Test,Segment
#+TESTY: prompt = ">>"	set the prompt for interactive programs	Suite,Test,Segment
#+TESTY: timeout=5.0	set maximum seconds before a session fails	Suite,Test,Segment
#+TESTY: max_out_bytes=2**20	set maximum bytes of output before session fails	Suite,Test,Segment
#+TESTY: exitcode_expect=0	change the expected exit code for sessions	Suite,Test,Segment
#+TESTY: skip_exitcode=False	skip checking the exit code / accept any exit code	Suite,Test,Segment
#+TESTY: skip_diff=False	skip checking that output matches some expectation	Suite,Test,Segment
#+TESTY: !rm somefile.txt	run precommands (shell one-liners) to do setup for a Segment	Segment
+#BEGIN_QUOTE filename.txt	creating files with content to be used during testing	Suite,Test,Segment
#+TESTY: use_valgrind=1	Use Valgrind to check for memory problems	Suite,Test,Segment
#+TESTY: valgrind_opts="--option"	set additional Valgrind options	Suite,Test,Segment
#+TESTY: post_filter='filtprog'	filter/transform session output before checking it	Suite,Test,Segment
#+TESTY: skip_diff=1	skip diffing the output; test succeeds irrespective of output	Suite,Test,Segment
#+TESTY: diff_ignore_blanklines=True	ignore blank lines when diffing output	Suite,Test,Segment
#+TESTY: diff_ignore_whitespace=True	treat one space the same as many spaces in a line during diffs	Suite,Test,Segment
#+TESTY: diff_ignore_trail_ws=True	ignore trailing spaces in output	Suite,Test,Segment
* COMMENT This test won't count	tests that have COMMENT are ignored and don't count	Test
SAVE_RAWFILES=1 testy tests.org	save raw input/output in the test-results/raw directory	Suite
#+TESTY: results_dir="test-results"	specify the directory suite results are in	Suite
POINTS SYSTEM	see examples/points.html
#+TESTY: use_points=False	report points earned instead of tests passed when True	Suite
#+TESTY: points_scale=1.0	multiply total points / earned points by this factor	Suite
#+TESTY: points=1.0	raw points for passing an individual test	Test

Other Test File Formats

Org-format test files are the only ones supported at the moment. However, in the not-to-distant future the following formats are likely to be added for support.

• Markdown test files as input
• JSON test files as input
• Other light, structured input like YAML or TOML

If you want to advocate for work on one of these, let me know. While Org files are convenient to write, parsing them is a bear as they are less structured. Markdown will be similar as it is fairly free-form. The structured inpu but JSON likely has an easy

Generating Expected Test Output from Programs

A common occurrence for me as a programming teacher is that I'll want to base the expected results for tests on the output produced by a solution program I've written. This is not fool proof as any bugs in my code will become the expectation BUT it's a lot faster than hand-writing expected output and bugs in tests ca be mitigated by patching the tests. AND tests can then be generated from the solution program in a fairly automated fashion.

1. Start with some test stubs which contain the program invocation for sessions and any input the session requires. An example of this is in <examples/test-stubs.md> which is used below
2. Run the tests with --regen file.org as a command line option to generate file.org. The test structure, options, and description will be taken from the original file but the session output will be drawn from the actual output of the program.
3. The freshly created file.org will have all the tests in it with the expected output generated from the program and therefore pass.

Here is a full example with commentary.

>> cd examples/
>> cat test-stubs.org                                          # examples/test-stubs.org leaves session
#+title: Test Stubs for Regeneration                           # output blank for its two tests

* Seq Test
Below is a program invocation of the standard ~seq~ program but with
an empty session. In the regenerated test, the output will be filled
in.
#+TESTY: program='seq 5 5 25'
#+BEGIN_SRC sh
                                                                # no output here, will be filled in
#+END_SRC


* Doctor Test
Below is only the input for the ~doctor.sh~ program. Regenerating this
test will fill in the expected output in the generated test.

#+TESTY: program='./doctor.sh -echo'
#+TESTY: prompt='DOCTOR>'
#+BEGIN_SRC sh
DOCTOR> I've got feelings of guilt                             # input only here, no output from program
DOCTOR> I don't think I'm giving my all at work
DOCTOR> I just don't feel motivated
DOCTOR> quit
#+END_SRC


>> ../testy --regen test-complete.org test-stubs.org           # regenerate into test-complete.org
===============================================
== test-stubs.org : Test Stubs for Regeneration
== Running 2 / 2 tests
1) Seq Test    : FAIL -> see test-results/test-01-result.org
2) Doctor Test : FAIL -> see test-results/test-02-result.org
===============================================
RESULTS: 0 / 2 tests passed
Regenerating test suite in file test-complete.org

>> cat test-complete.org                                       # show contents of test-complete.org
#+title: Test Stubs for Regeneration

* Seq Test
Below is a program invocation of the standard ~seq~ program but with
an empty session. In the regenerated test, the output will be filled
in.
#+TESTY: program='seq 5 5 25'
#+BEGIN_SRC sh
5                                                              # test results have been filled in
10                                                             # with the output of the program
15
20
25
#+END_SRC

* Doctor Test
Below is only the input for the ~doctor.sh~ program. Regenerating this
test will fill in the expected output in the generated test.

#+TESTY: program='./doctor.sh -echo'
#+TESTY: prompt='DOCTOR>'
#+BEGIN_SRC sh
What brings you to the socratic therapist today?               # output from the actual program
DOCTOR> I've got feelings of guilt                             # original input
Tell me more about that
DOCTOR> I don't think I'm giving my all at work
Tell me more about that
DOCTOR> I just don't feel motivated
Tell me more about that
DOCTOR> quit

Oh, that's time. We'll pick up on that next week.
#+END_SRC

>> ../testy test-complete.org                                  # testing the regenerated tests passes
==================================================
== test-complete.org : Test Stubs for Regeneration
== Running 2 / 2 tests
1) Seq Test    : ok
2) Doctor Test : ok
==================================================
RESULTS: 2 / 2 tests passed

License

testy is released under the terms of the GNU General Public License v3.0-or-later (GPLv3-or-later). A copy of the GPLv3-or-later is included in the file LICENSE in the source repository.

Planned and Completed Additions

See <NOTES.txt> which contains notes on planned and completed additions