Skip to main content

A simple testing framework for command line applications

Project description

Cram is a functional testing framework for command line applications based on Mercurial’s unified test format.

Cram tests look like snippets of interactive shell sessions. Cram runs each command and compares the command output in the test with the command’s actual output.

Here’s a snippet from Cram’s own test suite:

The $PYTHON environment variable should be set when running this
test from Python.

  $ [ -n "$PYTHON" ] || PYTHON="`which python`"
  $ if [ -n "$COVERAGE" ]; then
  >   coverage erase
  >   alias cram="`which coverage` run -a $TESTDIR/../"
  > else
  >   alias cram="$PYTHON $TESTDIR/../"
  > fi
  $ command -v md5 > /dev/null || alias md5=md5sum


  $ cram -h
  [Uu]sage: cram \[OPTIONS\] TESTS\.\.\. (re)

  [Oo]ptions: (re)
    -h, --help         show this help message and exit
    -V, --version      show version information and exit
    -q, --quiet        don't print diffs
    -v, --verbose      show filenames and test status
    -i, --interactive  interactively merge changed test output
    -y, --yes          answer yes to all questions
    -n, --no           answer no to all questions
    --keep-tmpdir      keep temporary directories
    -E                 don't reset common environment variables

The format in a nutshell:

  • Cram tests use the .t file extension.

  • Lines beginning with two spaces, a dollar sign, and a space are run in the shell.

  • Lines beginning with two spaces, a greater than sign, and a space allow multi-line commands.

  • All other lines beginning with two spaces are considered command output.

  • Output lines ending with a space and the keyword (re) are matched as Perl-compatible regular expressions.

  • Lines ending with a space and the keyword (glob) are matched with a glob-like syntax. The only special characters supported are * and ?. Both characters can be escaped using \, and the backslash can be escaped itself.

  • Output lines ending with either of the above keywords are always first matched literally with actual command output.

  • Command output in the test that ends with a percent sign will match actual output that doesn’t end in a newline.

  • Anything else is a comment.



You can use pip to install Cram:

$ sudo pip install cram

Or you can install Cram the old fashioned way:

$ wget
$ tar zxvf cram-0.4.tar.gz
$ cd cram-0.4.tar.gz
$ sudo python install


Cram will print a dot for each passing test. If a test fails, a unified context diff is printed showing the test’s expected output and the actual output. Skipped tests (empty tests and tests that exit with return code 80) are marked with s instead of a dot.

For example, if we run Cram on its own example tests:

--- /home/brodie/src/cram/examples/fail.t
+++ /home/brodie/src/cram/examples/fail.t.err
@@ -3,21 +3,22 @@
   $ echo 1
   $ echo 1
-  2
+  1
   $ echo 1

 Invalid regex:

   $ echo 1
-  +++ (re)
+  1

 Offset regular expression:

   $ printf 'foo\nbar\nbaz\n\n1\nA\n@\n'
+  bar

   \d (re)
   [A-Z] (re)
-  #
+  @
# Ran 6 tests, 2 skipped, 1 failed.

Cram will also write the test with its actual output to examples/fail.t.err.

When you’re first writing a test, you might just write the commands and run the test to see what happens. If you run Cram with -i or --interactive, you’ll be prompted to merge the actual output back into the test. This makes it easy to quickly prototype new tests.

Note that the following environment variables are reset before tests are run:

  • TMPDIR, TEMP, and TMP are set to the test runner’s tmp directory.

  • LANG, LC_ALL, and LANGUAGE are set to C.

  • TZ is set to GMT.

  • COLUMNS is set to 80.

  • CDPATH and GREP_OPTIONS are set to an empty string.

Cram also provides the following environment variables to tests:

  • CRAMTMP, set to the test runner’s temporary directory.

  • TESTDIR, set to the directory containing the test file.


Version 0.4 (Sep. 28, 2010)

  • The test format has changed: Output lines containing regular expressions must now end in (re) or they’ll be matched literally. Lines ending with keywords are matched literally first, however.

  • Regular expressions are now matched from beginning to end. In other words \d (re) is matched as ^\d$.

  • In addition to (re), (glob) has been added. It supports *, ?, and escaping both characters (and backslashes) using \.

  • Environment settings have changed: The -D flag has been removed, $TESTDIR is now set to the directory containing the .t file, and $CRAMTMP is set to the test runner’s temporary directory.

  • -i/--interactive now requires patch(1). Instead of .err files replacing .t files during merges, diffs are applied using patch(1). This prevents matching regular expressions and globs from getting clobbered.

  • Previous .err files are now removed when tests pass.

  • Cram now exits with return code 1 if any tests failed.

  • If a test exits with return code 80, it’s considered a skipped a test. This is useful for intentionally disabling tests when they only work on certain platforms or in certain settings.

  • The number of tests, the number of skipped tests, and the number of failed tests are now printed after all tests are finished.

  • Added -q/--quiet to suppress diff output.

  • Added contrib/cram.vim syntax file for Vim. Contributed by Steve Losh.

Version 0.3 (Sep. 20, 2010)

  • Implemented resetting of common environment variables. This behavior can be disabled using the -E flag.

  • Changed the test runner to first make its own overall random temporary directory, make tmp inside of it and set TMPDIR, etc. to its path, and run each test with a random temporary working directory inside of that.

  • Added --keep-tmpdir. Temporary directories are named by test filename (along with a random string).

  • Added -i/--interactive to merge actual output back to into tests interactively.

  • Added ability to match command output not ending in a newline by suffixing output in the test with %.

Version 0.2 (Sep. 19, 2010)

  • Changed the test runner to run tests with a random temporary working directory.

Version 0.1 (Sep. 19, 2010)

  • Initial release.


Download the official development repository using Mercurial:

hg clone

Test Cram using Cram:

make tests

Get a test coverage report using

make coverage

Visit Bitbucket if you’d like to fork the project, watch for new changes, or report issues.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

cram-0.4.tar.gz (20.1 kB view hashes)

Uploaded source

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page