Skip to main content

Cram is a functional testing framework for command line applications. Cram tests look like snippets of interactive shell sessions. Cram runs each command and compares the command output in the test with the command’s actual output.

Here’s a snippet from Cram’s own test suite:

The $PYTHON environment variable should be set when running this test
from Python.

  $ [ -n "$PYTHON" ] || PYTHON="`which python`"
  $ [ -n "$PYTHONPATH" ] || PYTHONPATH="$TESTDIR/.." && export PYTHONPATH
  $ if [ -n "$COVERAGE" ]; then
  >   coverage erase
  >   alias cram="`which coverage` run --branch -a $TESTDIR/../scripts/cram"
  > else
  >   alias cram="$PYTHON $TESTDIR/../scripts/cram"
  > fi
  $ command -v md5 > /dev/null || alias md5=md5sum

Usage:

  $ cram -h
  [Uu]sage: cram \[OPTIONS\] TESTS\.\.\. (re)

  [Oo]ptions: (re)
    -h, --help          show this help message and exit
    -V, --version       show version information and exit
    -q, --quiet         don't print diffs
    -v, --verbose       show filenames and test status
    -i, --interactive   interactively merge changed test output
    -d, --debug         write script output directly to the terminal
    -y, --yes           answer yes to all questions
    -n, --no            answer no to all questions
    -E, --preserve-env  don't reset common environment variables
    --keep-tmpdir       keep temporary directories
    --shell=PATH        shell to use for running tests (default: /bin/sh)
    --shell-opts=OPTS   arguments to invoke shell with
    --indent=NUM        number of spaces to use for indentation (default: 2)
    --xunit-file=PATH   path to write xUnit XML output

The format in a nutshell:

  • Cram tests use the .t file extension.

  • Lines beginning with two spaces, a dollar sign, and a space are run in the shell.

  • Lines beginning with two spaces, a greater than sign, and a space allow multi-line commands.

  • All other lines beginning with two spaces are considered command output.

  • Output lines ending with a space and the keyword (re) are matched as Perl-compatible regular expressions.

  • Lines ending with a space and the keyword (glob) are matched with a glob-like syntax. The only special characters supported are * and ?. Both characters can be escaped using \, and the backslash can be escaped itself.

  • Output lines ending with either of the above keywords are always first matched literally with actual command output.

  • Lines ending with a space and the keyword (no-eol) will match actual output that doesn’t end in a newline.

  • Actual output lines containing unprintable characters are escaped and suffixed with a space and the keyword (esc). Lines matching unprintable output must also contain the keyword.

  • Anything else is a comment.

Download

Installation

Install Cram using make:

$ wget https://bitheap.org/cram/cram-0.7.tar.gz
$ tar zxvf cram-0.7.tar.gz
$ cd cram-0.7
$ make install

Usage

Cram will print a dot for each passing test. If a test fails, a unified context diff is printed showing the test’s expected output and the actual output. Skipped tests (empty tests and tests that exit with return code 80) are marked with s instead of a dot.

For example, if we run Cram on its own example tests:

.s.!
--- examples/fail.t
+++ examples/fail.t.err
@@ -3,21 +3,22 @@
   $ echo 1
   1
   $ echo 1
-  2
+  1
   $ echo 1
   1

 Invalid regex:

   $ echo 1
-  +++ (re)
+  1

 Offset regular expression:

   $ printf 'foo\nbar\nbaz\n\n1\nA\n@\n'
   foo
+  bar
   baz

   \d (re)
   [A-Z] (re)
-  #
+  @
s.
# Ran 6 tests, 2 skipped, 1 failed.

Cram will also write the test with its actual output to examples/fail.t.err, allowing you to use other diff tools. This file is automatically removed the next time the test passes.

When you’re first writing a test, you might just write the commands and run the test to see what happens. If you run Cram with -i or --interactive, you’ll be prompted to merge the actual output back into the test. This makes it easy to quickly prototype new tests.

You can specify a default set of options by creating a .cramrc file. For example:

[cram]
verbose = True
indent = 4

Is the same as invoking Cram with --verbose and --indent=4.

To change what configuration file Cram loads, you can set the CRAMRC environment variable. You can also specify command line options in the CRAM environment variable.

Note that the following environment variables are reset before tests are run:

  • TMPDIR, TEMP, and TMP are set to the test runner’s tmp directory.

  • LANG, LC_ALL, and LANGUAGE are set to C.

  • TZ is set to GMT.

  • COLUMNS is set to 80. (Note: When using --shell=zsh, this cannot be reset. It will reflect the actual terminal’s width.)

  • CDPATH and GREP_OPTIONS are set to an empty string.

Cram also provides the following environment variables to tests:

  • CRAMTMP, set to the test runner’s temporary directory.

  • TESTDIR, set to the directory containing the test file.

  • TESTFILE, set to the basename of the current test file.

  • TESTSHELL, set to the value specified by --shell.

Also note that care should be taken with commands that close the test shell’s stdin. For example, if you’re trying to invoke ssh in a test, try adding the -n option to prevent it from closing stdin. Similarly, if you invoke a daemon process that inherits stdout and fails to close it, it may cause Cram to hang while waiting for the test shell’s stdout to be fully closed.

Development

Download the official development repository using Mercurial:

hg clone https://bitbucket.org/brodie/cram

Or Git:

git clone https://github.com/brodie/cram.git

Test Cram using Cram:

pip install -r requirements.txt
make test

Visit Bitbucket or GitHub if you’d like to fork the project, watch for new changes, or report issues.

Release files for cram 0.7

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for cram 0.7
File Size Uploaded
cram-0.7.tar.gz 33.5 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for cram 0.7
File Interpreter ABI Platform
cram-0.7-py2.py3-none-any.whl Python 3, Python 2 none any Details

Total release size: 55.7 kB

Release files / cram-0.7.tar.gz

Download URL cram-0.7.tar.gz
Size 33.5 kB
Tags Source
SHA-256 checksum
How to use checksums
7da7445af2ce15b90aad5ec4792f857cef5786d71f14377e9eb994d8b8337f2f
BLAKE2b-256 checksum
How to use checksums
38855a8a3397b2ccb2ffa3ba871f76a4d72c16531e43d0e58fc89a0f2983adbd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Release files / cram-0.7-py2.py3-none-any.whl

Download URL cram-0.7-py2.py3-none-any.whl
Size 22.1 kB
Tags Python 2 Python 3
SHA-256 checksum
How to use checksums
008e4e8b4d325cf040964b5f62460535b004a7bc816d54f8527a4d299edfe4a3
BLAKE2b-256 checksum
How to use checksums
a209c119252f12b35c21b030c01d38d292936660c167e725ce07517fae2b01a0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Release history Release notifications | RSS feed

This release

0.7 This release

2 release files

0.6

2 release files

0.5

1 release file

0.4

1 release file

0.3

1 release file

0.2

1 release file

0.1

1 release file

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page