repomate-junit4, a JUnit 4.12 test runner plugin for repomate

This is a plugin for repomate that runs JUnit4 test classes on production classes in a cloned student repo.

Requirements

repomate-junit4 has a few non-Python dependencies.

1. java must ba available from the command line.
2. javac must be available from the command line.
   • In other words, install a JDK version that is compatible with the files you intend to test!
3. junit-4.12.jar must be available on the CLASSPATH variable, or configured (see Added CLI arguments and Configuration file).
4. hamcrest-core-1.3.jar must be available on the CLASSPATH variable or configured in order to make use of hamcrest matchers.

The hamcrest and junit jars ar available from Maven Central:

wget http://central.maven.org/maven2/junit/junit/4.12/junit-4.12.jar
wget http://central.maven.org/maven2/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar

Install

The recommended way to install repomate-junit4 is with pip.

python3 -m pip install --user repomate-junit4

The plugin itself does not actually require repomate, but it is fairly useless without. If repomate and repomate-junit4 are both installed in the same environment, then repomate should just find repomate-junit4. For repomate to actually use repomate-junit4, it must be configured in the repomate configuration file. Refer to the repomate docs for information on the configuration file and its expected location, and the Configuration file section here for info on what you need to add to it.

Usage

Terminology and conventions

This is terminology added in excess to that which is defined in the repomate docs. For brevity, some conventions expected by repomate-junit4 are baked into these definitions.

• Production class: A Java file/class written in the student repo.
• Test file: A file ending in Test.java which contains a test class for some production class. If the students are supposed to write a file called LinkedList.java, the corresponding test class must be called LinkedListTest.java.
• Test dir: A directory named after a master repo, containing tests for student repos based on that master repo. Should contain test files as defined above.
• Reference tests directory (RTD): A local directory containing subdirectories with reference tests. Each subdirectory should be a test dir as defined above.

Security aspects

There are some inconvenient security implications to running untrusted code on your own computer. repomate-junit4 tries to limit what a student's code can do by running with a very strict JVM Security Policy. This is enforced by the Java SecurityManager. The policy used looks like this:

// empty grant to strip all permissions from all codebases
grant {
};

// the `junit-4.12.jar` needs this permission for introspection
grant codeBase "file:{junit4_jar_path}" {{
    permission java.lang.RuntimePermission "accessDeclaredMembers";
}};
"""

This policy disallows student code from doing most illicit things, such as accessing files outside of the codebases's directory, or accessing the network. The {junit4_jar_path} is dynamically resolved during runtime, and will lend the actual junit-4.12.jar archive that is used to run the test classes sufficient permissions to do so.

This policy seems to work well for introductory courses in Java, but there may be snags because of how restrictive it is. If you find that some permission should definitely be added, please open an issue about it There are plans to add the ability to specify a custom security policy, but currently, your only choice is to either use this default policy or disable it with --disable-security.

Important: The security policy relies on the correctness of the Java SecurityManager. It is probably not bulletproof, so if you have strict security requirements, you should probably only run this plugin inside of a properly secured environment (for example, a virtual machine).

Added CLI arguments

repomate-junit4 adds four new CLI arguments to the repomate clone command.

• -rtd|--reference-tests-dir
  • Path to the RTD.
  • Required unless specified in the configuration file.
• -junit|--junit-path
  • Path to the junit-4.12.jar library.
  • Required unless specified on the CLASSPATH variable, or in the configuration file.
• -ham|--hamcrest-path
  • Path to the hamcrest-core-1.3.jar library.
  • Required unless specified on the CLASSPATH variable, or in the configuration file.
• -i|--ignore-tests
  • A whitespace separated list of test files (e.g. LinkedListTest.java) to ignore. This is useful for example if there are abstract test classes in the test dir.
• --disable-security
  • Disable the seurity policy.
• -v|--verbose
  • Display more verbose information (currently only concerns test failures).
  • Long lines are truncated.
• -vv|--very-verbose
  • Same as -v, but without truncation.

Configuration file

First and foremost, junit4 must be added to the plugins option under the [DEFAULTS] section in order to activate the plugin, see details here. The --hamcrest-path, --junit-path and --reference-tests-dir arguments can be configured in the standard repomate configuration file by adding the [junit4] section heading. Example:

[DEFAULTS]
plugins = junit4

[junit4]
reference_tests_dir = /absolute/path/to/rtd
junit_path = /absolute/path/to/junit-4.12.jar
hamcrest_path = /absolute/path/to/hamcrest-core-1.3.jar

Important: All of the paths in the config must be absolute for repomate-junit4 to behave as expected.