gatorgrader 1.1.0

Automated Grading Tool that Checks the Work of Writers and Programmers

GatorGrader

The only tool you'll need to ensure your student's code and writing is up to speed!

Table of Contents

• Quickstart Guide
• Key Features
• What Do People Think about GatorGrader?
• Installing GatorGrader
• Testing GatorGrader
  • Automated Testing
  • Test Coverage
  • Testing with Multiple Python Versions
  • Code Linting
• Running GatorGrader
• Using Docker
• Comparison to Other Tools
• Presentations
• Contributing
• Contributors

Notes for PyPI Users

This release (1.1.0) of GatorGrader is intended to be a stop-gap solution. It is released from the feature/publish-to-pypi branch on GitHub. In future versions, some changes may be made to the structure and distribution of this package. Additionally, the documentation available in the rest of this document may or may not be fully accurate. For users who just wish to use GatorGrader on the command line, all that is needed is to install this package and run gatorgrader. For help on syntax and available options, see gatorgrader --help.

For users who would like to interact with GatorGrader programmatically, refer to the docstrings available in gatorgrader.py and gator/__init__.py (or view them in your editor-of-choice with support for a Python Language Server). In summary, however, GatorGrader can be run by importing the gator package and calling gator.grader(["list", "--of", "command", "--line", "arguments"]), the syntax of which matches previous versions of GatorGrader. For additional help or guidance, see gatorgrader --help or reach out to any of the core maintainers.

Quickstart Guide

• Starter Repositories An easy way to get started with GatorGrader is to check out our sample starter repositories. The following starter repositories provide examples of how GatorGrader files should be created to check programs and documentation for different languages: Java, Python, LaTex and HTML with CSS. These examples also show how to integrate GatorGrader with GitHub Classroom and Travis CI. When you follow these examples, the Gradle plugin for GatorGrader will install it automatically when you run gradle grade in a terminal window. Please be aware that these repositories are meant to have a majority of red checks. This is only meant to be a “starter” and give an insight into what must be accomplished within the lab.
• Solution Repositories The next step to get involved with GatorGrader is to checkout our sample solution repositories. The following solution repositories provide examples of how GatorGrader files should be created to check programs and documentation for different languages: Java, Python, LaTex , and HTML with CSS. These examples also show how to integrate GatorGrader with GitHub Classroom and Travis CI. When you follow these examples, the Gradle plugin for GatorGrader will install it automatically when you run gradle grade in a terminal window. Please be aware that these repositories are meant to have a majority of green checks. This is meant to show what must be accomplished within a lab/practical and what it looks like when those tasks are completed.

Key Features

GatorGrader automatically checks the work of technical writers and programmers. It can:

• Use its Gradle plugin to check projects implemented and documented in a wide variety of languages (e.g., Java, Python, LaTeX, Markdown, HTML, and CSS).

• Integrate with GitHub Classroom to check solution and starter repositories created for professors and students, respectively.

• Run in a cloud-based environment like Travis CI or on the command-line of a developer's workstation.

• Operate as a "batteries included" grading tool, supporting automated checks like the following:

  • Does a file exist in the correct directory with the requested name?

  • Does technical writing contain the desired number of words and paragraphs?

  • Does source code contain the designated number of language-specific comments?

  • Does source code or technical writing contain a required fragment or match a specified regular expression?

  • Does a command execute correctly and produce the expected number of output lines?

  • Does a command execute and produce output containing a fragment or matching a regular expression?

  • Does a GitHub repository contain the anticipated number of commits?

Aligning with key recommendations in a recent National Academies report, GatorGrader helps instructors automatically check student submissions in both introductory and application-oriented classes using languages like Markdown, Java, Python, JavaScript, CSS, and HTML. GatorGrader does not aim to solve problems related to building and linting a project or managing an assignment's submission, instead integrating with existing tools and systems like Gradle, GitHub, and GitHub Classroom to effectively handle those tasks.

Installing GatorGrader

Installing GatorGrader is not necessary if you intend to use it through its Gradle plugin. If you want to participate in the development of GatorGrader, the project maintainers suggest the use of Pyenv to install Python 3.6 or above. In addition to installing Git to access the project's GitHub repository, you should also install Pipenv for its support of package and virtual environment management. After completing the installation of these tools, you can type the following command in your terminal window to clone GatorGrader's GitHub repository:

git clone https://github.com/GatorEducator/gatorgrader.git

If you plan to develop new features for GatorGrader or if you want to run the tool's test suite in Pytest, then you will need to install the developer dependencies by typing pipenv install --python="$(pyenv which python)" --dev in the directory that contains GatorGrader. If you want to use GatorGrader, then you can type pipenv install --python="$(pyenv which python)" instead. Once these commands complete successfully, you have officially installed GatorGrader! Note that running these commands will ensure that Pipenv creates a virtual environment for GatorGrader that is bound to the version of Python that Pyenv set for use.

Testing GatorGrader

Automated Testing

The developers use Pytest for the testing of GatorGrader. Depending on your goals, there are several different configurations in which you can run the provided test suite. If you want to run the test suite to see if the test cases are passing, then running this command in a terminal window will perform testing with the version of Python to which Pipenv's virtual environment is currently bound.

pipenv run test

Test Coverage

Along with running the test suite, the developers of GatorGrader use statement and branch coverage to enhance their testing activities. To see the coverage of the tests while also highlighting the lines that are not currently covered by the tests, you can run this command in a terminal window. As with the previous command, this will run the tests in the version of Python to which Pipenv's virtual environment is currently bound.

pipenv run cover

Testing with Multiple Python Versions

The previous two commands are restricted to running the test suite in the version of Python to which Pipenv was bound. If you have installed multiple versions of Python with Pyenv and you want to iteratively bind Pipenv to each version and then run the test suite, then you should first run the following commands to install Pipx and use Pipx to install Invoke. The first of these three commands will install pipx, a program that supports the execution of Python packages in isolated environments. The second command makes the directory ~/.local/bin/ a part of the search path for executable Python programs and the third one installs the invoke command so that it is available on your workstation outside of a virtual environment managed by Pipenv, thereby ensuring that it is always available to run tasks.

pip install pipx --user
python -m userpath append ~/.local/bin/
pipx install invoke

Now you can run the test suite in the specified versions of Python with the following command. This example command will run the test suite in Python 3.6.8 and Python 3.7.3.

invoke -c scripts/tasks test --pyenv 3.6.8 --pyenv 3.7.3

If you want to track test coverage while running the tests in both Python 3.6.8 and 3.7.3, then you can run the following command.

invoke -c scripts/tasks cover --pyenv 3.6.8 --pyenv 3.7.3

You can switch the version to which Pipenv is bound by running the following command that adopts, for instance, Python 3.7.3.

invoke -c scripts/tasks switch --pyenv 3.7.3

Code Linting

The developers of GatorGrader use linting and code formatting tools, such as Pylint, Pydocstyle, and Black. After installing GatorGrader's development dependencies with Pipenv, you can run all of the linters by typing this command in a terminal window.

pipenv run lint --check

Automated Checks

Want to learn about our linting checks? Check us out on our website, GatorGrader! We have detailed descriptions of our linting checks and more! To get an idea of the linting checks we offer, here is a quick list:

1. ConfirmFileExists

2. CountCommandOutput

3. CountCommits

4. CountFileLines

5. CountFileParagraphs

Want to learn about our automated checks? Check them out on our website, gatorgrader.org! We have detailed descriptions of our automated checks and more!

Something you should know when working with our checks is that all of them come with some optional arguments. Optional arguments that you are likely to encounter:

• -h
• --help
• --exact
• --advanced

If --help is tagged along with a check then a help message will be displayed and then exited. If further assistance is needed, please contact us on GitHub.

Another feature with our automated checks is the plug-in based approach. This allows users to implement their own check if our initial 15 do not fulfill a check that you find necessary.

Running GatorGrader

Students and instructors normally use GatorGrader through its Gradle plugin, specifying the requested checks in a config/gatorgrader.yml file. When run through Gradle, GatorGrader reports each check that it performed, additionally sharing a diagnostic message for each check that did not pass. Individuals who want to run GatorGrader as a stand-alone Python application should first install it's application dependencies with Pipenv and then learn about the supported checks and their defaults by typing pipenv run python3 gatorgrader.py --help in a terminal window.

Instructors often run GatorGrader in conjunction with other tools that check source code and technical writing. For instance, in a Java-based introductory course, instructors could verify student submissions with Checkstyle, thereby ensuring that the Java source code adheres to the requirements in the Google Java Style Guide. In this course, an instructor could require that Markdown files with technical writing meet the standards described in the Markdown Syntax Guide, meaning that all Markdown files must pass the checks performed by the Markdown linting tool. These assignments could also require that all submitted technical writing must adhere to the standards set by the Proselint tool. Since GatorGrader can run an arbitrary command and check its error code, it is also possible to integrate it with a wide variety of other linters, code formatters, and testing tools.

Instructors may at times need to see a full list of checks to have a better understanding and therefore, we feel that it is important to know that there is an easy way for that to happen. This action will be completed through command line and therefore, you can type pipenv run python gatorgrader.py ListChecks into your terminal. This allows for all of the checks to be printed out as output. This output will have the necessary name labeled with the required and optional arguments. If this output does not give enough content, we warmly invite you to navigate to our website, where we go into more detail about our Automated Checks.

Using Docker

A vital part of our process for GatorGrader is to implement and use new techniques to further our tool to grow. This is why we chose to use Docker! Docker is a container platform and therefore, allows students using GatorGrader to just open a container and have easy access to run all commands that would allow them to build, run, and grade their labs and practicals. Docker is an industry standard and therefore, gives us an advantage. To open a container that will allow for the use of GatorGrader, run the following command in your terminal window:

docker run -it --rm --name dockagator \
  -v "$(pwd)":/project \
  -v "$HOME/.dockagator":/root/.local/share \
  gatoreducator/dockagator /bin/bash

From here, you are set! Test it out by building, running, or grading your lab/practical! If you would like to learn more about Docker, please follow this link.

Comparison to Other Tools

Other automated grading tools include:

• autograde-github-classroom: "scripts to download and grade submissions to Github Classroom"
• check50: "a tool for checking student code"
• Classroom Assistant: "desktop application to help you get student repositories for grading"
• nbgrader: "a system for assigning and grading notebooks"
• nerfherder: "scripts for automating grading with GitHub Classroom and Moodle"
• Submitty: "homework submission, automated grading, and TA grading system"
• WebCat: "all-in-one plugin for full processing and feedback generation"

Designed for instructors who want an alternative to simple scripts or stand-alone platforms that do not integrate with industry-standard infrastructure like GitHub and Travis CI, GatorGrader is a tool that automatically checks the work of technical writers and programmers. Unlike other systems, GatorGrader provides a "batteries included" approach to automated grading that makes it easy for instructors to combine well-tested checks for projects that students implement in a wide variety of programming languages. GatorGrader's developers take its engineering seriously, maintaining standards-compliant source code, a test suite with 100% statement and branch coverage, and top-notch source code and user documentation.

Presentations

GatorGrader's creators give presentations about the development, use, and assessment of the tool. Please contact one of the developers if you would like to feature a presentation about GatorGrader at your technical conference. The following list includes some of our team's recent presentations:

• A Hands-on Guide to Teaching Programming with GitHub, Travis CI, and Python
  at PyOhio 2018
• Using GitHub, Travis CI, and Python to Introduce Collaborative Software Development
  at PyCon Education Summit 2018
• Using Python, Travis CI, and GitHub to Effectively Teach Programming
  at PyGotham 2018

Contributing

Are you interested in contributing to GatorGrader, GatorGradle, or any of the sample assignments (e.g., Java, LaTeX, or HTML with CSS)? Great, because we appreciate the involvement of new contributors! Before you raise an issue or start to make a contribution to GatorGrader's repository, we ask that you review the project's code of conduct and the contribution guidelines.

Contributors

Thanks goes to these wonderful people (emoji key):

Gregory M. Kapfhammer 📢 💻 📖 🎨 🚇 ⚠️	Saejin Mahlau-Heinert 💻 📖 🚇 👀	Christian Walker 🖋	Andrew Everitt 💻 📖	Christian Lussier 💻 📖	Simon Burrows 💻	Austin Bristol 💻	Matt 💻 📖	Christopher Miller 💻 🎨 📖	Spencer Huang 💻 📖
Mohammad Khan 💻	Zachary Shaffer 💻 📖	Alexander Yarkosky 🐛 📖	Dillon 📖	Zachary Leonardo 💻	Jonathan W. Mendez 💻 📖	Tyler Lyle 💻	finneyj2 💻 📖	schultzh 🐛 💻 📖 👀	alexheinle 🐛 💻 📖 👀
Zachary Andrews 💻 🎨 📖	Nicholas Tocci 📖	Devin Ho 📖	Matthew Baldeosingh 📖	Jordan Durci 💻	Karol Vargas 📖	Jerfenson Cerda Mejia 💻 📖	Tara Douglass 💻 📖	Alexander Butler 💻	corlettim 🐛 💻 📖 👀
Carson Quigley 💻	Joshua Yee 💻 📖	Madelyn Kapfhammer 📖 🚇 ⚠️

This project follows the all-contributors specification. Contributions of any kind are welcome!