gatorgrader 1.1.1

Automated Grading Tool that Checks the Work of Writers and Programmers

GatorGrader

The only tool you'll need to ensure your student's code and writing is up to speed!

Table of Contents

• Quickstart Guide
• Key Features
• What Do People Think about GatorGrader?
• Installing GatorGrader
• Testing GatorGrader
  • Automated Testing
  • Test Coverage
  • Testing with Multiple Python Versions
  • Code Linting
• Running GatorGrader
• Using Docker
• Comparison to Other Tools
• Presentations
• Contributing
• Contributors

Quickstart Guide

• Starter Repositories An easy way to get started with GatorGrader is to check out our sample starter repositories. The following starter repositories provide examples of how GatorGrader files should be created to check programs and documentation for different languages: Java, Python, LaTex and HTML with CSS. These examples also show how to integrate GatorGrader with GitHub Classroom and Travis CI. When you follow these examples, the Gradle plugin for GatorGrader will install it automatically when you run gradle grade in a terminal window. Please be aware that these repositories are meant to have a majority of red checks. This is only meant to be a “starter” and give an insight into what must be accomplished within the lab.
• Solution Repositories The next step to get involved with GatorGrader is to checkout our sample solution repositories. The following solution repositories provide examples of how GatorGrader files should be created to check programs and documentation for different languages: Java, Python, LaTex , and HTML with CSS. These examples also show how to integrate GatorGrader with GitHub Classroom and Travis CI. When you follow these examples, the Gradle plugin for GatorGrader will install it automatically when you run gradle grade in a terminal window. Please be aware that these repositories are meant to have a majority of green checks. This is meant to show what must be accomplished within a lab/practical and what it looks like when those tasks are completed.

Key Features

GatorGrader automatically checks the work of technical writers and programmers. It can:

• Enable GatorGrade to check projects implemented and documented in a wide variety of languages (e.g., Java, Python, LaTeX, Markdown, HTML, and CSS).

• Integrate with GitHub Classroom to check solution and starter repositories created for professors and students, respectively.

• Run in a cloud-based environment like Travis CI or on the command-line of a developer's workstation.

• Operate as a "batteries included" grading tool, supporting automated checks like the following:

  • Does a file exist in the correct directory with the requested name?

  • Does technical writing contain the desired number of words and paragraphs?

  • Does source code contain the designated number of language-specific comments?

  • Does source code or technical writing contain a required fragment or match a specified regular expression?

  • Does a command execute correctly and produce the expected number of output lines?

  • Does a command execute and produce output containing a fragment or matching a regular expression?

  • Does a GitHub repository contain the anticipated number of commits?

Aligning with key recommendations in a recent National Academies report, GatorGrader helps instructors automatically check student submissions in both introductory and application-oriented classes using languages like Markdown, Java, Python, JavaScript, CSS, and HTML. GatorGrader does not aim to solve problems related to building and linting a project or managing an assignment's submission, instead integrating with existing tools and systems like Gradle, GitHub, and GitHub Classroom to effectively handle those tasks.

Installing GatorGrader

Installing GatorGrader is not necessary if you intend to use it through GatorGrade. If you want to participate in the development of GatorGrader, the project maintainers suggest the use of Pyenv to install Python 3.7 or above. In addition to installing Git to access the project's GitHub repository, you should also install Poetry for its support of package and virtual environment management. After completing the installation of these tools, you can type the following command in your terminal window to clone GatorGrader's GitHub repository:

git clone https://github.com/GatorEducator/gatorgrader.git

If you plan to develop new features for GatorGrader or if you want to run the tool's test suite in Pytest, then you will need to install a virtual environment for development by typing poetry install in the directory that contains GatorGrader. For help with this process, refer to Poetry's documentation.

Testing GatorGrader

Automated Testing

The developers use Pytest for the testing of GatorGrader. Depending on your goals, there are several different configurations in which you can run the provided test suite. If you want to run the test suite to see if the test cases are passing, then running this command in a terminal window will perform testing with the version of Python with which Poetry's virtual environment was initialized.

poetry run task test

Test Coverage

Along with running the test suite, the developers of GatorGrader use statement and branch coverage to enhance their testing activities. To see the coverage of the tests while also highlighting the lines that are not currently covered by the tests, you can run this command in a terminal window. As with the previous command, this will run the tests in the version of Python with which Poetry's virtual environment was initialized.

poetry run task cover

Testing with Multiple Python Versions

The previous two commands are restricted to running the test suite in the version of Python with which Poetry was initialized. If you have installed multiple versions of Python with Pyenv and you want to iteratively initialized Poetry with each version and then run the test suite, then you should first run the following commands to install Pipx and use Pipx to install Invoke. The first of these three commands will install pipx, a program that supports the execution of Python packages in isolated environments. The second command makes the directory ~/.local/bin/ a part of the search path for executable Python programs and the third one installs the invoke command so that it is available on your workstation outside of a virtual environment managed by Poetry, thereby ensuring that it is always available to run tasks.

pip install pipx --user
python -m userpath append ~/.local/bin/
pipx install invoke

Now you can run the test suite in the specified versions of Python with the following command. This example command will run the test suite in Python 3.6.8 and Python 3.7.3.

invoke -c scripts/tasks test --pyenv 3.6.8 --pyenv 3.7.3

If you want to track test coverage while running the tests in both Python 3.6.8 and 3.7.3, then you can run the following command.

invoke -c scripts/tasks cover --pyenv 3.6.8 --pyenv 3.7.3

You can switch the version with which Poetry is initialized by running the following command that adopts, for instance, Python 3.7.3.

invoke -c scripts/tasks switch --pyenv 3.7.3

Code Linting

The developers of GatorGrader use linting and code formatting tools, such as Pylint, Pydocstyle, and Black. After installing GatorGrader's development dependencies with Poetry, you can run all of the linters by typing this command in a terminal window.

poetry run task lint --check

Automated Checks

Want to learn about our linting checks? Check us out on our website, GatorGrader! We have detailed descriptions of our linting checks and more! To get an idea of the linting checks we offer, here is a quick list:

1. ConfirmFileExists

2. CountCommandOutput

3. CountCommits

4. CountFileLines

5. CountFileParagraphs

Want to learn about our automated checks? Check them out on our website, gatorgrader.org! We have detailed descriptions of our automated checks and more!

Something you should know when working with our checks is that all of them come with some optional arguments. Optional arguments that you are likely to encounter:

• -h
• --help
• --exact
• --advanced

If --help is tagged along with a check then a help message will be displayed and then exited. If further assistance is needed, please contact us on GitHub.

Another feature with our automated checks is the plug-in based approach. This allows users to implement their own check if our initial checks do not fulfill a check that you find necessary.

Running GatorGrader

Students and instructors normally use GatorGrader through GatorGrade, specifying the requested checks in a gatorgrade.yml file. When run through GatorGrade, GatorGrader reports each check that it performed, additionally sharing a diagnostic message for each check that did not pass. Individuals who want to run GatorGrader as a stand-alone Python application can install GatorGrader itself through Pipx, and then run gatorgrader --help to get more details on the command-line interface.

Instructors often run GatorGrader in conjunction with other tools that check source code and technical writing. For instance, in a Java-based introductory course, instructors could verify student submissions with Checkstyle, thereby ensuring that the Java source code adheres to the requirements in the Google Java Style Guide. In this course, an instructor could require that Markdown files with technical writing meet the standards described in the Markdown Syntax Guide, meaning that all Markdown files must pass the checks performed by the Markdown linting tool. These assignments could also require that all submitted technical writing must adhere to the standards set by the Proselint tool. Since GatorGrader can run an arbitrary command and check its error code, it is also possible to integrate it with a wide variety of other linters, code formatters, and testing tools.

Instructors may at times need to see a full list of checks to have a better understanding and therefore, we feel that it is important to know that there is an easy way for that to happen. This action will be completed through command line and therefore, you can type gatorgrader ListChecks into your terminal, if you've installed GatorGrader as detailed above. This allows for all of the checks to be printed out as output. This output will have the necessary name labeled with the required and optional arguments. If this output does not give enough content, we warmly invite you to navigate to our website, where we go into more detail about our Automated Checks.

Using Docker

Note: Docker supports the older GatorGrader execution tool, GatorGradle. It is not needed for GatorGrade!

A vital part of our process for GatorGrader is to implement and use new techniques to further our tool to grow. This is why we chose to use Docker! Docker is a container platform and therefore, allows students using GatorGrader to just open a container and have easy access to run all commands that would allow them to build, run, and grade their labs and practicals. Docker is an industry standard and therefore, gives us an advantage. To open a container that will allow for the use of GatorGrader, run the following command in your terminal window:

docker run -it --rm --name dockagator \
  -v "$(pwd)":/project \
  -v "$HOME/.dockagator":/root/.local/share \
  gatoreducator/dockagator /bin/bash

From here, you are set! Test it out by building, running, or grading your lab/practical! If you would like to learn more about Docker, please follow this link.

Comparison to Other Tools

Other automated grading tools include:

• autograde-github-classroom: "scripts to download and grade submissions to Github Classroom"
• check50: "a tool for checking student code"
• Classroom Assistant: "desktop application to help you get student repositories for grading"
• nbgrader: "a system for assigning and grading notebooks"
• nerfherder: "scripts for automating grading with GitHub Classroom and Moodle"
• Submitty: "homework submission, automated grading, and TA grading system"
• WebCat: "all-in-one plugin for full processing and feedback generation"

Designed for instructors who want an alternative to simple scripts or stand-alone platforms that do not integrate with industry-standard infrastructure like GitHub and Travis CI, GatorGrader is a tool that automatically checks the work of technical writers and programmers. Unlike other systems, GatorGrader provides a "batteries included" approach to automated grading that makes it easy for instructors to combine well-tested checks for projects that students implement in a wide variety of programming languages. GatorGrader's developers take its engineering seriously, maintaining standards-compliant source code, a test suite with 100% statement and branch coverage, and top-notch source code and user documentation.

Presentations

GatorGrader's creators give presentations about the development, use, and assessment of the tool. Please contact one of the developers if you would like to feature a presentation about GatorGrader at your technical conference. The following list includes some of our team's recent presentations:

• A Hands-on Guide to Teaching Programming with GitHub, Travis CI, and Python
  at PyOhio 2018
• Using GitHub, Travis CI, and Python to Introduce Collaborative Software Development
  at PyCon Education Summit 2018
• Using Python, Travis CI, and GitHub to Effectively Teach Programming
  at PyGotham 2018

Contributing

Are you interested in contributing to GatorGrader, GatorGradle, or any of the sample assignments (e.g., Java, LaTeX, or HTML with CSS)? Great, because we appreciate the involvement of new contributors! Before you raise an issue or start to make a contribution to GatorGrader's repository, we ask that you review the project's code of conduct and the contribution guidelines.

Contributors

Thanks goes to these wonderful people (emoji key):

Gregory M. Kapfhammer 📢 💻 📖 🎨 🚇 ⚠️	Saejin Mahlau-Heinert 💻 📖 🚇 👀	Christian Walker 🖋	Andrew Everitt 💻 📖	Christian Lussier 💻 📖	Simon Burrows 💻	Austin Bristol 💻	Matt 💻 📖	Christopher Miller 💻 🎨 📖	Spencer Huang 💻 📖
Mohammad Khan 💻	Zachary Shaffer 💻 📖	Alexander Yarkosky 🐛 📖	Dillon 📖	Zachary Leonardo 💻	Jonathan W. Mendez 💻 📖	Tyler Lyle 💻	finneyj2 💻 📖	schultzh 🐛 💻 📖 👀	alexheinle 🐛 💻 📖 👀
Zachary Andrews 💻 🎨 📖	Nicholas Tocci 📖	Devin Ho 📖	Matthew Baldeosingh 📖	Jordan Durci 💻	Karol Vargas 📖	Jerfenson Cerda Mejia 💻 📖	Tara Douglass 💻 📖	Alexander Butler 💻	corlettim 🐛 💻 📖 👀
Carson Quigley 💻	Joshua Yee 💻 📖	Madelyn Kapfhammer 📖 🚇 ⚠️

This project follows the all-contributors specification. Contributions of any kind are welcome!