pygradethis 0.4.0

Python autograder to facilitate code output and static code checking.

pygradethis

A Python package to facilitate checking code output or static code checking using AST analysis. It can either be used with R using the learnr package, as a mirror of gradethis package, or as a standalone package for general Python use in educational settings.

Note: This package is in early development and will undergo rapid changes.

Install pygradethis

pip install pygradethis

Install Dev Dependencies

pip install -e .[dev]

Features

• Simple output checking based on pass / fail conditions with feedback
• Simple static code checking (AST), with feedback on how student's code differs from solution

Output checks

pygradethis mimics the cadence to gradethis::grade_result. For e.g., we can check that the student supplies the mpg dataset like so:

grade_result(
  pass_if_equals(mpg, "You also got the mpg dataframe!"),
  fail_if_equals(None, "")
)

Internally, these pass_if_equals(output, message) or fail_if_equals(output, message) will be checked sequentially in the order of arguments and return on first condition we match. The None here can be used if you simply want to execute a condition if none of the other conditions matched.

If we match a pass_if_equals or fail_if_equals, we will present a feedback message wrapped in a convenient dict:

dict(
    message = str,
    correct = True|False,
    type = "auto|success|info|warning|error|custom",
    location = "append|prepend|replace"
)

The message is the feedback, the correct is whether or not the student's solution is correct, type is the type of feedback. When used with learnr the location field here is useful for where the message is situated in the tutorial. However, for those using this package as a standalone the location is not an important field and it can be ignored. More on the flags here.

Internally, a random praise/encouragement message will be appended before any custom message supplied.

pass_if_equals(x = mpg, message = "You also got the mpg dataframe!")

Feedback:

Bravo! You also got the mpg dataframe!

fail_if_equals(x = None, message = "")

Feedback:

Try it again. You get better each time.

Code checks

For static code checking, we follow a similar cadence for gradethis::grade_code.

When there is a solution code being supplied, grade_code(user_code, solution_code) can be used to check the AST of the user and solution code, making sure to standardize function calls and producing a helpful message for the student to diagnose their issue.

Example:

grade_code(
  student_code="2 + sqrt(log(2))", 
  solution_code="2 + sqrt(log(1))"
)

Feedback:

I expected log(1), but what you wrote was interpreted as log(2) in sqrt(log(2)) at line 1.

Note how the feedback narrows in on the expression in which the problem occurs (sqrt(log(2))) so that the student can focus on the most relevant outer expression of the problem. In this case, the log(2) is the problem and the 2 on the left operand of the addition is not as relevant.

Similarly, here the feedback points out that the 2 within the log function is incorrect, similar to the gradethis example.

Call Standardization

pygradethis also knows how to take user's function call code and map positional arguments to proper parameter names and set defaults if not supplied. This is so that you don't penalize a student's code just because they did not explicitly spell out positional argument names, or write the default arguments out.

For e.g. suppose a student is calling the following silly function foo:

def foo(a, b=1): 
  pass

Grading the code with

grade_code(
  student_code="foo(1)", 
  solution_code="foo(1)"
)

In the example above, the grade_code doesn't give us a feedback message since they are equivalent expressions.

However, if the student supplies foo(2)

grade_code(
  student_code="foo(2)", 
  solution_code="foo(1)"
)

we get back this feedback:

I expected 1, but what you wrote was interpreted as 2 in foo(2) at line 1.

Note: Although underneath the hood we do standardize the arguments of both the student and the solution code before checking, we don't surface this standardized form to the feedback message. This is certainly possible to achieve but in certain cases can hinder learning by revealing too much information. For example, the builtin functions like sum is normally called without specifying its actual formal parameters (e.g. sum(1) versus sum(iterable=[1], start=0). In the future, a verbose mode could be made available such that the formal parameters are pointed out.

For call standardizing to work, the function definitions corresponding to function calls must be defined and 'live' in the environment, whether that is the globals()/locals(), builtins, or custom module imports pandas. This works if the student/solution source code also includes the definition (like foo above) in their own source code or it's included by instructor.

Currently, common modules like math is imported for grading within check_functions.py, but more modules will be included to serve data science grading as well, such as pandas or numpy in the future. We plan to make the code more extensible for the instructor to add them as dependencies.