dlai-grader 1.3.0

Grading utilities for DLAI courses

grader

Automatic grading for DLAI courses. Designed to be compatible with Coursera's grading requirements.

Installation

You use pip to install it!

pip install dlai_grader

How to use it

Initialize a grader

To start building your grader cd into the directory where you will be working and use the following command:

dlai_grader --init

This will ask you for:

• Name of the course (abbreviation is recommended)
• Number of the course
• Number of the week
• Version of the grader (defaults to 1 but can be 2 or any other number)

This will generate a file system tree like this:

.
└── grader (directory you invoked the command from)
    ├── data/               -> To store datasets (csv, TF Datasets, etc).
    ├── learner/            -> The learner facing version will be generated here.
    ├── solution/           -> Place solution.ipynb here
    ├── submission/         -> Necessary only in debug mode (no need to place anything here).
    ├── mount/              -> This mocks the bind mount that coursera will attach to the container. Should contain submission.ipynb or other file required for grading.
    ├── .conf               -> Configuration variables.
    ├── Dockerfile          -> Uses frolvlad/alpine-miniconda3:python3.7 as base image.
    ├── Makefile            -> Useful commands.
    ├── requirements.txt    -> Python dependencies.
    ├── entry.py            -> Entrypoint of the grader.
    └── grader.py           -> Grading logic.

Placing the solution and submission

Now that you have the layout of the grader you will need to place the solution of the assignment within the solution/ directory. This file must be named solution.ipynb.

A good starting point is to use the solution to create the first iteration of the grader. To do this place the solution within the mount/ directory and rename it to submission.ipynb. You can use the make submit-solution command to do this.

Your filesystem tree should look like this:

.
└── grader
    ├── ... 
    ├── solution/
    │    └── solution.ipynb
    ├── mount/ 
    │    └── submission.ipynb
    └── ...

Note that the grader can be used to grade files other than Jupyter notebooks. If this is the case you can leave the solution/ directory empty and place the file to grade within mount/. This file can be anything (.h5, .tar.gz, .zip, etc) the only requirement is that it should match the name of the file to submit in the coursera programming item.

Add versioning to the notebook

Why is this useful?

We have seen many learners facing issues when submitting their assignments because coursera does not show them the latest version. To address this, a good alternative it to always check that the submission of the learner is up to date and if not, tell them how they can upgrade to the latest version.

To ensure that a submission is compatible with a particular version of the grader the versioning feature has been created.

Within the .conf file you will find the current version of the grader under the variable GRADER_VERSION.

To compare against this version the submission notebook must include a variable called grader_version in its metadata. To add this variable to the submission.ipynb file you can use the make versioning command (this is just a wrapper of the dlai_grader --versioning command). This will add the variable matching the same version as the one found in the .conf file.

Upgrading the grader and notebook version

After a refactor with breaking changes to the grader it is a good idea to upgrade the version to a newer one. You can do this be using the make upgrade command. This will add 1 to the current version in the .conf file and in the notebook.

Tagging graded cells

You might decide to filter out cells created by learners (or other ones such as the ones that train models). If you take this approach you can add a tag to each cell's metadata and then filter out the ones that don't have this tag .

If you wish to add the graded tag to each cell in the submission.ipynb notebook you can use the make tag command.

Important note for make commands

Note that the make tag, make upgrade and make versioning commands change only the mount/submission.ipynb file.

Adding Python dependencies

The next step is to include all the necessary Python dependencies. For this add them to the requirements.txt file. By default only dlai_grader is included.