otter-service 2.0.13

Grading Service for Edx 8x courses

otter-nb service

This repo contains a tornado flask app that accepts .ipynb files and grades them in a dockerized environment. Assuming you are running a Jupyterhub, you can ask Jupyterhub to run this otter-service as a service; you also have the option to run it in a stand alone manner. Grades are saved to a gcloud Cloud Firestore.

A separate Jupyterhub extension, otter-submit, presents a "Submit" button to the user in a notebook rendered in Jupyterhub. The button is configured to serialize and send the notebook to this otter-service as well as notify the the user of the successful submission.

FireStore/Database setup

All grades are written to gcloud Cloud FireStore. During local testing with pytest, the tests remove the collection created after verifying the data was written. Local docker testing, does not delete the entries but the collection is called, otter-docker-local-test, and can be viewed and deleted by going to the gcloud console and navigating to Cloud Firebase or Cloud Firestore.

Configuration

Notebook(ipynb metadata)

The ipynb notebooks need to include the metadata for which assignment they are. There are three pieces of information that are relevant: the course name, section and lab. They should be nested in an "otter_service" attribute as shown below. These values correspond to the course-config.json file described in the section below. These are set in the metadata section of every notebook:

metadata:{
  "otter_service": {
    "course": "8x",
    "section": "1",
    "assignment": "lab01"
  }
  ...
}

Course Configuration

Each course needs to provide two pieces of information to whomever is handling otter-service(right now: [e-mail]) that is deployed in the secrets file of this application

1. The name of repository where the autograder.zip files are kept. (e.g. github.com/edx-berkeley/88E-autograders)

The GitHub App (OTTER_GH_APP_ID, OTTER_GH_APP_PRIVATE_KEY, OTTER_GH_APP_INSTALLATION_ID) handles access to autograder repos — no personal access token needed.

Finally, the private repository with the autograder.zip files needs to contain a file named: course-config.json. The file is structured like this:

(1) course name which matches the course name in every notebooks metadata
(2) section which matches the section in every notebooks metadata - if only one section put '1'
(3) Edx course id
(4) subpath to each autograder.zip in private autograder.zip repo
(5) each EDX assignment name and assignment Id -- the assignment name must match the notebook metadata

{
  "8x" : { //(1)
    "1": { //(2)
      "course_id": "BerkeleyX+Data8.1x+3T2022", //(3)
      "subpath_to_zips": "materials/x22/lab/1", //(4)
      "assignments": {  //(5)
        "lab01": "6333e19d6b4d46f88df671ba50f616d8",
        "lab02": "fbf6740d45094b9b977111d218969273",
        "lab03": "0c8f28bdc48d4231843f62b512d73638",
        "lab04": "8db69daf14cf4751a088106be912c0cd",
        "lab05": "4cb9e3491c284cd5ae29bb48219ee15b"
      }
    },
    "2":{
      "course_id": "BerkeleyX+Data8.2x+3T2022",
      "subpath_to_zips": "materials/x22/lab/2",
      "assignments": {
        "lab01": "7abc0025c10f4b8ab123dbc88d34faaf",
        ...

Note how the folder structure is mirrored in the course-config.json.

If you are not posting grades to an LTI server, than you do not need to worry about this.

Test files

The autograder zip files and test notebooks live in the per-course autograder repos (e.g. edx-berkeley/88E-autograders). The GitHub App fetches them at grading time — no Dockerfile changes needed when assignments change.

Docker Image

This just FYI. The Dockerfile pulls an image :

docker pull ucbdsinfra/otter-grader

This image is used by otter-grader to run the containerized grading.

EdX/LTI integration

The system posts the grade back to the EdX via LTI. You need to have the LTI_CONSUMER_KEY and LTI_CONSUMER_SECRET defined and encoded via sops for this to work correctly. The secrets are in otter-service/secrets/gke_key.yaml

Deployment

otter-service runs in-cluster on the edx GKE cluster (GCP project data8x-scratch) in the otter-prod and otter-staging namespaces. The Helm chart lives in edx-berkeley/edx-hub. Deployment is via the deploy-otter.yaml GitHub Actions workflow in that repo — push to prod branch or trigger manually from the Actions tab.

Deployment Details:

Rollback:

If we deploy and find problems the quickest way to rollback the deployment is to look at the revision history and undo the deployment by deploying to a previous revision number:

• kubectl rollout history deployment otter-pod -n otter-prod
• kubectl rollout history deployment otter-pod -n otter-prod --revision=# <-- to see details like the version of the image used
• kubectl rollout undo deployment/otter-pod -n otter-prod --to-revision=#

CI/CD:

If you push a tag in the standard form of a version number(XX.XX.XX), GitHub action creates a release from this tag, pushes the release to pypi.org, builds the docker image, pushes it google's image repository and deploys the new image into the GKE cluster.

pod size recommendations

There is a vertical pod autoscaler deployed to recommend memory and cpu sizing to the otter-pod pods. You can see recommendations via either of these commands:

• kubectl get vpa -n otter-prod
• kubectl get vpa -n otter-prod --output yaml

It is called an autoscaler but I configured the resource to just recommend and not actually autoscale vertically.

pod horizontal scaling

A horizontal autoscale is configured to spin up a new pod when 80% of CPU requested is utilized. There is maximum of 10 pods allowed.

You can see the status of the horizontal scaling via this command:

• kubectl get hpa -n otter-prod

pytest

Run ./deployment-utils/local/pytest.sh -- this will start the Firestore emulator and run the tests. If the emulator is already running it shuts it down. I shut down the emulator when the tests are done as well but you could comment out this line to check out the data that was stored.

Local installation for testing/developing

Install a FireStore Emulator so you test locally:

• Install FireStore CLI: https://firebase.google.com/docs/cli/#install-cli-mac-linux
• firebase login
• firebase projects:list
• firebase setup:emulators:firestore
• make java jdk installed
• firebase emulators:start --only firestore --project data8x-scratch
• You can see the UI here: http://localhost:4000/firestore
• python3 -m pip install google-cloud-firestore
• You will notice the re-direct in firebase_local fixture used by test_write_grade in test_otter_nb.py

With docker installed, you can use the Dockerfile-dev file to deploy a local instance of otter-service. The deployment/local/build.sh file gives some guidance to building and installing local changes to otter-service for testing. The usual process is to make changes, execute build.sh, which relies on a docker-compose.yml file. A sample is below but before we look, I would also study the file tests/integration.py. If you execute this file, you can test the service via a web connection.

Sample docker-compose.yml:

version: "3.9"
services:
  app:
    image: otter-srv
    build:
      context: .
      dockerfile: Dockerfile-dev
      args:
        OTTER_SERVICE_VERSION: whatever_version you specify in otter-service/__init__.py
    env_file:
      - ../.local-env
    ports:
      - 10101:10101
    volumes:
      - /tmp/otter:/mnt/data
    entrypoint: ''

networks:
  default:
    driver: bridge

Notes:

• .local-env These are environment variables that must be set. They mirror the variables in the file otter-service/values.yaml under the key otter_env. You do not need to encrypt your local-env file with sops.

Typical Workflow

• Activate/create the python environment with conda or virtualenv using requirements/dev.txt
• Make changes on a feature branch
• Add tests to tests dir
• Run Tests: sh deployment-utils/local/pytest.sh
• Deploy Locally: sh deployment-utils/local/build.sh
• Run Integration Test: python3 tests/integration.py local 88e(or 8x) -- see file
• Check local firestore to see progress: http://localhost:4000/firestore
• Open a PR to the staging branch in edx-berkeley/edx-hub to deploy to staging (requires STAGING_ENABLED=true)
• Once verified on staging, open a PR from staging → prod in edx-hub to deploy to prod

Service installation in JupyterHub

Instructions can be found here for running it as a service within your jupyterhub