PyICs

Python Interactive Courses

What is PyICs?

PyICs is a framework for interactive Python tutorials, inspired by the rustlings tutorial for the Rust programming language. It strives to strike a balance between ease of use and feature volume, for both instructors and students.

Installation

PyICs is available via pip. Install the latest version using pip install python-interactive-courses, requiring python>=3.14. PyICs is available on Windows (Windows 10 or later), MacOS, and Linux.

Using the PyICs App

As a Student

Accessing Courses

Courses come from your instructor: either as files (a course manifest, *.pyicscourse.json, and its task files) or as a course archive (a .zip file), or from a course library (a source) that you add once by the URL of its index, which your instructor gives you:

pyics [options] add_source [--name NAME] https://example.org/pyics-courses/index.json
pyics search                    # the courses of your sources; pyics search graphs finds matching ones
pyics pull search-algorithms    # pulls it into ./search-algorithms
pyics                           # start it from the course selector

• pyics pull can also take the URL or path of a course archive, and <source>/<course> if everal of your sources have a course of that name. --dest <directory> pulls it elsewhere.
• pyics update (in the course's directory, or pyics update <directory>) updates a pulled course to its latest version. Your exercise files are kept as they are, and so is your progress.
• pyics sources lists your sources, and pyics remove_source <name> removes one. Sources are stored in the environment pyics is installed in, so each virtual environment has its own: in etc/pyics/sources.json inside it (its cached indexes in var/cache/pyics). They stay when pyics is upgraded. Outside a virtual environment, they are stored in your pyics configuration directory (e.g. ~/.config/pyics on Linux, ~/Library/Application Support/pyics on macOS, %APPDATA%\pyics on Windows). Older versions of pyics stored sources there for all environments: a virtual environment without sources of its own gets a copy of them the first time pyics runs in it, and pyics asks once whether to delete the old file. pyics remove_legacy_sources deletes it any time later. Once it is deleted, environments that haven't copied the sources yet don't get them, so add them there again. Administrators can add sources for all users of a computer in the site configuration directory (e.g. /etc/xdg/pyics/sources.json). The PYICS_CONFIG_DIR environment variable sets another configuration directory, whose sources are then the only ones used.
• The course selector also offers the courses of your sources that you haven't pulled yet, as they were when you last added, searched or pulled from them. Selecting one pulls it into the working directory.

Courses run code on your computer, as their tests run your code together with theirs: only add sources and pull courses from people you trust. Sources are fetched over https (or from local files, e.g. on a network share); plain http has to be allowed with --allow_http.

Some courses need packages besides pyics, e.g. numpy. They have to be installed in the same environment where PyICs is installed, siunce the tests run in the same Python interpreter. pyics search and the course selector show which of them are missing, and pyics pull prints the command that installs them (or runs it with --install_deps, after asking). Packages are never installed without asking.

Navigating Courses

The main app starts with a course selector, listing the course manifests found in the working directory and up to two directories below it (.venv, .git and the like are skipped). Once you have selected a course, c goes back to the course selector, and esc from there returns to the course that was left. Courses left this way resume where they were left.

When a course is opened, it opens to the first unsolved task in the course. The main body of the app shows a number of tests which fail for the exercise file(s) for the current task, and a link to the file(s) at the very top.
Once all tests configured for a task pass, the task counts as completed. When that is the case n progresses to the next task, and p returns to the previous one. Tests your instructor marked as soft are shown in yellow when they fail: they point out something worth fixing, but don't keep the task from counting as completed. Some courses also let you move on with n before a task is completed.

Help Display

Tasks can have hints (help in the task file), which create_task and edit_task ask for after the instructions. In the main app, h reveals the current task's next hint above its test results, one at a time and in order. Revealed hints stay revealed until the app is quit, also when you go to another task and back.

Tracing Failed Tests

When a test fails, t runs the task's tests again with snoop, which shows below each failed test what your code did: every line that was run, the values of the variables it changed, and the calls it made, with their arguments and return values. This trace mode stays on while you keep saving your exercise file, until t is pressed again, and it is off again for every other task you move to.

Function tests (SimpleTest) trace the calls of the tested functions, output tests (PrintTest) the run of the whole file, and pytest tests (GeneralTest) the code each test runs. Their authors can turn tracing off for a test by creating it with traceable=False, e.g. so that its arguments aren't revealed. Tests that check the structure or the signatures of your code (StructureTest, SpecTest) aren't traced.

Jupyter Notebooks

• A notebook comes without a kernel, so Jupyter or VS Code asks you to select one. Choose the environment in which PyICs is installed, as that is where the tests run.
• The tests of a task run whenever you save its notebook (ctrl+s, or when Jupyter autosaves it), like when you save a Python file.
• They run the code cells of the saved notebook from the top, the way Restart Kernel and Run All Cells does. An error in a cell doesn't stop the run, so an unfinished exercise further down doesn't fail the tests of the exercises above it. When a test fails, its message names the cells that raised an error, e.g. cell 7 raised NameError: name 'qeue' is not defined. Cells are counted from the top, markdown cells included.
• Notebook-only syntax in a test run:
  • %%time, %%timeit and %%capture run their cell once, without timing it, and so do %time and %timeit for their line. What %%capture captures isn't part of the output.
  • Other cell magics (e.g. %%writefile, %%bash) skip their cell, and line magics (e.g. %matplotlib inline, %pip install) and shell commands (!ls) do nothing: files = !ls gives an empty list.
  • display(x) prints x as Jupyter shows it as text (its repr), whether or not you import it from IPython.display. Plots are drawn without a window, so plt.show() doesn't wait for one to be closed.
• Cells tagged skip-execution don't run in tests, e.g. a long download or training run. Tags are edited in JupyterLab's property inspector, or with Add Cell Tag in VS Code.
• Output tests (PrintTest) compare what all cells output: what they print, and what Jupyter shows as text, i.e. displayed objects, and the value of a cell whose last line is an expression (Out[n]), unless it is None or the line ends with ;. The expected output only has to be part of it, so what you output besides, e.g. while trying things out, doesn't fail the test.
• Traces (t) and the messages of structure tests name cells and their lines, e.g. bfs.ipynb:cell 3 or cell 3, line 2.

As an Instructor

PyICs packages a number of scripts to create, edit, and distribute courses. The process of creating a course and recommended practices are described below.

Courses and Tasks

In its background PyICs distinguishes courses and tasks. A course packages multiple tasks, and is detected by PyICs in the current working directory (recursively, up to 3 layers deep) via its file extension (.pyicscourse.json). A course does not store any information on its tasks other than paths to the course files. This will most likely be addressed in a future version.

Tasks on the other hand package information on a single task, including the paths and contents of exercise and solution files, as well as serialized versions of the configured dynamic tests (see below).

You can create a course by running the following command:

pyics create_course [--fp file_path]

The script will ask you to either add tasks you have already created, or directly launch the task creation script which is also available as a standalone using the following command:

pyics create_task [--fp file_path]

Note that while the task name is mostly for indexing purposes, the task instructions are displayed in the PyICs header while the task is active. Both task and course creation scripts run interactively, but the task creation script requires you to provide at least one exercise file. The script does not create the file for you; it expects a path to an existing file. Exercise files are created on the student's machine when a task is opened.

Similarly, solution files are provided as paths to existing files, but a task can be created without a solution file.

Finally, test creation is run interactively (see below for details on available tests). It is recommended to have a file ready which contains Python objects required for the tests, i.e. the parameters of the tests' constructors. If such a file is provided, the script allows you to select symbols for each of the parameters from those available in the provided file. This is particularly helpful for SpecTests, as the signatures can be inferred from class or function definitions in a file.
Note: If a symbol name matches a parameter name in the test's constructor, the symbol is automatically selected.

create_course also asks whether learners may move on to the next task before solving the current one. If so, the manifest stores "free_navigation": true, and n always moves on. By default, a task has to be solved first.

Existing tasks and courses can be edited using the following commands

pyics edit_task path-to-task-file
pyics edit_course path-to-course-file

Tests

The following tests can be configured. If a desired test can't be represented by the options below, please open an issue describing the desired behavior. Any test requires the following two parameters:

• name: The name of the test, displayed in the test results if the test fails.
• file_path: The path to the exercise file the test targets.

Every test also takes soft: a soft test that fails is shown in yellow, as a warning, and doesn't keep its task from counting as solved, e.g. for style checks or optional extras. Defaults to False. A soft GeneralTest makes all the tests its code defines soft.

SimpleTest

Simple tests are unit tests run on student's code. A SimpleTest additionally expects the following parameters:

• function_name: The name of the function the test executes.
• expected: The expected result, i.e. output of the function wrt...
• function_args: Positional arguments to pass to the function in order to produce the expected output
• function_kwargs: Keyword arguments to pass to the function in order to produce the expected output
• constructor: The name of a class (or function) of the exercise file that creates an object, whose methods are then called instead of the module's functions. Its arguments are constructor_args and constructor_kwargs. Defaults to None.
• traceable: Whether a detailed trace of the function can be displayed. Defaults to True, can be turned off if the trace would give away too much of the solution (use sparingly).
• isolated: Whether the test runs in a temporary working directory of its own, which is deleted afterwards, e.g. to clean up files the student's code writes. Defaults to False: tests run in the directory of the exercise file, as students run it, so that relative paths like open("edges.csv") work. This holds for all tests.

function_name, function_args, function_kwargs and expected can also be lists, with an item per call, for a sequence of calls, and expected can be Raises(...) for a call that must raise an exception. Arguments and expected values can be Make(...), e.g. Make("Graph", edges), to create an object of the exercise file while the test runs: objects of classes that the test information file defines can't be stored in a task. Both markers are available from the pyics.tests module (from pyics.tests import Make, Raises), and are in scope when entering values in create_task. See the SimpleTest documentation.

SpecTest

Spec tests are signature checks on the student's code. A SpecTest additionally expects the following parameters:

• symbol_name: The name of the symbol where the expected signature needs to be located. It can be dotted, e.g. Graph.out_degree to check a method of a class.
• expected: The expected signature. Pass a Type (class) or function/method here. It will be converted to a spec_dict, which can then be serialized and checked against.
• by_type: If this flag is set to true, check the signature based on type annotations instead of parameter/attribute names. This excludes the symbol_name, which still needs to match.

StructureTest

Structure tests can test how tasks are solved by leveraging the ast builtin library. A StructureTest additionally expects the following parameter:

• rules: A list of rules enforced by this test. A detailed documentation on how these rules can be configured is found in the StructureTest documentation.

PrintTest

Print tests can test whether the stdout of students' files match a desired output. A PrintTest additionally expects the following parameters:

• expected: The expected output of the targeted file to stdout. The output of a Python file has to match it, while the output of a notebook only has to contain it (see Jupyter Notebooks).
• inputs: A list of inputs to provide to the file in case the running script expects inputs (from the builtin input() function).
• traceable: Whether a detailed trace of teh function can be displayed. Defaults to True, can be turned off if the trace would give away too much of the solution (use sparingly).
• isolated: Whether the file runs in a temporary working directory of its own, which is deleted afterwards. Defaults to False.

GeneralTest

General tests are pytest code, for what the other tests can't express. Each test the code defines is a result of its own. A GeneralTest additionally expects the following parameters:

• source: The pytest code, e.g. read from a file of pytest tests. Its tests can use the fixture solution, the exercise file loaded as a module, and the other fixtures described in the GeneralTest documentation.
• traceable: Whether a detailed trace of the tests can be displayed. Defaults to True.
• isolated: Whether each test runs in a temporary working directory of its own, which is deleted afterwards. Defaults to False.

Distribution

A course is distributed as an archive holding the course manifest and its task files, but not the exercise or solution files, which are created from the data stored in the task files. Progress is not tracked via the course files, but rather via the exercises that are already solved (exercise files are not overwritten if they already exist). The following command automatically creates a course archive:

pyics [options] pack [--out OUT] [--version VERSION] [--description DESCRIPTION] [--slug SLUG] [--tags TAGS [TAGS ...]] [--include INCLUDE [INCLUDE ...]] [--no_solutions] [--yes] manifest

The --include option can be used to add other files the course needs (e.g. datasets), and --no_solutions removes the solution files from the tasks (if they existed). --version, --description, and --tags are stored in the manifest, so if a course is ever re-packed it will re-use the given values. --slug is the name a course is pulled by, and it is derived from the course's name unless an explicit name is provided via this option. --yes automatically accepts confirmation prompts.
pack also finds the packages the course uses besides pyics and the standard library (imported by its exercise and solution code, or needed for the values its tests store, like numpy arrays), and offers to declare those that aren't declared in the manifest's requires yet. A task file may only make PyICs import modules of the packages its course declares: any other module is refused before the task is loaded.

build indes lists the newest version of each course in the archives below the directory in index.json.

pyics [options] build_index [--out OUT] [--name NAME] [directory]

Students add its URL with pyics add_source. A course archive can also be pulled directly by its URL or path. An index looks like this, with the archives' URLs relative to it.

{
    "pyics_index": 1,
    "name": "Uni Bamberg AI courses",
    "courses": {
        "search-algorithms": {
            "title": "Search Algorithms",
            "description": "DFS and BFS",
            "version": "1.0.0",
            "requires_pyics": ">=0.0.1",
            "requires": ["numpy>=1.26"],
            "url": "search-algorithms-1.0.0.zip",
            "sha256": "...",
            "tags": ["graphs"]
        }
    }
}

Archives are checked before anything of them is used: they can't write files outsinde the directory they are pulled to, their has h has to match the index, and they can only name the modules of the packages they declare.

Jupyter Notebooks

• create_task and edit_task accept notebooks as exercise and solution files, and all tests work with them. SimpleTests and SpecTests find functions and classes by name, in whichever cell students define them. StructureTests check the code of all code cells: cells that can't be parsed are left out, and named in the message if the test fails. PrintTests compare the output of all cells. Both can be restricted to a part of the notebook instead (see Sections).
• The output of a notebook only has to contain the expected output of a PrintTest, rather than match it like the output of a Python file, since students also output other things in the cells of an exercise, e.g. the value of an expression they left at the end of a cell. Whitespace at the ends of lines is ignored, as usual. Short expected outputs are found easily: [a, b] is also found in [[a, b]], and 3 in 13, so expect enough to tell a right output from a wrong one, e.g. Sum: 3. A test that expects no output still fails if there is any.
• Notebooks are stored normalized in task files: without their outputs and execution counts, so that the outputs of solutions don't reach students and task files stay small. Their kernel (kernelspec) isn't stored either, as it names an environment on your machine: students are asked to select a kernel of their own when they first open or run the notebook. A notebook you merely ran, or ran with another kernel, doesn't count as changed when you edit the task. If it did change, edit_task shows the difference as the text of its cells (# %% cell 3 blocks) rather than as JSON. If only its metadata changed (e.g. the tags of its cells), it says so.
• A notebook, e.g. the solution notebook, can be the test information file. Its code cells run as they do in tests, and its variables are offered for the test's parameters, e.g. a function as the expected signature of a SpecTest. An error in a cell doesn't stop the run, and the prompts name the cell that raised it.
• The tests run in the directory PyICs was started in, while Jupyter runs a notebook in the notebook's directory: a file that a notebook opens by a relative path is looked up in different directories by the two.
• Keep the exercise notebooks in a visible directory, as examples/search_notebook does, rather than in .exercises/: JupyterLab only opens notebooks in hidden directories if it runs with --ContentsManager.allow_hidden=True.
• pack finds the packages imported in the code cells of notebooks, too. If the course's requires don't declare them, it shows the %pip install (and !pip install) lines of its notebooks as hints, e.g. lecture.ipynb (cell 2): %pip install numpy. These lines do nothing in tests, so the packages have to be declared. pack also warns about tasks that share a notebook, but store different contents for it (see Sections).
• An alternative that needs no notebook support is to pair a notebook with a Python file using Jupytext, and to use the .py file as the exercise file (magics are commented out in it). Students then need Jupytext in their Jupyter, and pairing tends to confuse beginners.

Sections link a task to a part of a notebook, e.g. to one exercise of a lecture's notebook that holds several.

• Where a section starts: at a cell tagged pyics:<name>, or at a cell whose first line is the marker <!-- pyics: <name> --> (a markdown cell) or # pyics: <name> (a code cell). Names consist of letters, digits, _, . and -, e.g. ex2. Tags are Jupyter's own way to mark cells, but hidden in most frontends. Markers are visible everywhere, and students can restore one they deleted.
• Where it ends: just before the next section starts, or at the end of the notebook. Cells above the first section belong to none. If several cells start a section of the same name (e.g. a copied cell), the first of them does.
• Where to put it: on the exercise's description, a markdown cell that students have no reason to change. The cells they add below it, split or move belong to the section, as long as they stay above the next one.
• A task's section: create_task and edit_task ask which section of each notebook exercise file the task is about (if the notebook has sections), and store it in the task file ("sections", by the notebook's path relative to the task file). Older versions of PyICs ignore it.
• Its tests: StructureTests check only the code cells of the section, and fail if one of them can't be parsed. PrintTests compare only the output of the section's cells, while the whole notebook still runs, as a section usually needs the imports and data above it. SimpleTests and SpecTests find functions and classes by name, wherever they are defined, and aren't affected by sections.
• A test's own section: StructureTests and PrintTests use the task's section, unless they are given one of their own: section="ex1" for another section, or section=None for the whole notebook. create_task offers The task's section, Whole notebook and the notebook's other sections. A test that uses the task's section stores nothing, so it follows when the task's section is changed.
• Finding it: the title of the main panel names the task's section after the notebook's path, e.g. lecture.ipynb › Exercise 2: Breadth-First Search: the first heading of the cell that starts it, or else its name, looked up in the saved notebook. JupyterLab's Table of Contents and VS Code's Outline (ctrl+shift+o) list the notebook's headings.
• A missing section fails its tests with an explanation, e.g. Section 'ex2' not found: the cell that starts it (tagged 'pyics:ex2', or starting with the marker '# pyics: ex2') was removed or its tag changed., and the title shows section 'ex2' not found. Tests never check another part of the notebook instead.
• Several tasks in one notebook: each task lists the same notebook among its exercise files, with its own section, and is solved on its own. The first of them that is set up creates the notebook (an existing exercise file is never overwritten), so they should all store the same contents for it: pack warns if they don't, and so does edit_task after writing a task (choose Update for the notebook to store it as it is on disk).

Technical Information

This project is managed using uv. If you want to contribute to the project, run uv sync after cloning the repository at the top level to create a virtual environment with the required packages installed.

Repository/App Structure

The src/pyics directory contains the following submodules:

• data which models Tasks and Courses
• scripts which contains the executable scripts
• tests which contains anything related to dynamic test objects (the core testing logic for student code)
• distribution which packs, publishes and pulls courses, and checks the packages they need
• notebook which reads Jupyter notebooks, runs their code cells as exercise files, and finds their sections
• plugins which loads plugins, packages that extend pyics through pluggy hooks, e.g. pyics-jupyter
• util containing logic for json handling and logging