Skip to main content

JPTest

JPTest is a unit testing framework for Jupyter Notebooks and aims for fast test writing in less lines of code. It creates the possibility to score and automatically grade exams with separate notebook (.ipynb) and test (.py) files.

JPTest uses testbook by nteract for interaction with notebooks. While we mainly focus on grading, testbook might fit your use case better. So please feel free to watch this presentation on YouTube by Rohit Sanjay if you want to know more about testbook.

Installation

JPTest relies on testbook and Jupyter. If you need any other libraries for executing your notebook cells they can be installed in your environment the usual way.

The preferred way to use JPTest is in a virtual environment:

python -m venv venv
source venv/bin/activate

Use pip to download and install JPTest:

pip install jptest

Usage

You may use the decorator @JPTest with any function within a test file. It accepts the following parameters. Any other named parameters are copied to the JPTestBook object.

name type description
name str name used in the output
max_score float maximum score (can be exceeded, used to calculate total score)
execute see below cells or code to execute prior to the test
timeout int execution timeout in seconds (default: 2 minutes)

yield can be used to grant points:

yield condition: bool, value: float, [comment if True: str, [comment if False: str]]

Run the jptest module with the notebook and test file as parameters. Every test with the decorator is automatically discovered and executed.

python -m jptest notebook.ipynb tests.py

For more parameters see the included help:

python -m jptest --help

Useful Functions

name description
ref get a reference to an object or function
get like ref, but copy to a name with a random suffix before

Examples

a simple unit test

a simple unit test

# Mark as unit test and execute the cell with the tag `task-1`
# prior to this test function.
@JPTest('Task 1', max_score=2.0, execute=('task-1',))
def test_task1(tb: JPTestBook):
    # Use `ref` to reference an object or function from
    # the notebook.
    fib_from_nb = tb.ref('fib')

    # Grant points if output matches.
    yield fib_from_nb(0) == 0, 0.5
    yield fib_from_nb(1) == 1, 0.5
    yield fib_from_nb(2) == 1, 0.5
    yield fib_from_nb(20) == 6765, 0.5

independent grading

independent grading

@JPTest('Task 2', max_score=1.0, execute=[
    # Inject a correct `fib` function instead of executing the
    # cell with the tag `task-1`.
    '''
    def fib(n):
        if n <= 1:
            return n
        else:
            return fib(n-1) + fib(n-2)
    ''',
    # Execute the cell with the tag `task-2` only.
    ('task-2',)
])
def test_task2(tb: JPTestBook):
    yield tb.ref('fib_list') == [0, 1, 1, 2, 3, 5, 8, 13, 21, 34], 1.0

track function calls

@JPTest('Bonus Point', max_score=2.0, execute=('task-1',))
def test_recursive(tb: JPTestBook):
    fib_from_nb = tb.ref('fib')

    # Track calls of `fib` function and store them into `params`
    # after the context manager's `__exit__` function is executed.
    with tb.track('fib') as params:
        fib_from_nb(20)

    # Award a bonus point if the function is recursive.
    yield len(params) > 1000, 1.0, None, 'recursion... neat!'

equality functions

equality functions

Some objects like pandas dataframes are neither serializable nor comparable via ==. One can call functions like equals on references instead.

@JPTest('Task 3', max_score=2.0, execute=('task-3',))
def test_task3(tb: JPTestBook):
    # Store `df` with a random suffix and get a reference.
    result = tb.get('df')

    # Inject sample solution and get a reference.
    test = tb.inject('df = pd.read_csv("data.csv", ...)', 'df')

    # Call `equals` function on one of the dataframes.
    yield test.equals(result), 1.0

    # This also works with imported modules.
    # yield, tb.ref('nx').could_be_isomorphic(test, result), 1.0

Execute Parameter

One can control in detail what code is executed prior to a test. Therefore the execute parameter accepts different types, which can also be nested recursively.

String

If the parameter is of type str, the value is considered as code and injected into the notebook.

Tuple

If the parameter is of type tuple, the value is considered as tags. If there is one element in the tuple, every cell with this tag is executed. If there are two elements in the tuple, every cell between the first appearence of the first tag and the first appearence of the second tag (including) is executed.

Dictionary

If the parameter is of type dict, it must include a key named execute, which value is executed following the rules stated in this section.

It may include a key named track, which value is another dictionary with function names as key and a tuple or a list of tuples containing the parameters to track. The result is passed to the test function as a parameter in the order of the execution.

Function

If the parameter is a function (Callable), it will be called with the JPTestBook object as a parameter.

List

If the parameter is of type list, every element will be executed in the order of its appearence, following the rules stated above.

Metadata

Release files for jptest 0.8.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for jptest 0.8.5
File Size Uploaded Provenance Actions
jptest-0.8.5.tar.gz 11.6 kB Without provenance Details for jptest-0.8.5.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for jptest 0.8.5
File Interpreter ABI Platform Actions
jptest-0.8.5-py3-none-any.whl Python 3 none any Details for jptest-0.8.5-py3-none-any.whl

Total release size: 22.6 kB

Release files / jptest-0.8.5.tar.gz

Download URL jptest-0.8.5.tar.gz
Size 11.6 kB
Tags Source
SHA-256 checksum
How to use checksums
1accff011b44a5584076464f60c051bde3e2580dc730d3d8e5ef525e756d815c
BLAKE2b-256 checksum
How to use checksums
cf441d45fb964b6ce74e44d1655510bd2adc45c54d784803f9ccec89abb108fa
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.1 CPython/3.8.10

Release files / jptest-0.8.5-py3-none-any.whl

Download URL jptest-0.8.5-py3-none-any.whl
Size 11.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
eaf1e3b9d9236d706278d8b9c24e6f5e27c2f18f3b49809011e2fbfea4c3519f
BLAKE2b-256 checksum
How to use checksums
b71f681aebd496920e35fc76cabb85ef0e417a8f8f7426e185e6b5f5d8115956
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.1 CPython/3.8.10

Security for jptest 0.8.5

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page