pytest plugin for testing mypy types, stubs, and plugins

Installation

This package is available on PyPI

pip install pytest-mypy-plugins

and conda-forge

conda install -c conda-forge pytest-mypy-plugins

Usage

Running

Plugin, after installation, is automatically picked up by pytest therefore it is sufficient to just execute:

pytest

Asserting types

There are two ways to assert types. The custom one and regular typing.assert_type.

Our custom type assertion uses reveal_type helper and custom output matchers:

- case: using_reveal_type
  main: |
    instance = 1
    reveal_type(instance)  # N: Revealed type is 'builtins.int'

This method also allows to use # E: for matching exact error messages and codes.

But, you can also use regular assert_type, examples can be found here.

Paths

The PYTHONPATH and MYPYPATH environment variables, if set, are passed to mypy on invocation. This may be helpful if you are testing a local plugin and need to provide an import path to it.

Be aware that when mypy is run in a subprocess (the default) the test cases are run in temporary working directories where relative paths such as PYTHONPATH=./my_plugin do not reference the directory which you are running pytest from. If you encounter this, consider invoking pytest with --mypy-same-process or make your paths absolute, e.g. PYTHONPATH=$(pwd)/my_plugin pytest.

You can also specify PYTHONPATH, MYPYPATH, or any other environment variable in env: section of yml spec:

- case: mypy_path_from_env
  main: |
    from pair import Pair

    instance: Pair
    reveal_type(instance)  # N: Revealed type is 'pair.Pair'
  env:
    - MYPYPATH=../fixtures

What is a test case?

In general each test case is just an element in an array written in a properly formatted YAML file. On top of that, each case must comply to following types:

(*) Appendix to pseudo types used above:

class File:
    path: str
    content: Optional[str] = None
Parameter = Mapping[str, Any]

Implementation notes:

• main must be non-empty string that evaluates to valid Python code,
• content of each of extra files must evaluate to valid Python code,
• parametrized entries must all be the objects of the same type. It simply means that each entry must have exact same set of keys,
• skip - an expression set in skip is passed directly into eval. It is advised to take a peek and learn about how eval works.
• marks - custom markers should be registered in your pytest.ini / pyproject.toml just like any other pytest marker. Unregistered markers emit pytest's PytestUnknownMarkWarning and fail under --strict-markers.

Repository also offers a JSONSchema, with which it validates the input. It can also offer your editor auto-completions, descriptions, and validation.

All you have to do, add the following line at the top of your YAML file:

# yaml-language-server: $schema=https://raw.githubusercontent.com/typeddjango/pytest-mypy-plugins/master/pytest_mypy_plugins/schema.json

Example

1. Inline type expectations

# typesafety/test_request.yml
- case: request_object_has_user_of_type_auth_user_model
  main: |
    from django.http.request import HttpRequest
    reveal_type(HttpRequest().user)  # N: Revealed type is 'myapp.models.MyUser'
    # check that other fields work ok
    reveal_type(HttpRequest().method)  # N: Revealed type is 'Union[builtins.str, None]'
  files:
    - path: myapp/__init__.py
    - path: myapp/models.py
      content: |
        from django.db import models
        class MyUser(models.Model):
            pass

2. @parametrized

- case: with_params
  parametrized:
    - val: 1
      rt: builtins.int
    - val: 1.0
      rt: builtins.float
  main: |
    reveal_type({{ val }})  # N: Revealed type is '{{ rt }}'

Properties that you can parametrize:

• main
• mypy_config
• out

3. Longer type expectations

- case: with_out
  main: |
    reveal_type('abc')
  out: |
    main:1: note: Revealed type is 'builtins.str'

4. Regular expressions in expectations

- case: expected_message_regex_with_out
  regex: yes
  main: |
    a = 'abc'
    reveal_type(a)
  out: |
    main:2: note: .*str.*

5. Regular expressions specific lines of output.

- case: expected_single_message_regex
  main: |
    a = 'hello'
    reveal_type(a)  # NR: .*str.*

6. Markers

- case: test_slow_inference
  marks:
    - slow
  main: |
    reveal_type('abc')  # N: Revealed type is 'builtins.str'

Register the marker in your pytest.ini / pyproject.toml, then run only the marked cases with pytest -m slow:

[pytest]
markers =
    slow: slow tests

Options

mypy-tests:
  --mypy-testing-base=MYPY_TESTING_BASE
                        Base directory for tests to use
  --mypy-pyproject-toml-file=MYPY_PYPROJECT_TOML_FILE
                        Which `pyproject.toml` file to use
                        as a default config for tests.
                        Incompatible with `--mypy-ini-file`
  --mypy-ini-file=MYPY_INI_FILE
                        Which `.ini` file to use as a default config for tests.
                        Incompatible with `--mypy-pyproject-toml-file`
  --mypy-same-process
                        Run in the same process. Useful for debugging,
                        will create problems with import cache
  --mypy-extension-hook=MYPY_EXTENSION_HOOK
                        Fully qualified path to the extension hook function,
                        in case you need custom yaml keys. Has to be top-level
  --mypy-no-silence-site-packages
                        mypy will not silence errors from site-packages
  --mypy-modify-pythonpath
                        When running mypy in a subprocess, modify `PYTHONPATH`
                        to force packages to be treated as site-packages.
                        Incompatible with `--mypy-same-process`
  --mypy-closed-schema
                        Use closed schema to validate YAML test cases,
                        which won't allow any extra keys
                        (does not work well with `--mypy-extension-hook`)

Further reading

• Testing mypy stubs, plugins, and types

License

MIT