stubalyzer 0.1.1

Analysis tool comparing hand written stubs to stubgen-generated ones, reporting inconsistencies

An analysis tool to compare handwritten mypy stubs to stubgen-generated stubs.

stubalyzer makes the process of crafting types for untyped dependencies easier. Some example inconsistencies which are found and reported by stubalyzer are listed below:

• handcrafted and generated stubs have different method signatures

• generated stubs are missing functions/class members

• generated types are more specific than the handcrafted types

How to use?

Installation

Install stubalyzer with the following command:

$ pip install stubalyzer

How to run

Run stubalyzer with the following command:

$ stubalyzer -h
usage: stubalyzer [-h] -c CONFIG [-e EXPECTED_MISMATCHES] [-r REFERENCE_STUBS]
                  STUBS_HANDWRITTEN

Analyze a set of (handcrafted) mypy stubs by comparing them to (generated)
reference stubs

required arguments:
-c CONFIG, --config CONFIG
                        Mypy config file

positional arguments:
STUBS_HANDWRITTEN       Directory of handwritten stubs that need to be
                        analyzed

optional arguments:
  -h, --help            show this help message and exit
  -e EXPECTED_MISMATCHES, --expected-mismatches EXPECTED_MISMATCHES
                        A JSON file, which defines expected mismatching
                        symbols and their match results. If any symbol is
                        declared in an expected_mismatches JSON file,
                        stubalyzer will count it as an expected failure, and
                        ignore this inconsistency.

                        Example contents:
                        {
                           "my.module.function: "mismatch",
                           "another.module.Class: "not_found"
                        }

                        According to the example above, we expect the signature
                        of my.module.function to mismatch, and module.Class to
                        be missing in the generated stubs. stubalyzer will
                        ignore these inconsistencies.
  -r REFERENCE_STUBS, --reference REFERENCE_STUBS

                        Directory of reference stubs to compare against. If
                        not specified stubgen will be used to generate the
                        reference stubs.

Output

If the comparison ends successfully with zero inconsistencies, stubalyzer will print a success message to stdout, alongside with an ignore message.

The ignore message includes the number of failures ignored, which are declared as expected in the file for expected mismatches. If this file is not provided, the ignore message will list 0 ignored stubs.

Successfully validated 68 stubs.

0 fail(s) were ignored.

If there are mismatches in the given types, stubalyzer will print a list of all inconsistencies with a result message, alongside with an ignore message similar to the following:

Symbol "vars.any_var" not found in generated stubs.

Types for functions.additional_args do not match:

   def (foo: builtins.int, bar: builtins.int) -> builtins.str

   def (foo: builtins.int) -> builtins.str


Failure: 33 of 68 stubs seem not to be valid.

0 more fail(s) were ignored.

Development

The following section contains instructions on how to set up and use the development environment for this project.

Development Setup

Requirements for development:

• A recent Python version (we currently use 3.7)

• virtualenv and virtualenvwrapper

For a development setup, run the following shell script:

$ ./dev/setup.sh

This will create a virtual environment called stubalyzer and install the projects dependencies. The setup script also creates a .venv file so the environment activates automatically if you use auto-activation with virtualenv.

Tests

Tests are run using pytest:

$ pytest

Type Checking

Type checking is done with Mypy:

$ mypy stubalyzer

Code Formatting

To set up the pre-commit hook to automatically format files, create the following link:

$ ln -sf ../../dev/pre-commit.sh .git/hooks/pre-commit

The source code is formatted using black and isort. The following will format all files in the project:

$ ./dev/fmt.sh

Linting

Linting is done using flake8, in the root directory run:

$ flake8

Dependency Management

If you need new dependencies, add them in requirements.in and setup.py, then run the pip-compile command specified at the top of requirements.txt.

Documentation

The documentation is written using Sphinx.

First install the requirements:

$ pip install -r docs/requirements.txt

Then build the documentation using:

$ cd doc; make html

The output will be in docs/_build/html/index.html.

You can update the API documentation using the following:

$ ./dev/update-apidoc.sh