pytest-autoprofile 0.16.0

`line_profiler.autoprofile`-ing your `pytest` test suite

pytest-autoprofile

No-fuss auto-profiling of your pytest tests

Table of contents

Click to expand

[[TOC]]

Motivation

Leveraging the pre-existing test suite can be a good start for some quick-and-dirty benchmarking and profiling. ^{[citation needed]} While existing solutions like pytest-line-profiler already serve to bridge pytest and line_profiler, they aren't using newer features of the latter like auto-profiling, which mitigates the need to explicitly supply the profiled items via either the command line, or shudders changing the source code to insert the @profile decorator. And what's a quicker and dirtier way to do your profiling, than to just feed the whole test suite to the profiler and see what happens?

Requirements

• python >= 3.8
• pytest >= 7.0
• pluggy >= 1.2
• line_profiler >= 5.0

Notes

If you're having issues with installation, try upgrading pip:

$ pip install --upgrade pip

Example

Click to expand

$ pytest --help
usage: pytest [options] [file_or_dir] [file_or_dir] [...]

...

auto-profiling options (requires `line-profiler`):
  --always-autoprof=MODULE.DOTTED.PATH[::OBJECT.DOTTED.PATH][,...]
                        ...
  --recursive-autoprof=[MODULE.DOTTED.PATH[,...]]
                        ...
  --postimport-autoprof=[MODULE.DOTTED.PATH[::OBJECT.DOTTED.PATH][,...]]
                        ...
  --autoprof-mod=MODULE.DOTTED.PATH[::OBJECT.DOTTED.PATH][,...]
                        ...
  --autoprof-imports=[yes|no]
                        ...
  --autoprof-tests=[yes|no]
                        ...
  --autoprof-doctests=[all|yes|no]
                        ...
  --autoprof-rewrite-doctests=[yes|no]
                        ...
  --autoprof-subprocs=[yes|no]
                        ...
  --autoprof-outfile=FILENAME
                        ...
  --autoprof-view=[yes|no|FLAGS]
                        ...
  --autoprof-global-profiler=[always|yes|no]
                        ...

...

$ pytest --verbose --verbose --verbose \
> --autoprof-imports --autoprof-tests --autoprof-doctests \
> --always-autoprof=my_pkg.funcs::foo --autoprof-view=-tm
================================= test session starts ==================================

...

==================================== auto-profiling ====================================

pytest_autoprofile.importers.ProfileTestsImporter: rewrote module (1):

  test_misc

pytest_autoprofile.importers.ProfileModulesImporter: rewrote module (1):

  my_pkg.funcs

Doctests (5) profiled in files (2):

  packages/my_pkg/classes.py (2 doctests):
    my_pkg.classes.SomeClass, my_pkg.classes.SomeClass.instance_method

  packages/my_pkg/funcs.py (3 doctests):
    my_pkg.funcs, my_pkg.funcs.bar, my_pkg.funcs.foo

Doctests (2) omitted from profiling output in file (1):

  packages/my_pkg/funcs.py (2 doctests):
    my_pkg.funcs.baz, my_pkg.funcs.foobar

Wrote profile results to .pytest_cache/pytest_autoprofile.lprof

...
Function: foo at line 20

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    20                                           def foo():
    21                                               """This test has a statically-defined doctest which will be
    22                                               profiled. The docstring is also formatted differently (no leading
    23                                               newline) to test if it has an effect (which it shouldn't) on
    24                                               profiling and collection.
    25                                           
    26         1          2.0      2.0     33.3      >>> x = foo() + 1
    27         1          0.0      0.0      0.0      >>> assert x == 2
    28                                           
    29                                               Doctest chunk containing a function definition:
    30                                           
    31         1          0.0      0.0      0.0      >>> def foofoo(s):
    32                                               ...     stop = foo()
    33                                               ...     return s[:stop]
    34                                               ...
    35                                               >>>
    36         1          3.0      3.0     50.0      >>> foofoo('string')
    37                                               's'
    38                                               """
    39         3          1.0      0.3     16.7      return 1

...

=============================== short test summary info ================================
...

How it works

The meta-path finders-slash-loaders ~.importers.AutoProfImporter rewrite modules/packages and tests on import, much like how pytest also does its own rewrites for assertions in tests. A specialized doctest.DocTestRunner subclass is used for doctest profiling and rewriting.

Module rewriting

Modules and packages are rewritten with ~.importers.ProfileModulesImporter, which does profiling on full module objects and/or specific objects therein with ~.rewriting.StrictModuleRewriter.transform_module(). This allows for more fine-grained control over module profiling than what AstTreeModuleProfiler affords, before they are even imported by tests, and regardless of whether they are directly imported or not.

• --always-autoprof (dotted-path flag; default: nil):
  importable targets to auto-profile throughout the entire test session, not only when imported by individual tests:
  • If a module-level target, all function and method definition local to the module (see Notes) are rewritten on import to be decorated with @line_profiler.profile, like what AstProfileTransformer does;
  • Otherwise (i.e. if the target has an object part), it is either decorated as above (if a function/method/class definition local to its module), or via (a patched version of) line_profiler.LineProfiler.add_imported_function_or_module() (original, patched).
• --recursive-autoprof (dotted-path flag (optional); default: nil):
  module- or package-name targets to auto-profile as with above, but inclusive of all their sub-modules and -packages; can also be supplied without the argument, which indicates that all module-level targets in --always-autoprof are to be treated recursively.

Notes

• Only targets accessible from the module-local namespace are decorated:

  def some_func():  # This is decorated
      def nested_func(): ...  # This isn't
  
  
  class SomeClass:
      def some_method(self):  # This is decorated
          def nested_method(): ...  # This isn't
  
      class SomeInnerClass:
          def some_inner_method(self):  # This is decorated
              def nested_inner_method(): ...  # This isn't
  

• If a target in --recursive-autoprof has an object part, it is transferred to --always-autoprof with a warning:

  $ pytest --recursive-autoprof=foo.bar::baz,spam,ham::eggs ...
  ...
  ...: OptionParsingWarning: --recursive-autoprof: found 2 invalid (non-module) targets (foo.bar::baz,ham::eggs), moving them to `--always-autoprof`
  ...
  

• --recursive-autoprof resolves to True (i.e. all --always-autoprof module targets should be profiled recursively) only if all of the passed --recursive-autoprof flags are in the no-argument form, so

  $ pytest --always-autoprof=foo --recursive-autoprof=bar --recursive-autoprof
  

  profiles bar recursively but not foo.

Test rewriting

Test are rewritten with ~.importers.ProfileTestsImporter, which does the same rewrites on tests as for scripts run with kernprof -l --prof-mod=... [--prof-imports] ..., using AstTreeProfiler.profile(). This is on top of the pytest assertion rewrites (when appropriate), so rich comparisons in assertions (or the lack thereof) work the same as otherwise.

• --autoprof-tests (boolean flag; default: False):
  whether to profile entire test modules; equivalent to adding each test file to --autoprof-mod as it is run.
• --autoprof-mod (dotted-path flag: default: nil):
  equivalent to kernprof -l's --prof-mod flag; if any of those targets is directly imported in the tests, it is profiled with the facilities of line_profiler.
• --autoprof-imports (boolean flag; default: False):
  equivalent to kernprof -l's --prof-imports flag; if an entire test module is already profiled (via --autoprof-tests or explicit inclusion in --autoprof-mod), also profile all its imports with line_profiler.

Doctest profiling

Doctests are line-profiled as they are executed with ~._doctest.get_runner_class() (which builds a doctest.DocTestRunner subclass, like what _pytest.doctest._init_runner_class() does); optionally, it also does the same import and function-/method-definition rewrites according to --autoprof-mod and --autoprof-imports as with normal tests.

• --autoprof-doctests (extended boolean flag with special value 'all'; default: False):
  • 'all', 'A', etc.:
    Profile all the collected doctests that are tractable, similar to how --autoprof-tests works for regular tests; this is the value defaulted to (instead of true) when using the zero-argument form of the option
  • True:
    Profile collected tractable doctests if they belong to modules, classes, functions, etc. covered by --autoprof-mod
  • False:
    Don't profile doctests
• --autoprof-rewrite-doctests (boolean flag; default False) whether to profile imports (resp. function/method definitions) in doctests via AST rewriting if resp.:
  • --autoprof-mod or --autoprof-doctests indicate that the Python files they reside in should have function/method definitions rewritten
  • --autoprof-mod or --autoprof-imports indicate that the specific imports in the Python files they reside in should be rewritten

Notes

• "Tractable" doctests are those that

  • Reside in bodied Python objects (i.e. modules and function/method/class definitions), and
  • Defined literally as the first expressions in said bodies.
  Click to expand

  def some_function():  # This doctest can be profiled...
      """
      >>> something = ...
      >>> some_function(something)
      """
  
  
  def other_function():
      ...
  
  
  # But this can't since it defies static analysis
  other_function.__doc__ = f'>>> other_function()\n{expected_output!r}'
  
  
  def add_doc(doc):
      def wrapper(func):
          func.__doc__ = doc
          return func
  
      return wrapper
  
  
  @add_doc("""
  >>> yet_another_function()
  SOME_EXPECTED_OUTPUT
  """)
  def yet_another_function():  # And neither can this
      ...
  

  Only tractable doctests are included in the profiling output, because line_profiler expects the profiled files to be Python source files, parsing their lines into snippets consisting of blocks of bodied objects to format its output. As such, profiled doctests must be able to be attributed to specific lines in a Python source file with static analysis.

• Compatibility with other doctest plugins is limited:

  • pytest-doctestplus >= 0.13 is partially supported: get_runner_class() checks at runtime whether its doctest facilities or that of vanilla pytest's should be used, and sets up the appropriate overrides; of course, the aforementioned restrictions on tractable tests still apply.
  • xdoctest will probably never be supported since it reimplements doctest and fundamentally functions independently and differently therefrom.
  • And as discussed above, profiling is not possible for doctests residing in various non-Python files (like .rst and .md files) discovered by the various doctest-related plugins, including pytest-doctestplus.

Subprocess profiling

Both forked and spawned Python subprocesses can be profiled, setting up the same profiling targets as outlined in Module rewriting and Test rewriting, and writing the profiling data on exit which are later collated. This is achieved by:

• Writing the appropriate (temporary) .pth file so that spawned subprocesses install the appropriate hooks,
• Monkey-patching os.fork() to set up profiling for forked processes, and
• Monkey-patching multiprocessing to ensure the writing of profiling data after code is executed in subprocesses.

See however especially the following notes.

• --autoprof-subprocs (boolean flag; default: False):
  whether to profile subprocesses.

Notes

• Since this writes a .pth file, write permission is needed for the directory sysconfig.get_path('purelib') points to.
• To avoid affecting other Python processes using the same installation paths, the environment variable ${PYTEST_AUTOPROFILE_TEST_PID} is (temporarily) set so that only subprocesses inheriting it are affected by said .pth file, and the file itself is removed as soon as the test session terminates. Still, this means extra code is imported and executed for all Python interpreters on startup throughout the (short) lifetime of the file, and has obvious performance implications.
• To ensure that profiling data is correctly gathered from multiprocessing subprocesses, one should take care to properly finalize them, e.g. by explicitly .close()-ing and .join()-ing process pools (see also coverage.py's caveat on multiprocessing). Otherwise, incomplete profiling data may be written and temporary files may not be properly cleaned up.

Output

The .lprof file and terminal output is only written if any actual function, method, property, code, etc. has been passed to the profiler (or profilers in subprocesses if --autoprof-subprocs is set).

• --autoprof-outfile (default: <root_dir> / <cache_dir> / 'pytest_autoprof.lprof'):
  filename to which the profiling data should be written, equivalent to kernprof -l's --outfile flag.
• --autoprof-view (extended boolean flag also taking python -m line_profiler options (see below); default: False):
  equivalent to kernprof -l's --view flag, showing the profiling results at the end of the test session. Can also be a string to be shlex.split() into the CLI options for python -m line_profiler, which then causes the results to be displayed as if the .lprof file has been passed thereto with said options; valid options are:
  • -c CONFIG/--config=CONFIG, --no-config:
    Load configuration from the provided file (or the default config file that ships with line_profiler if --no-config); if any of the following flags/flag pairs is not passed, the value is resolved therefrom
  • -u UNIT/--unit=UNIT:
    Set the output_unit argument (positive finite real number) for the LineProfiler.print_stats() call
  • -z/--skip-zero, --no-skip-zero:
    Whether to set stripzeros=True for said call
  • -r/--rich, --no-rich; -t/--sort, --no-sort; -m/--summarize, --no-summarize:
    Whether to set the synonymous arguments to True for said call

Notes

• <root_dir> refers to the pytest root directory, which usually is where your project specs are.
• <cache_dir> is '.pytest_cache' unless you otherwise configured pytest.
• At neutral verbosity level or above, the filename where the profiling data is written to (if any) is always shown at the end of the test session unless it is explicitly specified by the --autoprof-outfile flag.
• If --autoprof-view doesn't resolve to false and the profiler has been used, the terminal output is always written regardless of verbosity level.
• If none of -c/--config/--no-config is specified in --autoprof-view, the configuration is loaded from the default resolved location (see line_profiler.toml_config).

Miscellaneous options

• --postimport-autoprof (dotted-path flag (optional); default: nil):
  importable targets to auto-profile throughout the entire test session like --always-autoprof and --recursive-autoprof, except that instead of rewriting entire modules at import time, profiling targets are explicitly imported then profiled; can also be supplied without the argument, which indicates that all targets in --always-autoprof and --recursive-autoprof are to be profiled post-import.
• --autoprof-global-profiler (extended boolean flag with special value 'always'; default: False):
  • 'always', 'A', etc.:
    Profile everything passed to @line_profiler.profile, regardless of whether it is .enabled or not; this is the value defaulted to (instead of true) when using the zero-argument form of the option
  • True:
    Profile everything passed to @line_profiler.profile when it is .enabled
  • False:
    Don't profile objects passed to @line_profiler.profile (unless it is otherwise already included in profiling by e.g. module, test, or doctest rewriting)

Notes

• When supplied with arguments, --postimport-autoprof infers which module/package targets to recurse into and which not to:
  • If the target is also found in --recursive-autoprof (or is implied by the no-argument form thereof), the module is recursed into.
  • Else, if the target is also found in --always-autoprof, it is taken to be explicitly specified to not be recursed into.
  • Otherwise, it is recursed into.
• --postimport-autoprof is in a sense highly overlapping with --always-autoprof and --recursive-autoprof in function, but it covers the following corner cases:
  • When modules have already been loaded and for whatever reasons cannot be unloaded at the beginning of the test session. This can happen e.g. with Cython modules, which persists even after a call to importlib.invalidate_caches().
  • When profiling line_profiler itself or its components.
• The default behavior prior to v0.12.0 was roughly equivalent to --autoprof-global-profiler=always; however, since it altered the internal state of @line_profiler.profile (replacing its inner LineProfiler instance) using the same hook that kernprof uses, the normal outputs (e.g. profile_output.txt, profile_output.lprof, and stdout print-outs) written by @line_profiler.profile were suppressed. This is no longer the case, and @line_profiler.profile now functions identically (up to overhead) between when this plugin is used or not.

Boolean parsing

If a flag can be supplied without arguments, doing so is equivalent to setting it to true. If one does supply an argument, it should be any of the following (case-insensitive):

truey_strings = {'1', 'T', 'True', 'Y', 'yes'}
falsy_strings = {'0', 'F', 'False', 'N', 'no'}

Dotted-path parsing

Dotted paths consist of a dotted module part, and an optional sub-module-level dotted attribute-access (object) part, separated from the module part. As examples, a path like foo.bar.baz should correspond to the module importlib.import_module('foo.bar.baz'), while a path like foo.bar::Baz.foobar represents the object operator.attrgetter('Bar.foobar')(importlib.import_module('foo.bar')).

Multiple paths can be supplied both:

• Together and joined with commas, and/or
• By passing multiple copies of the corresponding dotted-path flag, like

  $ pytest --always-autoprof=foo.bar::baz,foobar --always-autoprof=spam.eggs
  

Notes

These semantics are a bit different and somewhat limited, compared with those of pytest-line-profiler's --line-profile flag and kernprof's --prof-mod flag resp.:

• File paths are not accepted.
• The explicit separation of the module and object parts with '::' is required, instead of using '.' as the separator (thus not distinguishing between the parts) and only later inferring which part is the module and which exists under it.

Default hooks

Each of the command-line option added to pytest corresponds to a hook function, through which users (through conftest.py) or other plugins can provide alternative default values to the options (see ~.option_hooks):

• --always-autoprof: pytest_always_autoprof_default()
• --recursive-autoprof: pytest_recursive_autoprof_default()
• --postimport-autoprof: pytest_postimport_autoprof_default()
• --autoprof-tests: pytest_autoprof_tests_default()
• --autoprof-mod: pytest_autoprof_mod_default()
• --autoprof-imports: pytest_autoprof_imports_default()
• --autoprof-doctests: pytest_autoprof_doctests_default()
• --autoprof-rewrite-doctests: pytest_autoprof_rewrite_doctests_default()
• --autoprof-subprocs: pytest_autoprof_subprocs_default()
• --autoprof-outfile: pytest_autoprof_outfile_default()
• --autoprof-view: pytest_autoprof_view_default()
• --autoprof-global-profiler: pytest_autoprof_global_profiler_default()

Notes

Unfortunately, the defaults supplied by conftest.py and plugins are not (reliably) available at the time that the command-line option parser is created (see pytest issue #13304). Therefore, the default values of the options are left out of their pytest --help blurbs.

Tests

(Refer to the latest Pipelines.)

Tests are currently done in the following environments. Compatibility with other environments should be reasonable, but is not guaranteed; write me an issue if anything weird comes up.

Stacks

Component\Stack name	py3.8 ("oldest")	py3.11 ("middle")	py3.13 ("newest")
python	3.8.20 [Note 1]	3.11.10 [Note 1]	3.13.3
pytest	7.0.1	8.0.2	8.4.1
pluggy	1.2.0	1.3.0	1.5.0
pytest-doctestplus	0.13.0	1.0 [Note 2]	1.4.0

Platforms

OS	CI?	Machine	Notes
Linux	yes	saas-linux-small-amd64
Windows	yes	saas-windows-medium-amd64	1
macOS	no	Yours truly's M3 Mac	2, 3

Notes

1. The Python versions are different on Windows due to the unavailability of the target versions on NuGet:
   • "oldest": 3.8.20 → 3.8.10
   • "middle": 3.11.10 → 3.11.9
2. On other platforms, the test suite is always run both with and without pytest-doctestplus installed; on macOS, the test suite is always run without pytest-doctestplus on the "middle" stack and with it on the others.
3. I'd love to run CI pipelines for that too, but GitLab's macOS rate is too high... so just trust me bro. ^{[citation needed]}

Limitations

• There are no .ini-file-style equivalents for the command-line flags.
• Compatibility with other pytest plugins may be limited; see the Notes on Doctest profiling.
• Care must be taken when profiling code called in subprocesses, esp. via multiprocessing; see the caveat on Subprocess profiling.

Acknowledgements

This plugin makes use of, refers to, or is inspired by (in alphabetical order):

• coverage.py
• line_profiler
• pytest
• pytest-doctestplus
• pytest-line-profiler