pyflyby 1.11.1

Installation

$ pip install pyflyby

This creates an alias for your ipython named py which runs the pyflyby plug internally.

pyflyby has a dependency on ipython, if it isn’t already installed do install it with:

$ pip install ipython

Quick start: Autoimporter + IPython

$ py
In [1]: re.search("[a-z]+", "....hello...").group(0)
[PYFLYBY] import re
Out[1]: 'hello'

In [2]: scoreatpercentile(arange(5), 50)
[PYFLYBY] from numpy import arange
[PYFLYBY] from scipy.stats import scoreatpercentile
Out[2]: 2.0

To load pyflyby into an existing IPython session as a 1-off:

$ ipython
In [1]: %load_ext pyflyby

To configure IPython/Jupyter Notebook to load pyflyby automatically:

$ py pyflyby.install_in_ipython_config_file

or

$ echo 'c.InteractiveShellApp.extensions.append("pyflyby")' \
  >> ~/.ipython/profile_default/ipython_config.py

$ ipython
In [1]: b64decode('aGVsbG8=')
[PYFLYBY] from base64 import b64decode
Out[1]: b'hello'

from pyflyby import add_import

add_import("foo", "foo = 1")

add_import(
    "df, data as dd",
    '''
    import pandas as pd
    data = [1,2,3]
    df =  pd.DataFrame(data)
''')

Quick start: py command-line multi-tool

$ py b64decode aGVsbG8=
[PYFLYBY] from base64 import b64decode
[PYFLYBY] b64decode('aGVsbG8=', altchars=None)
b'hello'

$ py log2 sys.maxsize
[PYFLYBY] from numpy import log2
[PYFLYBY] import sys
[PYFLYBY] log2(9223372036854775807)
63.0

$ py 'plot(cos(arange(30)))'
[PYFLYBY] from numpy import arange
[PYFLYBY] from numpy import cos
[PYFLYBY] from matplotlib.pyplot import plot
[PYFLYBY] plot(cos(arange(30)))
<plot>

$ py 38497631 / 13951446
2.7594007818257693

$ py foo.py

Quick start: tidy-imports

To use tidy-imports, just specify the filename(s) to tidy.

For example:

$ echo 're.search("[a-z]+", "....hello..."), scoreatpercentile(arange(5), 50)' > foo.py

$ tidy-imports foo.py
--- /tmp/foo.py
+++ /tmp/foo.py
@@ -1 +1,5 @@
+from   numpy                    import arange
+from   scipy.stats              import scoreatpercentile
+import re
+
 re.search("[a-z]+", "....hello..."), scoreatpercentile(arange(5), 50)

Replace /tmp/foo.py? [y/N]

To exclude a file, use –exclude <pattern>.

$ tidy-imports --tidy-local-imports foo.py

import os  # tidy-imports: ignore-import

from mypackage import _setup_hooks  # tidy-imports: ignore-import

def foo():
    import pdb  # tidy-imports: ignore-import
    ...

Quick start: import libraries

Create a file named .pyflyby with lines such as

from mypackage.mymodule import MyClass, my_function
import anotherpackage.anothermodule

You can put this file in your home directory or in the same directory as your *.py files.

Details: automatic imports

AUTOMATIC IMPORTS - never type “import” again!

This module allows your “known imports” to work automatically in your IPython interactive session without having to type the ‘import’ statements (and also without having to slow down your Python startup with imports you only use occasionally).

Example:

In [1]: re.search("[a-z]+", "....hello...").group(0)
[PYFLYBY] import re
Out[1]: 'hello'

In [2]: scoreatpercentile(arange(5), 50)
[PYFLYBY] from numpy import arange
[PYFLYBY] from scipy.stats import scoreatpercentile
Out[2]: 2.0

In [3]: np.sin(arandom(5))
[PYFLYBY] from numpy.random import random as arandom
[PYFLYBY] import numpy as np
Out[3]: [ 0.0282  0.0603  0.4653  0.8371  0.3347]

In [4]: isinstance(42, Number)
[PYFLYBY] from numbers import Number
Out[4]: True

$ ipython
In [1]: nump<TAB>
In [1]: numpy
In [1]: numpy.arang<TAB>
[PYFLYBY] import numpy
In [1]: numpy.arange

$ ipython
In [1]: arange?
[PYFLYBY] from numpy import arange
... Docstring: arange([start,] stop[, step,], dtype=None) ...

$ ipython
In [1]: %timeit np.cos(pi)
[PYFLYBY] import numpy as np
[PYFLYBY] from numpy import pi
100000 loops, best of 3: 2.51 us per loop

$ echo 'print(arange(4))' > foo.py
$ ipython
In [1]: %run foo.py
[PYFLYBY] from numpy import arange
[0 1 2 3]

Details: import libraries

Pyflyby uses “import libraries” that tell how to import a given symbol.

An import library file is simply a python source file containing ‘import’ (or ‘from … import …’) lines. These can be generated automatically with collect-imports and collect-exports.

from numpy import arange, nan

print(arange(10))

PYFLYBY_PATH=/etc/pyflyby:~/.pyflyby:.../.pyflyby

PYFLYBY_PATH=/foo1/bar1:/foo2/bar2

PYFLYBY_PATH=/foo1/bar1:-:/foo2/bar2

/etc/pyflyby/stuff.py
/u/quarl/.pyflyby/blah1.py
/u/quarl/.pyflyby/more/blah2.py
/proj/share/mypythonstuff/.pyflyby
/proj/share/mypythonstuff/foo/bar/.pyflyby/baz.py
/.pyflyby

/etc/pyflyby/stuff.py
/u/quarl/.pyflyby/blah1.py
/u/quarl/.pyflyby/more/blah2.py
/proj/share/mypythonstuff/foo/bar/.pyflyby/baz.py
/proj/share/mypythonstuff/.pyflyby (a file)

__forget_imports__ = ["from numpy import nan"]

__mandatory_imports__ = ["from __future__ import division"]

__canonical_imports__ = {"oldmodule.oldfunction": "newmodule.newfunction"}

tidy-imports --transform=oldmodule.oldfunction=newmodule.newfunction

Soapbox: avoid “star” imports

When programming in Python, a good software engineering practice is to avoid using from foopackage import * in production code.

This style is a maintenance nightmare:

• It becomes difficult to figure out where various symbols (functions/classes/etc) come from.

• It’s hard to tell what gets shadowed by what.

• When the package changes in trivial ways, your code will be affected. Consider the following example: Suppose foopackage.py contains import sys, and myprogram.py contains from foopackage import *; if some_condition: sys.exit(0). If foopackage.py changes so that import sys is removed, myprogram.py is now broken because it’s missing import sys.

To fix such code, you can run tidy-imports --replace-star-imports to automatically replace star imports with the specific needed imports.

By default pyflyby works out what foopackage exports by parsing its source rather than running it, so it cannot see an __all__ built at run time – as numpy does. For those, nothing is found and the star import is left alone. Pass --exec-star-imports to import the module and read its real __all__:

tidy-imports --replace-star-imports --exec-star-imports myprogram.py

--exec-star-imports is also useful on its own. A star import normally makes pyflyby give up on the rest of the file, since any undefined name might have come from the star. Knowing what the star supplies, it can fix the rest and leave the star in place:

$ cat myprogram.py
from numpy import *
print(mkstemp)
print(sin)

$ tidy-imports --exec-star-imports myprogram.py
[PYFLYBY] myprogram.py: added 'from tempfile import mkstemp'

sin is left alone because numpy exports it; mkstemp is imported from where it actually lives.

The flag is off by default because it always executes the module, in pyflyby’s own process. Note that the default is not a guarantee that nothing is executed: locating foo.bar’s source imports its parent package foo, and a module whose source can’t be read – an extension module, say – is imported outright to find out where it lives. --exec-star-imports therefore widens how much of your code runs, rather than being the only way it can run at all.

The same option exists for the auto-importer, where a star import otherwise makes it give up on the rest of the code being run. py does this by default – the code it is about to run would import the module anyway – and py --no-exec-star-imports turns it off.

In IPython it is off by default; turn it on when enabling the auto-importer, e.g. from ipython_config.py:

import pyflyby
pyflyby.enable_auto_importer(exec_star_imports=True)

Per-Project configuration of tidy-imports

You can configure Pyflyby on a per-repository basis by using the [tool.pyflyby] section of pyproject.toml files. Pyflyby will look in current working directory and all it’s parent until it find a pyproject.toml file from which it will load the defaults.

Most of the long command line flags default values can be configured in this section. Simply use the long form option name by replacing dashes - by underscore _. For long option that have the form --xxx and --no-xxx, you can assign a boolean to xxx. For example:

.. code:: toml

[tool.pyflyby] add_missing=true from_spaces=7 remove_unused=false tidy_local_imports=true

To exclude files from tidy-imports, add an exclusion pattern to tool.pyflyby.tidy-imports.exclude:

[tool.pyflyby.tidy-imports]
exclude = [
    "foo.py",
    "baz/*.py"
]

Exclusions are assumed to be relative to the project root if a pyproject.toml exists, unless an absolute path is specified. Consult the documentation for pathlib.Path.match for information about valid exclusion patterns.

Emacs support

• To get a M-x tidy-imports command in GNU Emacs, add to your ~/.emacs:

  (load "/<site-packages>/pyflyby/share/emacs/site-lisp/pyflyby.el")

• Pyflyby.el doesn’t yet work with XEmacs; patches welcome.

saveframe: A utility for debugging / reproducing an issue

PyFlyBy provides a utility named saveframe which can be used to save information for debugging / reproducing an issue.

Usage: If you have a piece of code or a script that is failing due an issue originating from upstream code, and you cannot share your private code as a reproducer, use this utility to save relevant information to a file. Share the generated file with the upstream team, enabling them to reproduce and diagnose the issue independently.

Information saved in the file: This utility captures and saves error stack frames to a file. It includes the values of local variables from each stack frame, as well as metadata about each frame and the exception raised by your code.

This utility comes with 2 interfaces:

1. A function: For interactive usages such as IPython, Jupyter Notebook, or a debugger (pdb/ipdb), use pyflyby.saveframe function. To know how to use this function, checkout it’s documentation:

In [1]: saveframe?

2. A script: For cli usages (like a failing script), use pyflyby/bin/saveframe script. To know how to use this script, checkout its documentation:

$ saveframe --help

Authorship

This plugin was contributed back to the community by the D. E. Shaw group.

Pyflyby is written by Karl Chen <quarl@8166.clguba.z.quarl.org>

We love contributions! Before you can contribute, please sign and submit this Contributor License Agreement (CLA). This CLA is in place to protect all users of this project.

License

Pyflyby is released under a very permissive license, the MIT/X11 license; see LICENSE.txt.

Release

1. Check version number in lib/python/pyflyby/_version.py, maybe increase it.

2. Commit and tag if necessary, and push tags/commits.

3. Optional: Set SOURCE_DATE_EPOCH for reproducible build:

   export SOURCE_DATE_EPOCH=$(git show -s --format=%ct HEAD)

4. Build the SDIST:

   python -m build --sdist

5. Optional Repack the Sdist to make sure the ZIP only contain SOURCE_DATE_EPOCH date using IPython tools:

   python ~/dev/ipython/tools/retar.py dist/pyflyby-1.7.8.tar.gz
   shasum -a 256 dist/*

6. Optional, redo 4 & 5 to verify checksum is unchanged.

7. Upload using twine:

   twine upload dist/*

8. Check/update https://github.com/conda-forge/pyflyby-feedstock for new pyflyby release on conda-forge