Import Jupyter (ne IPython) notebooks into tests and scripts.
Project description
importnb imports notebooks as modules. Notebooks are reusable as tests, source code, importable modules, and command line utilities.
Installation
pip install importnb
conda install -c conda-forge importnb
importnb
for testing
After importnb
is installed, pytest will discover and import notebooks as tests.
pytest index.ipynb
importnb
imports notebooks as python modules, it does not compare outputs like nbval
. These two pytest
extensions complement each other well.
Notebooks are often used as informal tests, now they can be formally tested with pytest plugins
importnb
for the commmand line
importnb
can run notebooks as command line scripts. Any literal variable in the notebook, may be applied as a parameter from the command line.
ipython -m importnb -- index.ipynb --foo "A new value"
importnb
for Python and IPython
It is suggested to execute importnb-install
to make sure that notebooks for each IPython session.
Restart and run all or it didn't happen.
importnb
excels in an interactive environment and if a notebook will Restart and Run All then it may reused as python code. The Notebook
context manager will allow notebooks with valid names to import with Python.
>>> from importnb import Notebook
For brevity
with __import__('importnb').Notebook():
import readme
importnb.loader
will find notebooks available anywhere along thesys.path
.
or explicity
from importnb import Notebook
with Notebook():
import readme
foo = 42
import readme
assert readme.foo is 42
assert readme.__file__.endswith('.ipynb')
Modules may be reloaded
The context manager is required to reload
a module.
from importlib import reload
with Notebook(): __name__ == '__main__' and reload(readme)
Lazy imports
The lazy
option will delay the evaluation of a module until one of its attributes are accessed the first time.
with Notebook(lazy=True):
import readme
Traceback (most recent call last):
File "C:\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3265, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-6-a80fb0c23008>", line 2, in <module>
import readme
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "c:\users\deathbeds\pidgin\pidgin\loader.ipynb", line 71, in exec_module
super().exec_module(module)
File "<frozen importlib._bootstrap_external>", line 674, in exec_module
File "<frozen importlib._bootstrap_external>", line 781, in get_code
File "c:\users\deathbeds\importnb\src\importnb\loader.py", line 290, in source_to_code
nodes = ast.parse(nodes)
File "C:\Anaconda3\lib\ast.py", line 35, in parse
return compile(source, filename, mode, PyCF_ONLY_AST)
File "<unknown>", line 2
"""__importnb__ imports notebooks as modules. Notebooks are reusable as tests, source code, importable modules, and command line utilities.
^
IndentationError: unexpected indent
Fuzzy File Names
if __name__ == '__main__':
with Notebook():
import __a_me
assert __a_me.__file__ == readme.__file__
Traceback (most recent call last):
File "C:\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3265, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-7-15cace34bba8>", line 3, in <module>
import __a_me
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "c:\users\deathbeds\pidgin\pidgin\loader.ipynb", line 71, in exec_module
super().exec_module(module)
File "<frozen importlib._bootstrap_external>", line 674, in exec_module
File "<frozen importlib._bootstrap_external>", line 781, in get_code
File "c:\users\deathbeds\importnb\src\importnb\loader.py", line 290, in source_to_code
nodes = ast.parse(nodes)
File "C:\Anaconda3\lib\ast.py", line 35, in parse
return compile(source, filename, mode, PyCF_ONLY_AST)
File "<unknown>", line 2
"""__importnb__ imports notebooks as modules. Notebooks are reusable as tests, source code, importable modules, and command line utilities.
^
IndentationError: unexpected indent
Python does not provide a way to import file names starting with numbers of contains special characters. importnb
installs a fuzzy import logic to import files containing these edge cases.
import __2018__6_01_A_Blog_Post
will find the first file matching *2018*6?01?A?Blog?Post
. Importing Untitled314519.ipynb
could be supported with the query below.
import __314519
Docstring
The first markdown cell will become the module docstring.
if __name__ == '__main__':
print(readme.__doc__.splitlines()[0])
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-8-1bd926f9e7a3> in <module>
1 if __name__ == '__main__':
----> 2 print(readme.__doc__.splitlines()[0])
NameError: name 'readme' is not defined
Meaning non-code blocks can be executeb by doctest.
if __name__ == '__main__':
__import__('doctest').testmod(readme)
Import notebooks from files
Notebook names may not be valid Python paths. In this case, use Notebook.from_filename
.
Notebook.from_filename('index.ipynb')
Import under the __main__
context.
Notebook('__main__').from_filename('index.ipynb')
Parameterize Notebooks
Literal ast statements are converted to notebooks parameters.
In readme
, foo
is a parameter because it may be evaluated with ast.literal_val
if __name__ == '__main__':
from importnb import Parameterize
f = Parameterize.load(readme.__file__)
The parameterized module is a callable that evaluates with different literal statements.
if __name__ == '__main__':
assert callable(f)
f.__signature__
assert f().foo == 42
assert f(foo='importnb').foo == 'importnb'
Run Notebooks from the command line
Run any notebook from the command line with importnb. Any parameterized expressions are available as parameters on the command line.
!ipython -m importnb -- index.ipynb --foo "The new value"
Integrations
IPython
IPython Extension
Avoid the use of the context manager using loading importnb as IPython extension.
%load_ext importnb
%unload_ext importnb
will unload the extension.
Default Extension
importnb
may allow notebooks to import by default with
!importnb-install
If you'd like to play with source code on binder then you must execute the command above. Toggle the markdown cell to a code cell and run it.
This extension will install a script into the default IPython profile startup that is called each time an IPython session is created.
Uninstall the extension with importnb-install
.
Run a notebook as a module
When the default extension is loaded any notebook can be run from the command line. After the importnb
extension is created notebooks can be execute from the command line.
ipython -m readme
In the command line context, __file__ == sys.arv[0] and __name__ == '__main__'
.
See the deploy step in the travis build.
Parameterizable IPython commands
Installing the IPython extension allows notebooks to be computed from the command. The notebooks are parameterizable from the command line.
ipython -m readme -- --help
py.test
importnb
installs a pytest plugin when it is setup. Any notebook obeying the py.test discovery conventions can be used in to pytest. This is great because notebooks are generally your first test.
!ipython -m pytest -- src
Will find all the test notebooks and configurations as pytest would any Python file.
Setup
To package notebooks add recursive-include package_name *.ipynb
Developer
Format and test the Source Code
if __name__ == '__main__':
if globals().get('__file__', None) == __import__('sys').argv[0]:
print(foo, __import__('sys').argv)
else:
from subprocess import call
!ipython -m pytest
"""Formatting"""
from pathlib import Path
from importnb.utils.export import export
root = 'src/importnb/notebooks/'
for path in Path(root).rglob("""*.ipynb"""):
if 'checkpoint' not in str(path):
export(path, Path('src/importnb') / path.with_suffix('.py').relative_to(root))
!jupyter nbconvert --to markdown --stdout index.ipynb > readme.md
if __name__ == '__main__':
try:
from IPython.display import display, Image
from IPython.utils.capture import capture_output
from IPython import get_ipython
with capture_output():
get_ipython().system("cd docs && pyreverse importnb -opng -pimportnb")
display(Image(url='docs/classes_importnb.png', ))
except: ...
CHANGELOG
0.4.0
- Fuzzy name completion.
- A configurable extension system for magics.
Interactive(shell=False)
is the default loader.
0.3.2
- Add
remote
loader. Load notebooks from remote urls. - Support a fuzzy name import system. Files with special characters and numbers are importable.
- An IPython magic to allow relative imports during interactive computing.
0.3.1
- In loaders
Notebook
,Interactive
,Execute
, andParameterize
- Remove
Partial
,Lazy
, andNotebookTest
loaders. - The first Markdown cell imports as a docstrings, permitting doctests on markdown cells.
Notebook(globals={})
passes global values to the moduleNotebook(dir="..")
will change the working directory and path.- The code is pure python and uses IPython when possible.
ipython -m importnb nodebook.ipynb
runs a notebook.
0.2.9
- Include
Partial
,Lazy
, andNotebookTest
loaders. - Transform markdown cells to literate block strings so they are included in the ast.
__doc__
's are extracted from the first markdown cell or normal source code from a code cell.
- Export the python source code with
black
. Notebook.from_filename
is a loader for paths and strings.- Add
importnb.nbtest
for notebook testing tools.. - Benchmark
importnb
against existing notebooks. - Include a
watchdog
trick to watch tests.. - Extend the project to >= 3.4
- Use nbviewer/github hierachy for the docs.
0.2.4
- Use
tox
for testing - Use a source directory folder structure for pytest and tox testing.
- Create a pytest plugin that discovers notebooks as tests. With
importnb
notebooks can be used as fixtures in pytest. - Install
importnb
as an IPython extension. - Support running notebooks as modules from the
ipython
command line - Create a
setuptools
command to allow notebooks as packages.
0.2.1
-
importnb
supports notebook inputs from pure python environments. Two compatible compiler were created from IPython and Python -
importnb.Partial
works appropriately by improving exceptions. -
All of the IPython magic syntaxes were removed to support Pure Python.
-
The generated Python files are formatted with black.
-
Tests were added to:
- Validate the line number in tracebacks
- Test someone elses notebooks
0.1.4
- Pypi supports markdown long_description with the proper mimetype in long_description_content_type.
0.1.3
- Include the RST files in the
MANIFEST.in
.
0.1.2 (Unreleased)
- Use RST files to improve the literacy of the pypi description.
0.1.1
- Released on PyPi
0.0.2
- Initial Testing Release
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.