Easily create compiled namespace Numba extensions.
Project description
Numba Namespace Extension
The idea of this package is to facilitate the creation of compiled numba extensions without modifying the original code implementation. The recommended approach is to use python namespaces so that we can have the compiled version of the main package with the same access pattern. This looks like the following:
- Have the original package in a namespace module called
my_package
:
$ pip install -e my_package
from my_package.utils import my_function
Simple main package configuration using `my_package` namespace.
├── setup.py
├── README.md
└── src
└──my_package
└── utils.py
- Create a compiled version of the functions of the original package, in an independent python package using the same
my_pacakge
namespace.
$ pip install -e my_package_compiled
from my_package.compiled.utils import my_function
Simple compiled package configuration using `my_package` namespace.
├── setup.py
├── README.md
├── registry.json
└── src
└──my_package
└── compiled
└── __init__.py
By following the namespace suggestion, you can keep your original code in the my_package
library and have an extra
independent python package with the compiled version but sharing the same access point (namespace).
Shared configuration between my_package
and my_package_compiled
:
- They both use a
my_package
namespace. In practice, this means that there's no__init__.py
on under themy_package
directory. - They are probably using the
setuptools.find_namespace_packages
function in thesetup.py
file.
Differences between the my_package
and my_package_compiled
implementations:
- The
my_package_compiled
library depends on already having themy_pacakge
library installed. - The
registry.json
file contains the information needed for the numba compiler (for each function). - The
my_package_compiled
does not has custom-made code; only the compiled version of the functions defined in theregistry.json
file.
What are python namespace packages? In short, namespace modules are like regular python modules but
without the __init__.py
file. This allows you to share the same "module name" among different packages.
Consider the following example:
- Shared namespace:
my_package
- Package 1 (
package-1
): Implement hello world in english - Package 2 (
package-2
): Implement hello world in spanish
Package 1: contains a hello_world.py
python submodule within the my_package
namespace.
README.md
setup.py
src/
my_package/
hello_world.py
Package 2: contains a hola_mundo.py
python submodule within the my_package
namespace.
README.md
setup.py
src/
my_package/
hola_mundo.py
Expected behavior: you can install each package independently. Nonetheless, they will share a common namespace. Take a look at a complete working implementation here.
Learn more about namespace packages:
- In this twitter thread.
- From the documentation.
Installation
You can install this package via pip
by running:
$ pip install numba_namespace_extension
Alternatively, for development installation, clone this repo and run:
$ pip install -e .
Usage
The suggested usage is to create a python package with a matching namespace to the library with the original code. You can do this by manually creating the new package or leveraging our cookiecutter template with a minimal python namespace package setup.
Install cookiecutter (i.e., pip install cookiecutter
) and execute the template:
(Option 1) Referencing Github:
cookiecutter https://github.com/RHDZMOTA/numba-namespace-extension.git
(Option 2) Or locally if you have this repo already cloned:
cookiecutter path/to/repo
You can now start registering functions in the registry.json
file. For each function, provide the following:
module (str)
: the module name where the original function is located.function_name (str)
: the original function name.signature
: the numba type signature (read more this here).
Compilation is done when installing your package. You can install it locally by running:
- Replace
path/to/my_package
with the full or relative path to your package (i.e., directory where thesetup.py
is located).
pip install -e path/to/my_package
Key components:
- The
Registry
class is basically a wrapper over the Numba AOT implementation that allows us to define source functions on the configuration file (i.e.,registry.json
) - Use the
from_json
static method to create aRegistry
instance referencing the json config file. - Once you have an instance, run the
ext_modules
and pass the result to theext_modules
argument from thesetup
function. - Consider that all the modules referenced by the
registry.json
file MUST be available in the installation runtime.
Example of minimal setup.py
file:
from setuptools import setup, find_namespace_packages
from numba_namespace_extension.registry import Registry
setup(
name="<package-name>",
version="0.1.0",
packages=find_namespace_packages(where="src"),
package_dir={
"": "src"
},
ext_modules=Registry.from_json("registry.json").ext_modules()
)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Hashes for numba_namespace_extension-0.1.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | e01b22f0b292dadfc00d758224f86e6819c1bbac1876dbb5f9c8c828d7c09e2a |
|
MD5 | 25da2e0f0d38ff26f0c6bc8afc07ad4b |
|
BLAKE2b-256 | 17f3d0502bde2de8773fc117910813b0fe2651bd4954f0bbf53e9bfd4c99bf84 |