Easily speedup your Python code with Pythran
FluidPythran is still in a quite early stage. Remarks and suggestions are very welcome.
FluidPythran is a pure Python package (requiring Python >= 3.6 or Pypy3) to help to write Python code that can use Pythran if it is available.
Let’s recall that “Pythran is an ahead-of-time (AOT) compiler for a subset of the Python language, with a focus on scientific computing. It takes a Python module annotated with a few interface description and turns it into a native Python module with the same interface, but (hopefully) faster.”
Pythran is able to produce very efficient C++ code and binaries from high level Numpy code. If the algorithm is easier to express without loops, don’t write loops!
Pythran is not a hard dependency of FluidPythran: Python code using FluidPythran run fine without Pythran and without compilation (and of course without speedup)!
Python + Numpy + Pythran is a great combo to easily write highly efficient scientific programs and libraries.
To use Pythran, one needs to isolate the numerical kernels functions in modules that are compiled by Pythran. The C++ code produced by Pythran never uses the Python interpreter. It means that only a subset of what is doable in Python can be done in Pythran files. Some language features are not supported by Pythran (for example no classes) and most of the extension packages cannot be used in Pythran files (basically only Numpy and some Scipy functions).
Another cause of frustration for Python developers when using Pythran is related to manual writting of Pythran function signatures in comments, which can not be automated. Pythran uses C++ templates but Pythran users can not think with this concept. We would like to be able to express the templated nature of Pythran with modern Python syntax (in particular type annotations).
Finally, another limitation is that it is not possible to use Pythran for just-in-time (JIT) compilation so one needs to manually write all argument types.
With FluidPythran, we try to overcome these limitations. FluidPythran provides few supplementary Pythran commands and a small Python API to accelerate functions and methods with Pythran without writing the Pythran modules. The code of the numerical kernels can stay in the modules and in the classes where they were written. The Pythran files (i.e. the files compiled by Pythran), which are usually written by the user, are produced automatically by FluidPythran.
Bonus: There are FluidPythran syntaxes for both ahead-of-time and just-in-time compilations!
At run time, FluidPythran uses when possible the pythranized functions, but let’s stress again that codes using FluidPythran work fine without Pythran (of course without speedup)!
To summarize, a strategy to quickly develop a very efficient scientific application/library with Python could be:
- Use modern Python coding, standard Numpy/Scipy for the computations and all the cool libraries you want.
- Profile your applications on real cases, detect the bottlenecks and apply standard optimizations with Numpy.
- Add few lines of FluidPythran to compile the hot spots.
Implementation details: Under the hood, FluidPythran creates Pythran files (one per module for AOT compilation and one per function for JIT compilation) that can be compiled at build, import or run times depending of the cases. Note that the developers can still read the Pythran files if needed.
- The whole code can be gathered in one Python file.
- With the
@cachedjitdecorator, we don’t need to add the types and to launch compilation commands!
- Even without
@cachedjit(i.e. with AOT compilation), it is easy to trigger a mode in which FluidPythran automatically takes care of all compilation steps (see set_pythranize_at_import).
FluidPythran can be used in libraries and applications using MPI (as FluidSim).
pip install fluidpythran
The environment variable
FLUIDPYTHRAN_DIR can be set to control where
the cached files are saved.
A short tour of FluidPythran syntaxes
boost and command
# pythran def
import h5py import mpi4py from fluidpythran import boost # pythran def myfunc(int, float) @boost def myfunc(a, b): return a * b ...
Most of this code looks familiar to Pythran users. The differences:
- One can use (for example) h5py and mpi4py (of course not in the Pythran functions).
# pythran definstead of
# pythran export(to stress that it is not the same command).
- A tiny bit of Python… The decorator
@boostreplaces the Python function by the pythranized function if FluidPythran has been used to produced the associated Pythran file.
Pythran using type annotations
The previous example can be rewritten without Pythran commands:
import h5py import mpi4py from fluidpythran import boost @boost def myfunc(a: int, b: float): return a * b ...
Nice (shorter and clearer than with the Pythran command) but very limited… So
one can also elegantly define many Pythran signatures using in the annotations
type variables and Pythran types in strings (see these examples).
Moreover, it is possible to mix type hints and
# pythran def commands.
Cached Just-In-Time compilation
With FluidPythran, one can use the Ahead-Of-Time compiler Pythran in a Just-In-Time mode. It is really the easiest way to speedup a function with Pythran, just by adding a decorator! And it also works in notebooks!
It is a “work in progress” so (i) it could be buggy and (ii) the API is not great, but it is a good start!
import numpy as np # pythran import numpy as numpy from fluidpythran import cachedjit, used_by_cachedjit @used_by_cachedjit("func1") def func0(a, b): return a + b @cachedjit def func1(a, b): return np.exp(a) * b * func0(a, b)
Note that the
@cachedjit decorator takes into account type hints (see
the example in the documentation).
Implementation details for just-in-time compilation: A Pythran file is
produced for each “cachedjited” function (function decorated with
@cachedjit). The file is compiled at the first call of the function and
the compiled version is used as soon as it is ready. The warmup can be quite
long but the compiled version is saved and can be reused (without warmup!) by
# pythran block
FluidPythran blocks can be used with classes and more generally in functions with lines that cannot be compiled by Pythran.
from fluidpythran import FluidPythran fp = FluidPythran() class MyClass: ... def func(self, n): a, b = self.something_that_cannot_be_pythranized() if fp.is_transpiled: result = fp.use_pythranized_block("name_block") else: # pythran block ( # float a, b; # int n # ) -> result # pythran block ( # complex a, b; # int n # ) -> result result = a**n + b**n return self.another_func_that_cannot_be_pythranized(result)
For blocks, we need a little bit more of Python.
- At import time, we have
fp = FluidPythran(), which detects which Pythran module should be used and imports it. This is done at import time since we want to be very fast at run time.
- In the function, we define a block with three lines of Python and special
Pythran annotations (
# pythran block). The 3 lines of Python are used (i) at run time to choose between the two branches (
is_transpiledor not) and (ii) at compile time to detect the blocks.
Note that the annotations in the command
# pythran block are different
(and somehow easier to write) than in the standard command
I’m not satisfied by the syntax for Pythran blocks so I (PA) proposed an alternative syntax in issue #29.
@cachedjit for methods
For simple methods only using attributes, we can write:
import numpy as np from fluidpythran import boost A = "float[:]" @boost class MyClass: arr0: A arr1: A def __init__(self, n): self.arr0 = np.zeros(n) self.arr1 = np.zeros(n) @boost def compute(self, alpha: float): return (self.arr0 + self.arr1).mean() ** alpha
Calling another method in a Pythranized method is not yet supported!
More examples of how to use FluidPythran for Object Oriented Programing are given here.
Make the Pythran files
There is a command-line tool
fluidpythran which makes the associated
Pythran files from Python files with annotations and fluidpythran code. By
default and if Pythran is available, the Pythran files are compiled.
There is also a function
make_pythran_files that can be used in a
setup.py like this:
from pathlib import Path from fluidpythran.dist import make_pythran_files here = Path(__file__).parent.absolute() paths = ["fluidsim/base/time_stepping/pseudo_spect.py"] make_pythran_files([here / path for path in paths], mocked_modules=["h5py"])
Note that the function
make_pythran_files does not use Pythran.
Compiling the associated Pythran file can be done if wanted (see for example
how it is done in the example package example_package_fluidpythran or in
If the environment variable
PYTHRANIZE_AT_IMPORT is set, FluidPythran
compiles at import time (i.e. only when needed) the Pythran file associated
with the imported module. This behavior can also be triggered programmatically
by using the function
FluidDyn is distributed under the CeCILL-B License, a BSD compatible french license.
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size fluidpythran-0.1.7.tar.gz (37.7 kB)||File type Source||Python version None||Upload date||Hashes View|
|Filename, size fluidpythran-0.1.7-py3-none-any.whl (49.6 kB)||File type Wheel||Python version py3||Upload date||Hashes View|
Hashes for fluidpythran-0.1.7-py3-none-any.whl