A library for parallel execution of Python code in the Ufora runtime
Project description
pyfora - Compiled, parallel python
pyfora is the client package for Ufora - a compiled, automatically parallel Python for data science and numerical computing.
Ufora achieves speed and scale by reasoning about your python code to compile it to machine code (so it’s fast) and find parallelism in it (so that it scales). The Ufora runtime is fully fault tolerant, and handles all the details of data management and task scheduling transparently to the user.
The Ufora runtime is invoked by enclosing code in a “ufora.remote” block. Code and objects are shipped to a Ufora cluster and executed in parallel across those machines. Results are then injected back into the host python environment either as native python objects, or as handles (in case the objects are very large). This allows you to pick the subset of your code that will benefit from running in Ufora - the remainder can run in your regular python environment.
For all of this to work properly, Ufora places one major restriction on the code that it runs: it must be “pure”, meaning that it cannot modify data structures or have side effects. This restriction allows the Ufora runtime to agressively reorder calculations, which is crucial for parallelism, and allows it to perform compile-time optimizations than would not be possible otherwise. For more on the subset of python that Ufora supports, see python restrictions.
Installation
The pyfora client is a pure python package and can be installed by running:
pip install pyfora
Getting Started with Ufora
The ufora backend is available as a docker image that can be run locally on your machine, or in a cluster of machines on a local network or in the cloud.
Credits
Pyfora is developed and maintained by the Ufora team. Find us on Github.
Pyfora News
0.1a1
Release date: UNRELEASED
Initial pre-release of pyfora!
Includes support for core language features and builtin types.
Some support for builtin functions like all, any, sum, etc.
pyfora.aws module and pyfora_aws script help setup a Ufora cluster in EC2.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.