pyfora 0.1a3

A library for parallel execution of Python code in the Ufora runtime

pyfora - Compiled, parallel python

pyfora is the client package for Ufora - a compiled, automatically parallel Python for data science and numerical computing.

Ufora achieves speed and scale by reasoning about your python code to compile it to machine code (so it’s fast) and find parallelism in it (so that it scales). The Ufora runtime is fully fault tolerant, and handles all the details of data management and task scheduling transparently to the user.

The Ufora runtime is invoked by enclosing code in a “ufora.remote” block. Code and objects are shipped to a Ufora cluster and executed in parallel across those machines. Results are then injected back into the host python environment either as native python objects, or as handles (in case the objects are very large). This allows you to pick the subset of your code that will benefit from running in Ufora - the remainder can run in your regular python environment.

For all of this to work properly, Ufora places one major restriction on the code that it runs: it must be “pure”, meaning that it cannot modify data structures or have side effects. This restriction allows the Ufora runtime to agressively reorder calculations, which is crucial for parallelism, and allows it to perform compile-time optimizations than would not be possible otherwise. For more on the subset of python that Ufora supports, see python restrictions.

Installation

The pyfora client is a pure python package and can be installed by running:

pip install pyfora

Getting Started with Ufora

The ufora backend is available as a docker image that can be run locally on your machine, or in a cluster of machines on a local network or in the cloud.

• Getting started with local Ufora

• Getting started with Ufora on AWS

• Running Ufora on a local cluster

Credits

Pyfora is developed and maintained by the Ufora team. Find us on Github.

• Distribute

Pyfora News

0.1a1

Release date: UNRELEASED

• Initial pre-release of pyfora!

• Includes support for core language features and builtin types.

• Some support for builtin functions like all, any, sum, etc.

• pyfora.aws module and pyfora_aws script help setup a Ufora cluster in EC2.