Skip to main content

A library for parallel execution of Python code in the Ufora runtime

Project description

pyfora - Compiled, parallel python

pyfora is the client package for Ufora - a compiled, automatically parallel Python for data science and numerical computing.

Ufora achieves speed and scale by reasoning about your python code to compile it to machine code (so it’s fast) and find parallelism in it (so that it scales). The Ufora runtime is fully fault tolerant, and handles all the details of data management and task scheduling transparently to the user.

The Ufora runtime is invoked by enclosing code in a “ufora.remote” block. Code and objects are shipped to a Ufora cluster and executed in parallel across those machines. Results are then injected back into the host python environment either as native python objects, or as handles (in case the objects are very large). This allows you to pick the subset of your code that will benefit from running in Ufora - the remainder can run in your regular python environment.

For all of this to work properly, Ufora places one major restriction on the code that it runs: it must be “pure”, meaning that it cannot modify data structures or have side effects. This restriction allows the Ufora runtime to agressively reorder calculations, which is crucial for parallelism, and allows it to perform compile-time optimizations than would not be possible otherwise. For more on the subset of python that Ufora supports, see python restrictions.

Installation

The pyfora client is a pure python package and can be installed by running:

pip install pyfora

Getting Started with Ufora

The ufora backend is available as a docker image that can be run locally on your machine, or in a cluster of machines on a local network or in the cloud.

Credits

Pyfora is developed and maintained by the Ufora team. Find us on Github.

Pyfora News

0.1a1

Release date: UNRELEASED

  • Initial pre-release of pyfora!
  • Includes support for core language features and builtin types.
  • Some support for builtin functions like all, any, sum, etc.
  • pyfora.aws module and pyfora_aws script help setup a Ufora cluster in EC2.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for pyfora, version 0.1a2
Filename, size File type Python version Upload date Hashes
Filename, size pyfora-0.1a2-py2-none-any.whl (84.6 kB) File type Wheel Python version 2.7 Upload date Hashes View hashes
Filename, size pyfora-0.1a2.tar.gz (64.1 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page