Skip to main content
pyfora - Compiled, parallel python
==================================

pyfora is the client package for Ufora_ - a compiled, automatically parallel Python for data science
and numerical computing.

Ufora achieves speed and scale by reasoning about your python code to compile
it to machine code (so it's fast) and find parallelism in it (so that it scales). The Ufora
runtime is fully fault tolerant, and handles all the details of data
management and task scheduling transparently to the user.

The Ufora runtime is invoked by enclosing code in a "ufora.remote" block. Code
and objects are shipped to a Ufora cluster and executed in parallel across
those machines. Results are then injected back into the host python
environment either as native python objects, or as handles (in case the
objects are very large). This allows you to pick the subset of your code that
will benefit from running in Ufora - the remainder can run in your regular
python environment.

For all of this to work properly, Ufora places one major restriction on
the code that it runs: it must be "pure", meaning that it cannot modify data
structures or have side effects. This restriction allows the Ufora runtime to
agressively reorder calculations, which is crucial for
parallelism, and allows it to perform compile-time
optimizations than would not be possible otherwise. For more on the subset of python
that Ufora supports, see `python restrictions`_.

.. _python restrictions: https://ufora.github.io/ufora/documentation/python-restrictions.html


Installation
============

The pyfora client is a pure python package and can be installed by running:

.. code::

pip install pyfora


Getting Started with Ufora
==========================

The ufora backend is available as a docker image that can be run locally on your machine, or in a
cluster of machines on a local network or in the cloud.

- `Getting started with local Ufora`_
- `Getting started with Ufora on AWS`_
- `Running Ufora on a local cluster`_


.. _Getting started with local Ufora: https://ufora.github.io/ufora/tutorials/getting-started-local.html
.. _Getting started with Ufora on AWS: https://ufora.github.io/ufora/tutorials/getting-started-aws.html
.. _Running Ufora on a local cluster: https://ufora.github.io/ufora/tutorials/getting-started-cluster.html


Credits
-------

Pyfora is developed and maintained by the Ufora_ team. Find us on Github_.


- `Distribute`_

.. _Distribute: http://pypi.python.org/pypi/distribute

.. _Ufora: https://ufora.github.io/ufora
.. _Github: https://github.com/ufora/ufora


Pyfora News
===========

0.4.2
-----

* Release date Mar-4-2016

* [bug #203]: Avoid socketIO exception we were hitting
* [bug #205]: Ensure that we can pass Futures into with and submit blocks naturally
* [bug #206]: Ensure that we propagate S3 errors correctly in pyfora
* [bug #234]: Ensure we visit paged vectors in correct order
* [bug #221]: Class instances have consistent ordering for their members
* [bug #228]: Ensure mutually recursive objects have stable definitions
* [bug]: Fix some bugs moving large lists from server to client
* [feature]: Implement trust region congugate gradient solver for logistic regression
* [feature]: Bring back if(`split) model for dynamic parallelism
* [feature]: Starting a compiler cache
* [feature]: Preliminary features for GPU computing
* [feature]: Adding 'pyfora_aws deploy' command
* [feature]: Adding vpc, subnet, and security-group args for all pyfora_aws commands
* [enhancement]: Improved compiler performance
* [enhancement]: Improve error messages for accessing nonexistent S3 buckets
* [enhancement]: Improve withBlockExecutor behavior when passed futures containing exceptions
* [enhancement]: Raise the right kind of exception when we try to convert a "with" block
* [enhancement]: Ensure that hashes of Pyfora list objects are stable.

0.4.1
-----

* Release date: Feb-24-2016

* [feature]: Supporting member initialization in base-class __init__ functions
* [feature]: Adding support for numpy.linalg.svd
* [bug #208]: Can't convert bound instance methods from base classes


0.4
---

*Release date: Feb-17-2016

* [feature #78]: Improved error reporting for untranslatable code
* [feature #133]: Initial support for object inheritance
* [enhancement]: New compiler implementation produces much more efficient code
* [enhancement]: Implementation of beta function better matches scipy


0.3.4
-----

* speed up fora compiler
* speed up pyfora data upload time
* fix bug in hyp2f1
* hook up many more scipy/numpy special (math) functions

0.3.3
-----

*Release date: Jan-27-2016

* Make scipy optional


0.3.1
-----

*Release date: Jan-26-2016

* Add support for scipy.special.gamma and scipy.special.hyp2f1


0.3
---

*Release date: Jan-22-2016

* [bug #17]: Can’t call static methods on instances in fora, can in python
* [bug #83]: Possibly Uninitialized Variable Analysis cannot deal with complex data-flow
* [bug #107]: Bad error message when non-bound function gets too many call args
* [feature #124]: Implement `assert`
* [bug #134]: PyInt.fora doesn't have an implementation of __mod__
* [bug #138]: Dictionary comprehensions don't work
* [feature #153]: Read files from local file-system
* [feature #154]: Logistic regression in pyfora
* [feature #155]: Gradient-boosted trees in pyfora
* [feature #159]: Add 'add worker' command to pyfora_aws
* [bug #163]: pyfora_aws has problems if "ufora" security group is already created
* [feature #168]: No feedback in pyfora_aws when things go wrong on an instance
* [bug #170]: Confusing error message when client and server versions don't match
* [feature #172]: Operator Coalescing
* [bug #176]: `isinstance` bug
* [feature #179]: Inline fora in pyfora


0.2.1
-----

*Release date: Dec-10-2015

* [feature] provide pyfora wrapper for scipy.special.beta
* [feature] provide pyfora wrapper for math.log
* [feature] perf improvements for mixin binding calculations.

0.2
----

*Release date: Dec-08-2015

* [bug #165]: Set good default value for EXTERNAL_DATASET_LOADER_SERVICE_THREADS.
* [bug #162]: pyfora_aws docs indicate that ec2 region is optional, but parameter is in fact required.
* [feature]: pyfora_aws should propagate AWS credentials.
* [bug #145]: Cannot access data in S3.
* [bug #144]: pyfora_aws raises exception when --num-instances is 1.
* [bug #140]: ufora-worker launched with pyfora_aws only uses 8GB of memory.
* [bug #136]: Collisions with pandas and numpy on case-insensitive file-systems.
* [bug #127]: Correctly propegating communication errors up to Executor.
* [feature]: Support @property decorator.
* [feature]: Improved download performance of large lists of small objects.
* [bug #122]: Wrong exception type from `list + non_list`.
* [bug #120]: Failure when trying to convert a list of mapped functions.
* [bug #119]: Can't convert bound instance methods.
* [bug #116]: Builtin "reduce" function is not parallelizable when applied over lists, xrange, etc.
* [bug #115]: Fixing __getitem__ for strings and tuples
* [bug #111]: Wrong exception when accessing unbound variables.
* [bug #110]: Incorrect conversion of class functions in user-defined classes.
* [bug #109]: list __getitem__ doesn't throw with step 0
* [feature]: Implement `map` builtin
* [feature]: Support `isinstance` on user-defined classes.
* [feature]: Add versioning scheme to socket.io protocol.
* [feature]: Add support for the python REPL.
* [bug #90]: Improved error message for unbound free variables.
* [bug #89]: Ctrl+C doesn't break out of `with` block.
* [bug #68]: Disallow `return` statements in pyfora `with` blocks.
* [bug #67]: tuple unpacking doesn't work
* [feature]: basic linear regression on data-frames
* [feature]: basic CSV parsing
* [feature]: basic data-frames
* [bug #59]: `sequence(0)` not iterable
* [bug #47]: int/float mismatch in `**` operator
* [bug #21]: certain python variables "survive" longer than fora values


*Known Issues:

* `def` order is important in non-module function definition (closures). If functions
`g()` and `h()` are defined inside of function `f` and `g()` calls `h()`, then `def h():` must
appear BEFORE `def g():`.
This also implies that mutually-recursive functions are only possible at module or class level.

* Class static methods cannot be used as values. They can be invoked, but it's not possible
to pass a class static method as an argument to another function.

* Named argument calls are not supported. If you have a function `def f(x):...` you can call it as
`f(42)` but you can't use `f(x=42)`.

* Keyword arguments are not supported.

* Class members can only be initialized inside of `__init__`. If `__init__` calls another function
that initializes members, those members will not be seen by pyfora.

* `return` statements not allowed in `__init__()`

* @classmethod decorator is not supported.

* No support for `*args`.

* `assert` is not implemented.

* Bad error message when using `self` inside of `__init__` for things other than setting or getting
members. For example, calling `str(self)` inside of `__init__` results in
"PythonToForaConversionError: An internal error occurred: we didn't provide a definition for the following variables: ['self'].
Most likely, there is a mismatch between our analysis of the python code and the generated FORA code underneath. Please file a bug report."

* No support for object inheritance.


0.1
-----

*Release date: Nov-06-2015

* Initial release of pyfora!
* Includes support for core language features and builtin types.
* Some support for builtin functions like all, any, sum, etc.
* pyfora.aws module and pyfora_aws script help setup a Ufora cluster in EC2.

Metadata

Release files for pyfora 0.4.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pyfora 0.4.2
File Size Uploaded Provenance Actions
pyfora-0.4.2.tar.gz 109.0 kB Without provenance Details for pyfora-0.4.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pyfora 0.4.2
File Interpreter ABI Platform Actions
pyfora-0.4.2-py2-none-any.whl Python 2 none any Details for pyfora-0.4.2-py2-none-any.whl

Total release size: 319.4 kB

Release files / pyfora-0.4.2.tar.gz

Download URL pyfora-0.4.2.tar.gz
Size 109.0 kB
Tags Source
SHA-256 checksum
How to use checksums
177048e819b9d839e2ee0da67c3d4f8bd90f3b394c568f93b9f3c57b2df9e8ac
BLAKE2b-256 checksum
How to use checksums
ecbc96e2a91cd557024be8124c9da8ce7d57dc5f92560ae394f22e215d3b4937
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Release files / pyfora-0.4.2-py2-none-any.whl

Download URL pyfora-0.4.2-py2-none-any.whl
Size 210.4 kB
Tags Python 2
SHA-256 checksum
How to use checksums
ec12e13f62c454a3ee9f7e299b1b35cb04ae0652ebba4455ba200238959542fe
BLAKE2b-256 checksum
How to use checksums
2252933ab92e09ddef8b665bf396b52898845f2ce7ebf3e1625cdcce233d5691
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Security for pyfora 0.4.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.4.2 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page