Skip to main content
This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!

Scatters tasks across processes.

Project Description

The buckshot library contains multiprocessing functions designed to help developers create distributed applications quickly.

Installation

The easiest way to install buckshot is via pip:

$ pip install buckshot

Usage

Currently, there are two ways of leveraging buckshot to distribute work across processes: the @distribute decorator and with distributed(...) context manager.

The following example shows how we can distibute the work of calculating many harmonic sum values across processes.

Serial Approach

import fractions

def harmonic_sum(x):
    F = fractions.Fraction
    return sum(F(1, d) for d in xrange(1, x + 1))

for value in range(1, 100):
    result = harmonic_sum(value)
    print result

Using @distribute

import fractions

from buckshot import distribute

@distribute(processes=4)
def harmonic_sum(x):
    F = fractions.Fraction
    return sum(F(1, d) for d in xrange(1, x + 1))

for result in harmonic_sum(range(1, 100)):  # Pass in values list.
    print result

Using with distributed(...)

import fractions

from buckshot import distributed

def harmonic_sum(x):
    F = fractions.Fraction
    return sum(F(1, d) for d in xrange(1, x + 1))

with distributed(harmonic_sum, processes=4) as distributed_harmonic_sum:
    for result in distributed_harmonic_sum(range(1, 100)):
        print result

All processes are destroyed when inputs are exhausted and/or the context is exited.

Known Issues

  • Function inputs must be picklable.
  • Function outputs must be picklable.
  • If a child process is killed externally, buckshot will block forever waiting for results.
  • This uses os.fork() under the hood, so there is a risk of rapidly exhausting memory.

LICENSE

Read the LICENSE file for details.

Release History

Release History

This version
History Node

0.0.10

History Node

0.0.9

History Node

0.0.8

History Node

0.0.7

History Node

0.0.6

History Node

0.0.5

History Node

0.0.4

History Node

0.0.3

History Node

0.0.2

History Node

0.0.1

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
buckshot-0.0.10.tar.gz (10.0 kB) Copy SHA256 Checksum SHA256 Source Sep 7, 2016

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting