This is a pre-production deployment of Warehouse, however changes made here WILL affect the production instance of PyPI.
Latest Version Dependencies status unknown Test status unknown Test coverage unknown
Project Description

## Introduction

The dones module can be used to mark whether a key is “done” and check whether a key has been marked “done”. Keys can also be unmarked, so that they are no longer “done”. Also, all keys can be unmarked by clearing the Dones. Keys are kept in their own namespace to avoid conflicts with other sets of other keys and to make it easy to implement clearing.

Why? I use dones to keep track of what I’ve already done. More specifically, I run large computations (a few million tasks) on a large cluster (a few thousand cores) with a slow filesystem (Isilon). When tasks inevitably fail, perhaps because the network storage goes offline, or a computer dies, or another user overwhelms the database with connections, I need to resubmit the tasks to the batch queuing system (LSF) of the cluster that are not already done.

The solution in this module fits my constraints. It handles the concurrent writes of a thousand jobs marking things done (not all at once). It is reasonably fast for reading and writing up to millions of jobs. This is important because or batch queue (LSF) only handles a few thousand jobs at a time gracefully. Finally, dones uses MySQL as a backend, which is important because I cannot run a key-value server like Redis on the cluster I use.

## Contribute

Feel free to make a pull request on github.

## Testing

Awkwardly, dones is configured with a MySQL database url from the environment, so to test it, you need to add a url. For example:

DONES_DB_URL=mysql://myuser:password@localhost/mydb nosetests

## Requirements

  • Probably Python 2.7 (since that is the only version it has been tested with.)
  • MySQL-python PyPI package.

## Installation

### Install from pypi.python.org

Download and install using pip:

pip install dones

### Install from github.com

Using github, one can clone and install a specific version of the package:

cd ~ git clone git@github.com:todddeluca/dones.git cd dones python setup.py install

Or use pip:

pip install git+git://github.com/todddeluca/dones.git#egg=dones

## Usage

import dones

if not dones.get(‘my_pipeline’).done(‘task1’):
dotask(‘task1’) dones.get(‘my_pipleline’).mark(‘task1’)
Release History

Release History

0.2.0

This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.1.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

TODO: Brief introduction on what you do with files - including link to relevant help section.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
dones-0.2.0.tar.gz (7.6 kB) Copy SHA256 Checksum SHA256 Source Mar 5, 2013

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS HPE HPE Development Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting