Skip to main content

Long task runner.

Project description

Long task runner with few nice features.

longtask is a simple framework to run long tasks (eg. data migration, data processing) in console environment. During process execution you can track changes on progress bar. If something goes wrong or you don’t have time now - you can stop task and rerun it later. longtask collects all your task stats and store it in JSON file.

How to create task?

If you want to create task you need to implement derived class from longtask.Task. You also need to override few things:

  • name - task name,

  • get_items() - this method should return you items collection,

  • process_item(item) - in this method you can process your item.

Simple example - resize many pictures (

import longtask
import glob
from PIL import Image

class ResizeTask(longtask.Task):
    name = 'resize'

    def get_items(self):
        return glob.glob('*.png')

    def process_item(self, item):
        original_image =
        resized_image = original_image.resize((100, 100))'output/' + item)

if __name__ == '__main__':

Now in directory with *.png images you can run:

$ python
[*] Starting task: resize [2013-03-03 18:36:29.953306]
[*] Processed: 100% (1000/1000) errors: 10% (100/1000) |###################| Elapsed Time: 1:00:00 ETA:  0:00:00
[*] Finished task: resize [2013-03-03 18:36:30.080421]
[*] Stats:
  - processed: 100% (1000/1000)
  - success: 90% (900/1000)
  - errors: 10% (100/1000)
    - 100 x IOError with traceback:

Traceback (most recent call last):
  File "/Users/konradhalas/dev/workspace/personal/longtask/longtask/", line 59, in run
  File "", line 13, in process_item
    original_image =
  File "/Users/konradhalas/dev/virtualenvs/longtask/lib/python2.7/site-packages/PIL/", line 1980, in open
    raise IOError("cannot identify image file")
IOError: cannot identify image file

In our example we resized 1000 files - 100 of them have corrupted data (IOError). This task took 1h.

You can always stop task with Ctrl+C and rerun it later with flag -c. If some errors occured durring execution you can rerun it with flag -e (it works only with flag -c).


Copyright [2013] [Konrad Hałas]

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

longtask-0.1.1.tar.gz (4.9 kB view hashes)

Uploaded source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page