Skip to main content

ArchiveTeam seesaw kit

Project description

Seesaw toolkit

An asynchronous toolkit for distributed web processing. Written in Python and named after its behavior, it supports concurrent downloads, uploads, etc.

This toolkit is well-known for Archive Team projects. It also powers the Archive Team warrior.

Build Status Coverage Status


Requires Python 2 or 3.

Needs the Tornado library for event-driven I/O. The complete list of Python modules needed are listed in requirements.txt.

How to try it out

To run the example pipeline:

sudo pip install -r requirements.txt
./run-pipeline --help
./run-pipeline examples/ someone

Point your browser to

You can also use run-pipeline2 or run-pipeline3 to be explicit for the Python version.


General idea: a set of Tasks that can be combined into a Pipeline that processes Items:

  • An Item is a thing that needs to be downloaded (a user, for example). It has properties that are filled by the Tasks.
  • A Task is a step in the download process: it takes an item, does something with it and passes it on. Example Tasks: getting an item name from the tracker, running a download script, rsyncing the result, notifying the tracker that it's done.
  • A Pipeline represents a sequence of Tasks. To make a seesaw script for a new project you'd specify a new Pipeline.

A Task can work on multiple Items at a time (e.g., multiple Wget downloads). The concurrency can be limited by wrapping the task in a LimitConcurrency Task: this will queue the items and run them one-by-one (e.g., a single Rsync upload).

The Pipeline needs to be fed empty Item objects; by controlling the number of active Items you can limit the number of items. (For example, add a new item each time an item leaves the pipeline.)

With the ItemValue, ItemInterpolation and ConfigValue classes it is possible to pass item-specific arguments to the Task objects. The value of these objects will be re-evaluated for each item. Examples: a path name that depends on the item name, a configurable bandwidth limit, the number of concurrent downloads.

Consult the wiki for more information.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for seesaw, version 0.10.0
Filename, size File type Python version Upload date Hashes
Filename, size seesaw-0.10.0.tar.gz (149.4 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page