Seesaw toolkit
==============
An asynchronous toolkit for distributed web processing. Written in Python and named after its behavior, it supports concurrent downloads, uploads, etc.
This toolkit is well-known for [Archive Team projects](http://archiveteam.org). It also powers the [Archive Team warrior](http://archiveteam.org/index.php?title=Warrior).
[](http://travis-ci.org/ArchiveTeam/seesaw-kit)
Installation
------------
Requires Python 2 or 3.
Needs the Tornado library for event-driven I/O. The complete list of Python modules needed are listed in requirements.txt.
How to try it out
-----------------
To run the example pipeline:
sudo pip install -r requirements.txt
./run-pipeline --help
./run-pipeline examples/example-pipeline.py someone
Point your browser to `http://127.0.0.1:8001/`.
You can also use `run-pipeline2` or `run-pipeline3` to be explicit for the Python version.
Overview
--------
General idea: a set of `Task`s that can be combined into a `Pipeline` that processes `Item`s:
* An `Item` is a thing that needs to be downloaded (a user, for example). It has properties that are filled by the `Task`s.
* A `Task` is a step in the download process: it takes an item, does something with it and passes it on. Example Tasks: getting an item name from the tracker, running a download script, rsyncing the result, notifying the tracker that it's done.
* A `Pipeline` represents a sequence of `Task`s. To make a seesaw script for a new project you'd specify a new `Pipeline`.
A `Task` can work on multiple `Item`s at a time (e.g., multiple Wget downloads). The concurrency can be limited by wrapping the task in a `LimitConcurrency` `Task`: this will queue the items and run them one-by-one (e.g., a single Rsync upload).
The `Pipeline` needs to be fed empty `Item` objects; by controlling the number of active `Item`s you can limit the number of items. (For example, add a new item each time an item leaves the pipeline.)
With the `ItemValue`, `ItemInterpolation` and `ConfigValue` classes it is possible to pass item-specific arguments to the `Task` objects. The value of these objects will be re-evaluated for each item. Examples: a path name that depends on the item name, a configurable bandwidth limit, the number of concurrent downloads.
Consult [the wiki](https://github.com/ArchiveTeam/seesaw-kit/wiki) for more information.
==============
An asynchronous toolkit for distributed web processing. Written in Python and named after its behavior, it supports concurrent downloads, uploads, etc.
This toolkit is well-known for [Archive Team projects](http://archiveteam.org). It also powers the [Archive Team warrior](http://archiveteam.org/index.php?title=Warrior).
[](http://travis-ci.org/ArchiveTeam/seesaw-kit)
Installation
------------
Requires Python 2 or 3.
Needs the Tornado library for event-driven I/O. The complete list of Python modules needed are listed in requirements.txt.
How to try it out
-----------------
To run the example pipeline:
sudo pip install -r requirements.txt
./run-pipeline --help
./run-pipeline examples/example-pipeline.py someone
Point your browser to `http://127.0.0.1:8001/`.
You can also use `run-pipeline2` or `run-pipeline3` to be explicit for the Python version.
Overview
--------
General idea: a set of `Task`s that can be combined into a `Pipeline` that processes `Item`s:
* An `Item` is a thing that needs to be downloaded (a user, for example). It has properties that are filled by the `Task`s.
* A `Task` is a step in the download process: it takes an item, does something with it and passes it on. Example Tasks: getting an item name from the tracker, running a download script, rsyncing the result, notifying the tracker that it's done.
* A `Pipeline` represents a sequence of `Task`s. To make a seesaw script for a new project you'd specify a new `Pipeline`.
A `Task` can work on multiple `Item`s at a time (e.g., multiple Wget downloads). The concurrency can be limited by wrapping the task in a `LimitConcurrency` `Task`: this will queue the items and run them one-by-one (e.g., a single Rsync upload).
The `Pipeline` needs to be fed empty `Item` objects; by controlling the number of active `Item`s you can limit the number of items. (For example, add a new item each time an item leaves the pipeline.)
With the `ItemValue`, `ItemInterpolation` and `ConfigValue` classes it is possible to pass item-specific arguments to the `Task` objects. The value of these objects will be re-evaluated for each item. Examples: a path name that depends on the item name, a configurable bandwidth limit, the number of concurrent downloads.
Consult [the wiki](https://github.com/ArchiveTeam/seesaw-kit/wiki) for more information.
2 maintainers
Metadata
2 maintainers
Release files for seesaw 0.8
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| seesaw-0.8.tar.gz | 113.2 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| seesaw-0.8-py2.7.egg | Legacy Egg format | - | - | Details |
Total release size: 289.0 kB
Release files / seesaw-0.8.tar.gz
| Download URL | seesaw-0.8.tar.gz |
|---|---|
| Size | 113.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
485003154220335d7f9f8a5d63acb2d720001314d0d4e8f2035840ec72896904
|
|
BLAKE2b-256 checksum How to use checksums |
33532ad37617df4063d1bad80289021e0ec0f49fa8cb43e0335ebf1c93a7184c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Release files / seesaw-0.8-py2.7.egg
| Download URL | seesaw-0.8-py2.7.egg |
|---|---|
| Size | 175.8 kB |
| Tags | Egg |
|
SHA-256 checksum How to use checksums |
aab289c8805061589fc0c64c5ab445ff9ad6c70672d48972b90795fca70b511e
|
|
BLAKE2b-256 checksum How to use checksums |
3564aa1173ec39bcb56db2657e2c9af4914c843289f8664aecce085a0f02069e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Security for seesaw 0.8
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
