A set of tools for controlling processing workflow with spiders and script running in scrapinghub ScrapyCloud.

Installation

pip install shub-workflow

If you want to support s3 tools:

pip install shub-workflow[with-s3-tools]

For google cloud storage tools support:

pip install shub-workflow[with-gcs-tools]

Usage

Check Project Wiki for documentation. You can also see code tests for lots of examples of usage.

Claude Code plugin

shub-workflow ships a Claude Code plugin, shub-workflow-toolkit, that gives Claude working knowledge of shub-workflow tooling. It currently bundles seven skills:

scanjobs-programs — authoring and running the scanjobs job-scanning + plotting tool and its command-line "programs".
shub-workflow-scripts — writing or fixing scripts built on the shub_workflow.script base classes (BaseScript / BaseLoopScript / BaseLoopScriptAsyncMixin), i.e. any script that runs on or operates on Scrapy Cloud.
shub-workflow-crawl-managers — building, updating or understanding crawl managers (CrawlManager / PeriodicCrawlManager / GeneratorCrawlManager / AsyncSchedulerCrawlManagerMixin): the set_parameters_gen() pattern, outcome/retry hooks, and async scheduling.
shub-workflow-graph-managers — building, updating or understanding graph managers (GraphManager + Task / SpiderTask): declaring a DAG of tasks in configure_workflow(), dependency linking, on_finish/retry routing, resources, and parallelization.
shub-workflow-monitors — building, updating or understanding monitors (BaseMonitor): cross-job stat aggregation over a time window, ratios, reports, custom checks, and threshold alerts via Slack / Sentry.
shub-workflow-fshelper — reading/writing/listing files through the cloud-agnostic filesystem layer (shub_workflow.utils.futils / FSHelper): prefix-based s3:///gs:///local dispatch, role-assumed credentials, the listing variants, and op_kwargs/ACLs.
shub-workflow-issuers — building data-pipeline issuers (IssuerScript / IssuerScriptWithFileSystemInput / IssuerScriptWithSCJobInput): reading job/file input, dedup, batch output slots, and issuer-based delivery (replacing the deprecated BaseDeliverScript).

Install it from this repository's plugin marketplace, from inside Claude Code:

/plugin marketplace add scrapinghub/shub-workflow
/plugin install shub-workflow-toolkit@shub-workflow

To enable it automatically for a project, add it to that project's .claude/settings.json:

{
  "enabledPlugins": ["shub-workflow-toolkit@shub-workflow"]
}

Updates

The plugin is unversioned (its plugin.json has no version field), so each commit pushed to this repository is a new version. When Claude Code installs the plugin it copies it into a local cache (~/.claude/plugins/cache/) and uses that copy — it does not read your working tree or re-pull from GitHub on every session. You choose how new commits reach you:

Automatic. Turn on auto-update for this marketplace: run /plugin, open the Marketplaces tab, and enable auto-update for shub-workflow (or set it in settings — see below). With this on, Claude Code re-pulls the marketplace from GitHub and updates installed plugins at startup, so a new session always loads the latest pushed commit. This is the low-friction option for staying current.
```
{
  "extraKnownMarketplaces": {
    "shub-workflow": {
      "source": { "source": "github", "repo": "scrapinghub/shub-workflow" },
      "autoUpdate": true
    }
  },
  "enabledPlugins": ["shub-workflow-toolkit@shub-workflow"]
}
```
Manual. Leave auto-update off (the default for third-party marketplaces). The cached copy stays pinned until you explicitly update — nothing changes under you between sessions. To pull the latest when you want it:
```
/plugin marketplace update shub-workflow            # refresh the catalog from GitHub
/plugin update shub-workflow-toolkit@shub-workflow  # update the installed plugin
```

The plugin lives in plugins/shub-workflow-toolkit/; the marketplace manifest is .claude-plugin/marketplace.json.

Note

The requirements for this library are defined in setup.py as usual. The Pipfile files in the repository don't define dependencies. It is only used for setting up a development environment for shub-workflow library development and testing.

For developers

For installing a development environment for shub-workflow, the package comes with Pipfile and Pipfile.lock files. So, clone or fork the repository and do:

> pipenv install --dev
> cp pre-commit .git/hooks/

for installing the environment, and:

> pipenv shell

for initiating it.

There is a script, lint.sh, that you can run everytime you need from the repo root folder, but it is also executed each time you do git commit (provided you installed the pre-commit hook during the installation step described above). It checks code pep8 and typing integrity, via flake8 and mypy.

> ./lint.sh

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

shub_workflow-1.14.39.tar.gz (90.5 kB view details)

Uploaded Jul 13, 2026 Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

shub_workflow-1.14.39-py3-none-any.whl (99.6 kB view details)

Uploaded Jul 13, 2026 Python 3

File details

Details for the file shub_workflow-1.14.39.tar.gz.

File metadata

Download URL: shub_workflow-1.14.39.tar.gz
Upload date: Jul 13, 2026
Size: 90.5 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: twine/5.1.1 CPython/3.10.16

File hashes

Hashes for shub_workflow-1.14.39.tar.gz
Algorithm	Hash digest
SHA256	`a3276cd77d3d148619672286713289dd2a953c426e61a965ab84792b960f87e9`
MD5	`3cbca8ff9ae2784ed87bd2de12e22962`
BLAKE2b-256	`5886bd33f39a0f4810d1bda6d2012652bb0fe1efe05f65631037a74ba73bcca5`

See more details on using hashes here.

File details

Details for the file shub_workflow-1.14.39-py3-none-any.whl.

File metadata

Download URL: shub_workflow-1.14.39-py3-none-any.whl
Upload date: Jul 13, 2026
Size: 99.6 kB
Tags: Python 3
Uploaded using Trusted Publishing? No
Uploaded via: twine/5.1.1 CPython/3.10.16

File hashes

Hashes for shub_workflow-1.14.39-py3-none-any.whl
Algorithm	Hash digest
SHA256	`63149367343078b4695c9db76aca6ca24297239684c7f5a12b9de05620e414a9`
MD5	`1f50eb4d2aa45672e7522549d4a20117`
BLAKE2b-256	`3a7276e1156a49c38f32edb5aac5729f91ab2c128bb2bfea85b3e267a51afb3d`