A set of tools for controlling processing workflow with spiders and script running in scrapinghub ScrapyCloud.

Installation

pip install shub-workflow

If you want to support s3 tools:

pip install shub-workflow[with-s3-tools]

For google cloud storage tools support:

pip install shub-workflow[with-gcs-tools]

Usage

Check Project Wiki for documentation. You can also see code tests for lots of examples of usage.

Claude Code plugin

shub-workflow ships a Claude Code plugin, shub-workflow-toolkit, that gives Claude working knowledge of shub-workflow tooling. It currently bundles eight skills:

scanjobs-programs — authoring and running the scanjobs job-scanning + plotting tool and its command-line "programs".
shub-workflow-scripts — writing or fixing scripts built on the shub_workflow.script base classes (BaseScript / BaseLoopScript / BaseLoopScriptAsyncMixin), i.e. any script that runs on or operates on Scrapy Cloud.
shub-workflow-crawl-managers — building, updating or understanding crawl managers (CrawlManager / PeriodicCrawlManager / GeneratorCrawlManager / AsyncSchedulerCrawlManagerMixin): the set_parameters_gen() pattern, outcome/retry hooks, and async scheduling.
shub-workflow-graph-managers — building, updating or understanding graph managers (GraphManager + Task / SpiderTask): declaring a DAG of tasks in configure_workflow(), dependency linking, on_finish/retry routing, resources, and parallelization.
shub-workflow-monitors — building, updating or understanding monitors (BaseMonitor): cross-job stat aggregation over a time window, ratios, reports, custom checks, and threshold alerts via Slack / Sentry.
shub-workflow-fshelper — reading/writing/listing files through the cloud-agnostic filesystem layer (shub_workflow.utils.futils / FSHelper): prefix-based s3:///gs:///local dispatch, role-assumed credentials, the listing variants, and op_kwargs/ACLs.
shub-workflow-issuers — building data-pipeline issuers (IssuerScript / IssuerScriptWithFileSystemInput / IssuerScriptWithSCJobInput): reading job/file input, dedup, batch output slots, and issuer-based delivery (replacing the deprecated BaseDeliverScript).
shub-workflow-diagrams — generating a project's architecture diagrams from an editable model: banded workflow/pipeline diagrams and UML class-hierarchy diagrams, in the house style shared by these projects, with the rendering engine to copy and the Graphviz constraints that keep the output stable.

Install it from this repository's plugin marketplace, from inside Claude Code:

/plugin marketplace add scrapinghub/shub-workflow
/plugin install shub-workflow-toolkit@shub-workflow

To enable it automatically for a project, add it to that project's .claude/settings.json:

{
  "enabledPlugins": ["shub-workflow-toolkit@shub-workflow"]
}

Updates

The plugin is unversioned (its plugin.json has no version field), so each commit pushed to this repository is a new version. When Claude Code installs the plugin it copies it into a local cache (~/.claude/plugins/cache/) and uses that copy — it does not read your working tree or re-pull from GitHub on every session. You choose how new commits reach you:

Automatic. Turn on auto-update for this marketplace: run /plugin, open the Marketplaces tab, and enable auto-update for shub-workflow (or set it in settings — see below). With this on, Claude Code re-pulls the marketplace from GitHub and updates installed plugins at startup, so a new session always loads the latest pushed commit. This is the low-friction option for staying current.
```
{
  "extraKnownMarketplaces": {
    "shub-workflow": {
      "source": { "source": "github", "repo": "scrapinghub/shub-workflow" },
      "autoUpdate": true
    }
  },
  "enabledPlugins": ["shub-workflow-toolkit@shub-workflow"]
}
```
Manual. Leave auto-update off (the default for third-party marketplaces). The cached copy stays pinned until you explicitly update — nothing changes under you between sessions. To pull the latest when you want it:
```
/plugin marketplace update shub-workflow            # refresh the catalog from GitHub
/plugin update shub-workflow-toolkit@shub-workflow  # update the installed plugin
```

The plugin lives in plugins/shub-workflow-toolkit/; the marketplace manifest is .claude-plugin/marketplace.json.

Note

The requirements for this library are defined in setup.py as usual. The Pipfile files in the repository don't define dependencies. It is only used for setting up a development environment for shub-workflow library development and testing.

For developers

For installing a development environment for shub-workflow, the package comes with Pipfile and Pipfile.lock files. So, clone or fork the repository and do:

> pipenv install --dev
> cp pre-commit .git/hooks/

for installing the environment, and:

> pipenv shell

for initiating it.

There is a script, lint.sh, that you can run everytime you need from the repo root folder, but it is also executed each time you do git commit (provided you installed the pre-commit hook during the installation step described above). It checks code pep8 and typing integrity, via flake8 and mypy.

> ./lint.sh

Metadata

Release files for shub-workflow 1.14.43

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for shub-workflow 1.14.43
File	Size	Uploaded
shub_workflow-1.14.43.tar.gz	97.8 kB	Oct 13, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for shub-workflow 1.14.43
File	Interpreter	ABI	Platform	Reset
shub_workflow-1.14.43-py3-none-any.whl 106.4 kB Oct 13, 2026	Python 3	none	any	Details

Total release size: 204.2 kB

Release files / shub_workflow-1.14.43.tar.gz

Download URL	shub_workflow-1.14.43.tar.gz
Size	97.8 kB
Tags	Source
SHA-256 checksum How to use checksums	`d873421cd3e1b24f618a74ee995c1f8fe009258f8c31a0fd552cb0e915a4a813`
BLAKE2b-256 checksum How to use checksums	`98f6b2b10118d09acedd276efadb07fcf6c729e9ef0d6559d68f15814f80c1d1`
Upload date	Oct 13, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.10.16`

Release files / shub_workflow-1.14.43-py3-none-any.whl

Download URL	shub_workflow-1.14.43-py3-none-any.whl
Size	106.4 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`f1e5c3224c57d95fa3d2f16a82e8e4c7aff102567439bb35f4d1a99c62c92b24`
BLAKE2b-256 checksum How to use checksums	`e4877abfd8fde32f5be162dd43710e39831be729f7fe7f0bd5d38786fcf42362`
Upload date	Oct 13, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.10.16`