A set of tools for controlling processing workflow with spiders and script running in scrapinghub ScrapyCloud.

Installation

pip install shub-workflow

If you want to support s3 tools:

pip install shub-workflow[with-s3-tools]

For google cloud storage tools support:

pip install shub-workflow[with-gcs-tools]

Usage

Check Project Wiki for documentation. You can also see code tests for lots of examples of usage.

Claude Code plugin

shub-workflow ships a Claude Code plugin, shub-workflow-toolkit, that gives Claude working knowledge of shub-workflow tooling. It currently bundles seven skills:

scanjobs-programs — authoring and running the scanjobs job-scanning + plotting tool and its command-line "programs".
shub-workflow-scripts — writing or fixing scripts built on the shub_workflow.script base classes (BaseScript / BaseLoopScript / BaseLoopScriptAsyncMixin), i.e. any script that runs on or operates on Scrapy Cloud.
shub-workflow-crawl-managers — building, updating or understanding crawl managers (CrawlManager / PeriodicCrawlManager / GeneratorCrawlManager / AsyncSchedulerCrawlManagerMixin): the set_parameters_gen() pattern, outcome/retry hooks, and async scheduling.
shub-workflow-graph-managers — building, updating or understanding graph managers (GraphManager + Task / SpiderTask): declaring a DAG of tasks in configure_workflow(), dependency linking, on_finish/retry routing, resources, and parallelization.
shub-workflow-monitors — building, updating or understanding monitors (BaseMonitor): cross-job stat aggregation over a time window, ratios, reports, custom checks, and threshold alerts via Slack / Sentry.
shub-workflow-fshelper — reading/writing/listing files through the cloud-agnostic filesystem layer (shub_workflow.utils.futils / FSHelper): prefix-based s3:///gs:///local dispatch, role-assumed credentials, the listing variants, and op_kwargs/ACLs.
shub-workflow-issuers — building data-pipeline issuers (IssuerScript / IssuerScriptWithFileSystemInput / IssuerScriptWithSCJobInput): reading job/file input, dedup, batch output slots, and issuer-based delivery (replacing the deprecated BaseDeliverScript).

Install it from this repository's plugin marketplace, from inside Claude Code:

/plugin marketplace add scrapinghub/shub-workflow
/plugin install shub-workflow-toolkit@shub-workflow

To enable it automatically for a project, add it to that project's .claude/settings.json:

{
  "enabledPlugins": ["shub-workflow-toolkit@shub-workflow"]
}

Updates

The plugin is unversioned (its plugin.json has no version field), so each commit pushed to this repository is a new version. When Claude Code installs the plugin it copies it into a local cache (~/.claude/plugins/cache/) and uses that copy — it does not read your working tree or re-pull from GitHub on every session. You choose how new commits reach you:

Automatic. Turn on auto-update for this marketplace: run /plugin, open the Marketplaces tab, and enable auto-update for shub-workflow (or set it in settings — see below). With this on, Claude Code re-pulls the marketplace from GitHub and updates installed plugins at startup, so a new session always loads the latest pushed commit. This is the low-friction option for staying current.
```
{
  "extraKnownMarketplaces": {
    "shub-workflow": {
      "source": { "source": "github", "repo": "scrapinghub/shub-workflow" },
      "autoUpdate": true
    }
  },
  "enabledPlugins": ["shub-workflow-toolkit@shub-workflow"]
}
```
Manual. Leave auto-update off (the default for third-party marketplaces). The cached copy stays pinned until you explicitly update — nothing changes under you between sessions. To pull the latest when you want it:
```
/plugin marketplace update shub-workflow            # refresh the catalog from GitHub
/plugin update shub-workflow-toolkit@shub-workflow  # update the installed plugin
```

The plugin lives in plugins/shub-workflow-toolkit/; the marketplace manifest is .claude-plugin/marketplace.json.

Note

The requirements for this library are defined in setup.py as usual. The Pipfile files in the repository don't define dependencies. It is only used for setting up a development environment for shub-workflow library development and testing.

For developers

For installing a development environment for shub-workflow, the package comes with Pipfile and Pipfile.lock files. So, clone or fork the repository and do:

> pipenv install --dev
> cp pre-commit .git/hooks/

for installing the environment, and:

> pipenv shell

for initiating it.

There is a script, lint.sh, that you can run everytime you need from the repo root folder, but it is also executed each time you do git commit (provided you installed the pre-commit hook during the installation step described above). It checks code pep8 and typing integrity, via flake8 and mypy.

> ./lint.sh

Metadata

Release files for shub-workflow 1.14.42.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for shub-workflow 1.14.42.3
File	Size	Uploaded
shub_workflow-1.14.42.3.tar.gz	97.2 kB	Sep 22, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for shub-workflow 1.14.42.3
File	Interpreter	ABI	Platform	Reset
shub_workflow-1.14.42.3-py3-none-any.whl 105.9 kB Sep 22, 2026	Python 3	none	any	Details

Total release size: 203.1 kB

Release files / shub_workflow-1.14.42.3.tar.gz

Download URL	shub_workflow-1.14.42.3.tar.gz
Size	97.2 kB
Tags	Source
SHA-256 checksum How to use checksums	`74db5658c8601e423c7572429d283cd92f121e3b7e5d8fca918bc7da9c94be92`
BLAKE2b-256 checksum How to use checksums	`28352f8e079dc21a809c9b67a8c269bc09150a1e025ca9a6ab966d24062277fe`
Upload date	Sep 22, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.10.16`

Release files / shub_workflow-1.14.42.3-py3-none-any.whl

Download URL	shub_workflow-1.14.42.3-py3-none-any.whl
Size	105.9 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`ecaf0a61449b2cb5aeb6c75a2795cd3844c9e714f26803f088ac44eb52d38559`
BLAKE2b-256 checksum How to use checksums	`743127deba8f6f1e94649c0e3d690e63ac791e13ea22e754235c328adb7bcc90`
Upload date	Sep 22, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.10.16`