A set of tools for controlling processing workflow with spiders and script running in scrapinghub ScrapyCloud.
Installation
pip install shub-workflow
If you want to support s3 tools:
pip install shub-workflow[with-s3-tools]
For google cloud storage tools support:
pip install shub-workflow[with-gcs-tools]
Usage
Check Project Wiki for documentation. You can also see code tests for lots of examples of usage.
Claude Code plugin
shub-workflow ships a Claude Code plugin, shub-workflow-toolkit, that gives Claude working knowledge of shub-workflow tooling. It currently bundles seven skills:
- scanjobs-programs — authoring and running the
scanjobsjob-scanning + plotting tool and its command-line "programs". - shub-workflow-scripts — writing or fixing scripts built on the
shub_workflow.scriptbase classes (BaseScript/BaseLoopScript/BaseLoopScriptAsyncMixin), i.e. any script that runs on or operates on Scrapy Cloud. - shub-workflow-crawl-managers — building, updating or understanding crawl managers
(
CrawlManager/PeriodicCrawlManager/GeneratorCrawlManager/AsyncSchedulerCrawlManagerMixin): theset_parameters_gen()pattern, outcome/retry hooks, and async scheduling. - shub-workflow-graph-managers — building, updating or understanding graph managers
(
GraphManager+Task/SpiderTask): declaring a DAG of tasks inconfigure_workflow(), dependency linking,on_finish/retry routing, resources, and parallelization. - shub-workflow-monitors — building, updating or understanding monitors (
BaseMonitor): cross-job stat aggregation over a time window, ratios, reports, custom checks, and threshold alerts via Slack / Sentry. - shub-workflow-fshelper — reading/writing/listing files through the cloud-agnostic filesystem
layer (
shub_workflow.utils.futils/FSHelper): prefix-baseds3:///gs:///local dispatch, role-assumed credentials, the listing variants, andop_kwargs/ACLs. - shub-workflow-issuers — building data-pipeline issuers (
IssuerScript/IssuerScriptWithFileSystemInput/IssuerScriptWithSCJobInput): reading job/file input, dedup, batch output slots, and issuer-based delivery (replacing the deprecatedBaseDeliverScript).
Install it from this repository's plugin marketplace, from inside Claude Code:
/plugin marketplace add scrapinghub/shub-workflow
/plugin install shub-workflow-toolkit@shub-workflow
To enable it automatically for a project, add it to that project's .claude/settings.json:
{
"enabledPlugins": ["shub-workflow-toolkit@shub-workflow"]
}
Updates
The plugin is unversioned (its plugin.json has no version field), so each commit pushed to
this repository is a new version. When Claude Code installs the plugin it copies it into a local
cache (~/.claude/plugins/cache/) and uses that copy — it does not read your working tree or
re-pull from GitHub on every session. You choose how new commits reach you:
-
Automatic. Turn on auto-update for this marketplace: run
/plugin, open the Marketplaces tab, and enable auto-update forshub-workflow(or set it in settings — see below). With this on, Claude Code re-pulls the marketplace from GitHub and updates installed plugins at startup, so a new session always loads the latest pushed commit. This is the low-friction option for staying current.{ "extraKnownMarketplaces": { "shub-workflow": { "source": { "source": "github", "repo": "scrapinghub/shub-workflow" }, "autoUpdate": true } }, "enabledPlugins": ["shub-workflow-toolkit@shub-workflow"] }
-
Manual. Leave auto-update off (the default for third-party marketplaces). The cached copy stays pinned until you explicitly update — nothing changes under you between sessions. To pull the latest when you want it:
/plugin marketplace update shub-workflow # refresh the catalog from GitHub /plugin update shub-workflow-toolkit@shub-workflow # update the installed plugin
The plugin lives in plugins/shub-workflow-toolkit/; the
marketplace manifest is .claude-plugin/marketplace.json.
Note
The requirements for this library are defined in setup.py as usual. The Pipfile files in the repository don't define dependencies. It is only used for setting up a development environment for shub-workflow library development and testing.
For developers
For installing a development environment for shub-workflow, the package comes with Pipfile and Pipfile.lock files. So, clone or fork the repository and do:
> pipenv install --dev
> cp pre-commit .git/hooks/
for installing the environment, and:
> pipenv shell
for initiating it.
There is a script, lint.sh, that you can run everytime you need from the repo root folder, but it is also executed each time you do git commit (provided
you installed the pre-commit hook during the installation step described above). It checks code pep8 and typing integrity, via flake8 and mypy.
> ./lint.sh
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file shub_workflow-1.14.40.tar.gz.
File metadata
- Download URL: shub_workflow-1.14.40.tar.gz
- Upload date:
- Size: 91.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/5.1.1 CPython/3.10.16
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
22917330610563f171b1a2974acf79efb3dc0c75031d3e013ca777a5312d37c9
|
|
| MD5 |
3fb4e9813fa53f2adb9b0d99a638e42c
|
|
| BLAKE2b-256 |
b63dac7e025dbea2805bf2cf9da2d4bf4f543b9e7b4996c5781a29f1ca385a85
|
File details
Details for the file shub_workflow-1.14.40-py3-none-any.whl.
File metadata
- Download URL: shub_workflow-1.14.40-py3-none-any.whl
- Upload date:
- Size: 100.4 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/5.1.1 CPython/3.10.16
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
8726f7a3c1f07451caaf096760467642eb5af44ea15594004a1db0ee19ddbcfb
|
|
| MD5 |
1365d494c7bca09534c2ac6b3e8b2c6e
|
|
| BLAKE2b-256 |
33a4737d9f7435b4b68f7e1860d6ff3a03c02cf17bdb2101af3ec2e3a6b3ab62
|
