scrapyx-core

Shared base spider and typed service configuration utilities for Scrapy projects.

What you get

• BaseServiceSpider — production-ready base class that:

  • exposes service_config, captcha_needed, site_key
  • collects items/errors, has yield_empty_item()
  • initializes from a typed Service registry (from settings / optional file)

• ServiceConfig / ServiceRegistry — Pydantic models for strict config.

• load_registry_from_settings() — merges settings["SERVICES"] with optional JSON file pointed by SCRAPYX_SERVICES_FILE.

• validators — functions you can call (or wire from an extension) to fail fast in CI/Scrapyd.

Minimal usage

# in your spiders
from scrapyx_core.base import BaseServiceSpider

class InspectionSpider(BaseServiceSpider):
    name = "inspection"
    # use self.service_config / self.captcha_needed / self.site_key

Optionally, set SCRAPYX_SERVICES_FILE=/path/to/services.json to load/merge external config.

This package is middleware-agnostic and pairs well with scrapyx-mw (add-on + middlewares).

Install and optional extras

Base usage (BaseServiceSpider, ServiceRegistry, config) requires only the default install:

uv add scrapyx-core
# or: pip install scrapyx-core

Extensions that need extra dependencies:

• WebhookExtension (webhook callbacks on spider close): uv add scrapyx-core[webhook] or pip install scrapyx-core[webhook]
• CompletionPublisherExtension (FastStream/RabbitMQ/Redis): uv add scrapyx-core[completion] or pip install scrapyx-core[completion]
• Both: uv add scrapyx-core[webhook,completion]