Features

• Provides a docs tox environment.

• Generates a Read the Docs configuration.

• Keeps the sphinx-scrapy pins of tox.ini and docs/requirements.in in sync with the pre-commit rev.

• Makes documentation builds fail on Sphinx warnings. While you fix existing warnings, you can either suppress them by type in docs/conf.py, or allow all of them in pyproject.toml:

  [tool.sphinx-scrapy]
  fail-on-warning = false

• Enables commonly-used Sphinx extensions.

• Configures sphinx.ext.intersphinx for Python and Scrapy, and streamlines configuration for additional packages.

• Allows you to easily link to Scrapy settings, request metadata keys, signals, commands and stats:

  :setting:`BOT_NAME`
  :setting:`LOG_LEVEL <scrapy:LOG_LEVEL>`
  :reqmeta:`download_slot`
  :signal:`spider_opened`
  :command:`crawl`
  :stat:`item_scraped_count`

• Allows you to easily link to GitHub issues and pull requests:

  :gh:`1908`
  :gh:`parsel#12`
  :gh:`sphinx-doc/sphinx#13000`

  They render as #1908, parsel#12 and sphinx#13000. Use the number alone for the current repository, which comes from the GitHub URLs in pyproject.toml. For a different repository, its name alone is enough if it is a known repository of the Scrapy ecosystem or it belongs to the same owner as the current repository.

Setup

1. Configure in pyproject.toml the Python version for documentation builds, e.g.:

   [tool.sphinx-scrapy]
   python-version = "3.14"

   It must be supported by Read the Docs.

2. Add to docs/requirements.txt:

   sphinx-scrapy==0.14.0

3. Add to docs/conf.py:

   extensions = [
       "sphinx_scrapy",
   ]

   To automatically configure sphinx.ext.intersphinx for installed supported packages, set:

   scrapy_intersphinx_enable_installed = True

   You can also enable or disable the automatic sphinx.ext.intersphinx configuration of packages manually:

   scrapy_intersphinx_enable = [
       "parsel",
       "w3lib",
   ]
   scrapy_intersphinx_disable = [
       "scrapy",
   ]

   The html_baseurl option for sitemap generation is generated automatically based on the project name (pyproject.toml or conf.py) and known documentation URLs (the same used for easy intersphinx configuration) with a fallback to https://<project>.readthedocs.io/en/latest/. You can alternatively define the setting yourself in conf.py.

4. Add to docs/.gitignore:

   /_build/

5. Add to .pre-commit-config.yaml:

   repos:
   - repo: https://github.com/scrapy/sphinx-scrapy
       rev: 0.14.0
       hooks:
       - id: sphinx-scrapy

6. Add to tox.ini:

   [tox]
   requires =
       sphinx-scrapy[tox]==0.14.0
   envlist = …,docs

You can now build the docs with:

tox -e docs

Sphinx extensions

The following Sphinx extensions are automatically enabled when you enable sphinx_scrapy:

• sphinx.ext.autodoc

• sphinx.ext.intersphinx

• sphinx.ext.viewcode

• sphinx_copybutton

• sphinx_design

• sphinx-llm-friendly, which documents the llm_friendly_* settings

• sphinxcontrib.youtube

Intersphinx packages

scrapy_intersphinx_enable supports the following packages:

Release notes

See the release notes for a list of releases and their changes.