Skip to main content

Scrapy Settings Log

An extension that allows a user to display all or some of their scrapy spider settings at runtime. It will add a logline with some or all settings for your spider in JSON compatible format.

Install

pip install scrapy-settings-log

Quick-Start

Add the following to your settings:

EXTENSIONS = {
    'scrapy_settings_log.SpiderSettingsLogging': 999,
}

SETTINGS_LOGGING_ENABLED = True

When you run your spider you will see a log like below when spider is closing:

[scrapy_settings_log] INFO: {"SETTINGS_LOGGING_ENABLED": true, ...}

Additional Options

  • SETTINGS_LOGGING_REGEX - Add a regular expression to only show some settings - for example SETTINGS_LOGGING_REGEX = "SPIDERMON" will show settings with SPIDERMON in their name.
  • SETTINGS_LOGGING_INDENT - Add indentation to make log more human-readable.
  • MASKED_SENSITIVE_SETTINGS_ENABLED - Default is True - if settings logging is enabled it will mask the value of settings that may be sensitive (password, apikey). For example AWS_SECRET_ACCESS_KEY will have their value shown as **********

Advanced

Subclass and override the output_settings method if you want the settings to be reported in another way.

from scrapy_settings_log import SpiderSettingsLogging

class CustomSettingsLogger(SpiderSettingsLogging):

    def  output_settings(self, settings: dict, spider: scrapy.Spider):
        # custom code here

Metadata

Release files for scrapy-settings-log 1.4.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for scrapy-settings-log 1.4.0
File Size Uploaded Provenance Actions
scrapy_settings_log-1.4.0.tar.gz 16.4 kB Without provenance Details for scrapy_settings_log-1.4.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for scrapy-settings-log 1.4.0
File Interpreter ABI Platform Actions
scrapy_settings_log-1.4.0-py3-none-any.whl Python 3 none any Details for scrapy_settings_log-1.4.0-py3-none-any.whl

Total release size: 32.0 kB

Release files / scrapy_settings_log-1.4.0.tar.gz

Download URL scrapy_settings_log-1.4.0.tar.gz
Size 16.4 kB
Tags Source
SHA-256 checksum
How to use checksums
a7dbac1be30e5eb92d05bd9f7812202702459309ea4c06de0353f29339d497dc
BLAKE2b-256 checksum
How to use checksums
7c2f338af11430818db3e1bf879f6fe6c4cc4e1ff128d28de4faa7d572a0c62a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/5.1.1 CPython/3.11.9

Release files / scrapy_settings_log-1.4.0-py3-none-any.whl

Download URL scrapy_settings_log-1.4.0-py3-none-any.whl
Size 15.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
881bc113cb30226e2d7db961febee48a438eecb4ed42ab49c34b2ede88a929be
BLAKE2b-256 checksum
How to use checksums
eea6c7b36ab18351ca85ff474212d577c2470b442dd1db5a421b61e29d8e0a27
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/5.1.1 CPython/3.11.9

Security for scrapy-settings-log 1.4.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page