Skip to main content

A package providing code generation command for scrapy CLI

Project description

scrapy-command-new

A package providing code generation command for scrapy CLI.

The project is a WIP, so expect major changes and additions (latter, mostly). Master branch is to be considered as always ready to use, with major changes/features introduced in feature branches.

This is a part of a bigger project - Scrapy Boilerplate.

The command works with a specific scrapy project structure (not the default one). Rationale for this is described here.

Usage

This is a scrapy command to generate class files and automatically add imports to respective module's __init__ files. It can be used as follows:

scrapy new spider SampleSpider

The first argument (spider) is a type of class file to be generated, and can be one of the following:

  • command
  • extension
  • item
  • middleware
  • model
  • pipeline
  • spider_middleware
  • spider

The second argument is class name.

Also for pipeline and spider class an option --rabbit can be used to add RabbitMQ connection code to generated source.

Option --item with value CLASSNAME is supported for generating pipelines, which adds an import and type-check for a provided item class to the resulting code.

Option --settings is also supported for pipelines, with an optional integer value PRIORITY that adds the pipeline being generated to scrapy settings.py file with specified priority (default 300).

Installation

This command is included in the Scrapy Boilerplate out of the box. If you want to install it manually, you can get it from PyPi:

pip install scrapy-new

Please note that this package won't work with default Scrapy project structure, it requires a specific custom one, as described here.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for scrapy-new, version 0.0.8
Filename, size File type Python version Upload date Hashes
Filename, size scrapy_new-0.0.8-py3-none-any.whl (10.2 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size scrapy_new-0.0.8.tar.gz (6.9 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page