Scrapy-Heroku
A package to assist with running scrapy on heroku. This is accomplished by providing a custom application configuration at scrapy_heroku.app.application that launches the scrapyd web service using the PORT environment variable and a multi-process work queue implemented on a Postgres database specified by the DATABASE_URL environment variable.
Configuration
Create a git repo that has a scrapy project at the root (scrapy.cfg should be at the top level). Edit your scrapy.cfg to include the following:
[scrapyd] application = scrapy_heroku.app.application [deploy] url = http://<YOUR_HEROKU_APP_NAME>.herokuapp.com:80/ project = <YOUR_PROJECT_NAME> username = <A_USER_NAME> password = <A_PASSWORD>
Add a requirements.txt file that includes scrapy-heroku in it. It is strongly recommended that you version pin scrapy-heroku as well as the version of scrapy that your project is developed against (pip freeze > requirements.txt). Finally create a Procfile that consists of:
web: scrapy server
Make sure you have a postgres database that has been promoted to DATABASE_URL
Project page: <http://github.com/dmclain/scrapy-heroku>
1 maintainer
Release files for scrapy-heroku 0.7.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| scrapy-heroku-0.7.1.tar.gz | 5.2 kB | Details |
Release files / scrapy-heroku-0.7.1.tar.gz
| Download URL | scrapy-heroku-0.7.1.tar.gz |
|---|---|
| Size | 5.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a4d7d090442b55aaf83cf2957264ed210e869f7419fabb40b625ce47284a9a2b
|
|
BLAKE2b-256 checksum How to use checksums |
3b19876c22d5971aa48b86f7f25ee8abd6326d48f7abf7bf20f1fb799ece95e1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
