Auto Generated by os-scrapy-cookiecutter
Project description
os-scrapy-uvicorn
This project provide a extension to start a ASGI http server(Uvicorn) along with Scrapy in the same process.
You can use the ASGI framework(recommend FastAPI) to create app to communicate with Scrapy.
Require: Python 3.6+, Scrapy 2.0+
Install
pip install os-scrapy-uvicorn
You can run example spider directly in the project root path
scrapy crawl example
Settings
-
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
or you can use os-scrapy(installed with this project) to start crawling with
-r
command line optionos-scrapy crawl -r asyncio example
-
enable extension
EXTENSIONS = { "os_scrapy_uvicorn.Uvicron": 1, }
-
app path, you can use ASGI app frameworks to create your app, FastAPI is recommended
UVICORN_APP = "app_module:app"
-
uvicorn server settings, supported settings
UVICORN_CONFIG = {"host": "0.0.0.0", "port": 5000}
-
when the server started, Scrapy crawler instance is attached to the app, it is the entrypoint to commnicate with Scrapy
Unit Tests
sh scripts/test.sh
License
MIT licensed.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.