Scrapy downloader middleware to interact with agentfive API
Project description
This package provides a Scrapy Downloader Middleware to interact with the agentfive API.
Requirements
- Python 3.5+
- Scrapy 1.6+
Installation
pip install scrapy-agentfive-middleware
Configuration
Enable the AgentfiveMiddleware
via the
DOWNLOADER_MIDDLEWARES
setting:
DOWNLOADER_MIDDLEWARES = {
"agentfive_middleware.AgentfiveMiddleware": 585,
}
Please note that the middleware needs to be placed before the built-in HttpCompressionMiddleware
middleware (which has a priority of 590), otherwise incoming responses will be compressed and the
agentfive middleware won't be able to handle them.
Settings
-
AGENTFIVE_KEY
(typestr
)API key to be used to authenticate against the agentfive API.
-
AGENTFIVE_API_URL
(Typestr
, default"https://api.agentfive.cn/v1"
)The endpoint of a agentfive API.
-
AGENTFIVE_DEFAULT_ARGS
(typedict
, default{}
)Default values to be sent to the agentfive API. For instance, set to
{"profile": "mobile"}
to set all requests with a mobile profile.
Usage
If the middleware is enabled, by default all requests will be redirected to the specified agentfive API endpoint, and append necessary params which agentfive API expected.
For example:
scrapy.Request(url="https://httpbin.org/anything")
will be set to agentfive API and let agentfive to fetch the url.
Additional arguments
Additional arguments could be specified under the agentfive
Request.meta
key. For instance:
Request(
url="https://example.org",
meta={"crawlera_fetch": {"render": True, "wait_ms": 5000}},
)
for more information on agentfive API parameter, please refer to agentfive document.
Skipping requests
You can instruct the middleware to skip a specific request by setting the agentfive.skip
Request.meta
key:
Request(
url="https://example.org",
meta={"agentfive": {"skip": True}},
)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for scrapy-agentfive-middleware-0.0.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 074f20110f34d9abdab636d4e79a1bac05f2237ed4623cd2223e03c6606911ca |
|
MD5 | 3e1af596860de98395ae0d31ef5306ed |
|
BLAKE2b-256 | 6403b35f52b84eb5ed484e63c85c7c9925b12b7d772e8f4a8590239fa2d544ad |
Hashes for scrapy_agentfive_middleware-0.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | da045965320d380a4f996d6bd121f6b6be27ac590b54e141d0b208c69eba0c6c |
|
MD5 | 359898ebee67cbdcb7e1cd43144c70c8 |
|
BLAKE2b-256 | 02dfe0caaca4d06852fd12731043711fea60caf85181d58d0545258323a8dc9d |