Does your scrapy spider get identified and blocked by servers because you use the default user-agent or a generic one?
Use this random_useragent module and set a random user-agent for every request.
Installing
Installing it is pretty simple.
pip install git+https://github.com/cleocn/scrapy-random-useragent.gitUsage
In your settings.py file, update the DOWNLOADER_MIDDLEWARES variable like this.
DOWNLOADER_MIDDLEWARES = {
'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,
'random_useragent.RandomUserAgentMiddleware': 400
}This disables the default UserAgentMiddleware and enables the RandomUserAgentMiddleware.
Now all the requests from your crawler will have a random user-agent.
1 maintainer
Release files for scrapy-random-ua 0.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| scrapy-random-ua-0.3.tar.gz | 3.2 kB | Details |
Release files / scrapy-random-ua-0.3.tar.gz
| Download URL | scrapy-random-ua-0.3.tar.gz |
|---|---|
| Size | 3.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0b21843e0eb86eb7351c0762c1851343b865d4a0b750146250f0c86581e2b972
|
|
BLAKE2b-256 checksum How to use checksums |
0b6fd48d5c0f651e2ede25637ca91584550f793dc967f68f56e56541a154e216
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
