scrapy-ua-rotator
A modern, pluggable User-Agent rotator middleware for the Scrapy framework.
Supports rotation via:
fake-useragentFaker- Scrapy’s built-in
USER_AGENTsetting
Also supports per-proxy rotation and easy extensibility with custom providers.
📋 Requirements
- Python 3.9+
Faker >= 18.0.0fake-useragent >= 1.5.0
✅ Tested with: Scrapy 2.9, 2.10, 2.11, and 2.12
📦 Installation
pip install scrapy-ua-rotator
⚙️ Configuration
Disable Scrapy’s default middleware and enable ours:
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
'scrapy.downloadermiddlewares.retry.RetryMiddleware': None,
'scrapy_ua_rotator.middleware.RandomUserAgentMiddleware': 400,
'scrapy_ua_rotator.middleware.RetryUserAgentMiddleware': 550,
}
Recommended provider order:
USERAGENT_PROVIDERS = [
'scrapy_ua_rotator.providers.FakeUserAgentProvider', # Primary provider using the fake-useragent library
'scrapy_ua_rotator.providers.FakerProvider', # Fallback provider that generates synthetic UAs via Faker
'scrapy_ua_rotator.providers.FixedUserAgentProvider', # Final fallback: uses the static USER_AGENT setting
]
# Static user-agent string to be used if all providers fail to return a valid value
USER_AGENT = "Mozilla/5.0 (X11; Linux x86_64)..."
🧩 Provider Details
FakeUserAgentProvider
Assigns a new user-agent using fake-useragent.
Supports fine-tuned filtering via:
FAKE_USERAGENT_UA_TYPE = 'Chrome Mobile iOS' # str; browser to prioritize (default: 'random')
FAKE_USERAGENT_OS = ['Linux'] # str or list[str]; OS filter (default: None — all OSes)
FAKE_USERAGENT_PLATFORMS = ['mobile'] # str or list[str]; platform filter (default: None — all platforms)
FAKE_USERAGENT_FALLBACK = 'Mozilla/5.0 (...)' # str; fallback UA string (default: internal fallback)
💡 Note: See docs for supported options and advanced usage.
FakerProvider
Uses Faker to generate synthetic UA strings.
FAKER_UA_TYPE = 'chrome' # or 'firefox', 'safari', etc. (default: 'user_agent' — random web browser)
💡 Note: See docs for supported options and advanced usage.
FixedUserAgentProvider
Simply uses the provided USER_AGENT setting without rotation.
Useful as a fallback if other providers fail.
USER_AGENT = "Mozilla/5.0 ..."
🔀 Proxy-Aware Mode
If you’re using rotating proxies (e.g., via scrapy-proxies), enable per-proxy UA assignment:
RANDOM_UA_PER_PROXY = True
Make sure RandomUserAgentMiddleware has higher priority than your proxy middleware.
🧪 Example Output
To verify it’s working, log your request headers in your spider:
def parse(self, response):
self.logger.info("Using UA: %s", response.request.headers.get('User-Agent'))
🔧 Extending with Custom Providers
Add your own class:
USERAGENT_PROVIDERS = [
'your_project.providers.MyCustomProvider',
...
]
Just inherit from BaseProvider and implement get_random_ua().
🤝 Contributing
Contributions, suggestions, and issues are welcome!
Please see CONTRIBUTING.md for guidelines.
📄 License
MIT © Sergei Denisenko
See LICENSE
Metadata
Release files for scrapy-ua-rotator 1.0.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| scrapy_ua_rotator-1.0.1.tar.gz | 11.5 kB | Without provenance | Details for scrapy_ua_rotator-1.0.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| scrapy_ua_rotator-1.0.1-py3-none-any.whl | Python 3 | none | any | Details for scrapy_ua_rotator-1.0.1-py3-none-any.whl |
Total release size: 18.5 kB
Release files / scrapy_ua_rotator-1.0.1.tar.gz
| Download URL | scrapy_ua_rotator-1.0.1.tar.gz |
|---|---|
| Size | 11.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
32701c60190bc565be3cfcd2cb439cc40c15a7a6833165160e7e4ba90fab749f
|
|
BLAKE2b-256 checksum How to use checksums |
ccbb03b781cbf4705d2a43f3a2f58ee079b1dd8ac9c593a60929b46c25ded649
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.5
|
Release files / scrapy_ua_rotator-1.0.1-py3-none-any.whl
| Download URL | scrapy_ua_rotator-1.0.1-py3-none-any.whl |
|---|---|
| Size | 7.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
ff985f2b92736ecc3af0769bb276a51576738648a5bc8e700d697c06c30b7a49
|
|
BLAKE2b-256 checksum How to use checksums |
8ad58c33068c44c8ca3c3d8d0f9c6c9f813c0bcc5a6b79d62ad665e5f2eb42c4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.5
|
Security for scrapy-ua-rotator 1.0.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.