scrapy-ua-rotator

A modern, pluggable User-Agent rotator middleware for the Scrapy framework.

Supports rotation via:

• fake-useragent
• Faker
• Scrapy’s built-in USER_AGENT setting

Also supports per-proxy rotation and easy extensibility with custom providers.

---

📋 Requirements

• Python 3.9+
• Faker >= 18.0.0
• fake-useragent >= 1.5.0

✅ Tested with: Scrapy 2.9, 2.10, 2.11, and 2.12

---

📦 Installation

pip install scrapy-ua-rotator

---

⚙️ Configuration

Disable Scrapy’s default middleware and enable ours:

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
    'scrapy.downloadermiddlewares.retry.RetryMiddleware': None,
    'scrapy_ua_rotator.middleware.RandomUserAgentMiddleware': 400,
    'scrapy_ua_rotator.middleware.RetryUserAgentMiddleware': 550,
}

Recommended provider order:

USERAGENT_PROVIDERS = [
    'scrapy_ua_rotator.providers.FakeUserAgentProvider',  # Primary provider using the fake-useragent library
    'scrapy_ua_rotator.providers.FakerProvider',          # Fallback provider that generates synthetic UAs via Faker
    'scrapy_ua_rotator.providers.FixedUserAgentProvider', # Final fallback: uses the static USER_AGENT setting
]

# Static user-agent string to be used if all providers fail to return a valid value
USER_AGENT = "Mozilla/5.0 (X11; Linux x86_64)..."

---

🧩 Provider Details

FakeUserAgentProvider

Assigns a new user-agent using fake-useragent.
Supports fine-tuned filtering via:

FAKE_USERAGENT_UA_TYPE = 'Chrome Mobile iOS'           # str; browser to prioritize (default: 'random')
FAKE_USERAGENT_OS = ['Linux']                          # str or list[str]; OS filter (default: None — all OSes)
FAKE_USERAGENT_PLATFORMS = ['mobile']                  # str or list[str]; platform filter (default: None — all platforms)
FAKE_USERAGENT_FALLBACK = 'Mozilla/5.0 (...)'          # str; fallback UA string (default: internal fallback)

💡 Note: See docs for supported options and advanced usage.

FakerProvider

Uses Faker to generate synthetic UA strings.

FAKER_UA_TYPE = 'chrome'  # or 'firefox', 'safari', etc. (default: 'user_agent' — random web browser)

💡 Note: See docs for supported options and advanced usage.

FixedUserAgentProvider

Simply uses the provided USER_AGENT setting without rotation.
Useful as a fallback if other providers fail.

USER_AGENT = "Mozilla/5.0 ..."

---

🔀 Proxy-Aware Mode

If you’re using rotating proxies (e.g., via scrapy-proxies), enable per-proxy UA assignment:

RANDOM_UA_PER_PROXY = True

Make sure RandomUserAgentMiddleware has higher priority than your proxy middleware.

---

🧪 Example Output

To verify it’s working, log your request headers in your spider:

def parse(self, response):
    self.logger.info("Using UA: %s", response.request.headers.get('User-Agent'))

---

🔧 Extending with Custom Providers

Add your own class:

USERAGENT_PROVIDERS = [
    'your_project.providers.MyCustomProvider',
    ...
]

Just inherit from BaseProvider and implement get_random_ua().

---

🤝 Contributing

Contributions, suggestions, and issues are welcome!
Please see CONTRIBUTING.md for guidelines.

---

📄 License

MIT © Sergei Denisenko
See LICENSE