Skip to main content
Join the official 2019 Python Developers SurveyStart the survey!

Like `requests`, but shittier.

Project description

WebRequest Build StatusCoverage Status

Like the requests library, but less good.

Provides convenience functions for writing web-scrapers and other web-interactive things. Built-in support for working through CloudFlare's garbage JS browser checks without any intervention, as well as some other garbage web-application firewalls that seem intent on breaking the internet.

Built-in user-agent randomization. Support for fetching rendered content via headless chrome. Built on top of my ChromeController project, so it can avoid some of the spectacularly stupid design decisions in selenium.

Default support for compressed transfers.

Basically, the overall goal is to have a simple library that acts as much as possible like a "real" browser. Ideally, it should be indistinguishable from an actual browser from the perspective of the remote HTTP(s) server.

Other useful bits:

API wrappers for and which include automatic local reverse-proxy spinup for reflecting recaptcha requests through your local host address. This involves spinning up a transient SOCKS5 proxy, setting up a limited duration port-forward (10 minutes, via UPnP), and actual synchronous captcha solving calls.

The proxy setup/UPnP setup functions are generic enough that they should be useful for any captcha-solver-related tasks. If you'd like support for another solver site (and are willing to throw a few bucks of credit on a solver site to me), I can probably add that other site too.

Q: Why
A: Because I started writing horrible web-scraper things in 2008, when the requests library wasn't really a thing.

Q: Why still, then?
A: Anger and spite, mostly.

Q: No, really, why
A: Ok, Because I want to download the internet, and idiots post stuff, and then try to "protect" it from scraping with stupid jerberscript bullshit.

Note: If your non-interactive webite requires me to execute javascript to view it, FUCK YOU, you are a horrible person who is actively ruining the internet.

License: WTFPL

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for WebRequest, version 0.0.60
Filename, size File type Python version Upload date Hashes
Filename, size WebRequest-0.0.60.tar.gz (40.9 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page