Skip to main content

fetch in Python using your browser!

Project description

Fetch using your browser.

Let the browser manage cookies for you.

⚠️ Incomplete. Not tested thoroughly. Consider using Playwright, especially for more complex scenarios.

Usage

  1. You’ll run a Python script containing some code like this:

from asyncio import gather, new_event_loop

from browserfetch import fetch, get, post, run_server


async def main():
    response1, response2, reponse3 = await gather(
        get('https://example.com/path1', params={'a': 1}),
        fetch('https://example.com/image.png'),
        post('https://example.com/path2', data={'a': 1}),
    )
    # do stuff with retrieved responses


loop = new_event_loop()
loop.create_task(start_server())
loop.run_until_complete(main())
  1. Open your browser, goto http://example.com (perhaps solve a captcha and log in).

  2. Copy the contents of browserfetch.js file and paste it in browser’s console. (You can use a browser extensions like violentmonkey/tampermonkey to do this step for you.)

That’s it! Your Python script starts handling requests. The browser tab should remain open of-coarse.

The server can handle multiple websocket connections from different websites simultaneously.

How it works

browserfetch communicates with your browser using a websocket. The fetch function just passes the request to browser and it is the browser that handles the actual request. Response data is sent back to Python using the same WebSocket connection.

Motivations

  • browser_cookie3 stopped working on Chrome-based browsers. There is a workaround: ShadowCopy, but it requires admin privilege.

  • Another issue with browser_cookie’s approach is that it retrieves cookies from cookie files, but these files are not updated instantly. Thus, you might have to wait or retry a few times before you can successfully access newly set cookies.

  • ShadowCopying and File access are slow and inefficient operations.

Downsides

  • Setting up browserfetch is more cumbersome since it requires running a Python server and also injecting a small script into the webpage. Using browser_cookie3 might be a better choice if there are many websites that you need to communicate with.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

browserfetch-0.8.1.tar.gz (18.5 kB view details)

Uploaded Source

Built Distribution

browserfetch-0.8.1-py3-none-any.whl (19.3 kB view details)

Uploaded Python 3

File details

Details for the file browserfetch-0.8.1.tar.gz.

File metadata

  • Download URL: browserfetch-0.8.1.tar.gz
  • Upload date:
  • Size: 18.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.12.4

File hashes

Hashes for browserfetch-0.8.1.tar.gz
Algorithm Hash digest
SHA256 c9f876b5f98caef197f4b99b38eaee878f0a0a9f03aa0258a2e3b62366353785
MD5 f4656028d3a6039cfb9ea9a1b5d27a9d
BLAKE2b-256 dbc7d3a9b538e92dc7d37df26b5629fe6775b13a5569d9904526f9de910a4616

See more details on using hashes here.

File details

Details for the file browserfetch-0.8.1-py3-none-any.whl.

File metadata

  • Download URL: browserfetch-0.8.1-py3-none-any.whl
  • Upload date:
  • Size: 19.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.12.4

File hashes

Hashes for browserfetch-0.8.1-py3-none-any.whl
Algorithm Hash digest
SHA256 7bbba2c439a118818524aaff48d7707df071735de8f510988f821a211aecb2a5
MD5 da32ffc9db6c6c7a52d8ac10fb2dc3fc
BLAKE2b-256 5ef9df0e9105552985fb874419d7c5caaa2a546999e2d57d65f9fe220b4b6681

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page