Skip to main content

A python requests wrapper for spoofing common browser behaviour

Project description

Spoofbot

Web bot for spoofing browser behaviour when using python requests. Supports Firefox and Chrome browser on most generic Windows, MacOS and Linux spoofing.

Example usage

browser = Chrome()
cache = CacheAdapter()
browser.session.mount('https://', cache)
browser.session.mount('http://', cache)

browser.navigate('https://httpbin.org/')
spec = browser.get('https://httpbin.org/spec.json').json()
print(spec['info']['description'])
# A simple HTTP Request & Response Service.<br/> <br/> <b>Run locally: </b> <code>$ docker run -p 80:80 kennethreitz/httpbin</code>
headers = browser.navigate(
    'https://httpbin.org/headers',
    headers={'Accept': 'mime/type'}
).json()
print(headers['headers']['Accept'])
# mime/type

Browsers

spoofbot allows for useragents to be generate using information like the platform and browser version. Currently, only Firefox and Chrome are supported, but through inheritance one can add more browsers. Firefox indicates that brotli encoding (br) is acceptable, but that might lead to issue swhen parsing the responses. It is possible to change that default header:

browser = Firefox()
browser._accept_encoding = ['deflate', 'gzip']  # brotli (br) is cumbersome

The browsers have an automatic request delay built in which can be controlled with the request_timeout and honor_timeout. After requests have been made, did_wait and waiting_period provide further information.

File Cache

Using FileCacheAdapter, one can store responses (without metadata such as headers) in the filesystem. It supports deleting the last request or a specific file from the cache. The cache indicates whether the last made request got a cache hit. If there is a request that should be cached that cannot be adequately stored with only hostname and path, one can specify and alternative url to use instead of the request's prior to the request. This last case is also supported when deleting the last request.

Har Cache

Using HarAdapter, one is able to load .har files to use as cache. This cache does not make actual http requests to the net, but fails if no matching request could be found. It can be specified whether matching a request should be strict (must match all headers) or not.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

spoofbot-0.1.3.tar.gz (13.3 kB view details)

Uploaded Source

Built Distribution

spoofbot-0.1.3-py3-none-any.whl (17.2 kB view details)

Uploaded Python 3

File details

Details for the file spoofbot-0.1.3.tar.gz.

File metadata

  • Download URL: spoofbot-0.1.3.tar.gz
  • Upload date:
  • Size: 13.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.41.1 CPython/3.8.1

File hashes

Hashes for spoofbot-0.1.3.tar.gz
Algorithm Hash digest
SHA256 d63762fc55912c6534108ee9c6def422e3858b22033013907d30320c8859c990
MD5 293905816d49a4207402f060a5629aa3
BLAKE2b-256 81b347883eca384cc860e044f70ceae2c60b73b8e31ceaac39930c34454b1a3f

See more details on using hashes here.

File details

Details for the file spoofbot-0.1.3-py3-none-any.whl.

File metadata

  • Download URL: spoofbot-0.1.3-py3-none-any.whl
  • Upload date:
  • Size: 17.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.41.1 CPython/3.8.1

File hashes

Hashes for spoofbot-0.1.3-py3-none-any.whl
Algorithm Hash digest
SHA256 57ca8fe86f105e32c0030ed690e7cb04203cb911b43353f9391c5ba144e31660
MD5 2505e186000fa40328b63a0e009a7d52
BLAKE2b-256 c3a8244b85601bb858a8149309f04061735cdcf7f390c83ee069a1d44fa48339

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page