Skip to main content

A pure-python headless browser

Project description

https://github.com/jelford/activesoup/workflows/Build/badge.svg https://img.shields.io/pypi/v/activesoup.svg?maxAge=3600

A simple library for interacting with the web from python

Description

activesoup combines familiar python web capabilities for convenient headless “browsing” functionality:

  • Modern HTTP support with requests - connection pooling, sessions, …

  • Convenient access to the web page with an interface inspired by beautifulsoup - convenient HTML navigation.

  • Robust HTML parsing with html5lib - parse the web like browsers do.

Full documentation can be found at https://activesoup.dev.

Use cases

activesoup aims to provide just enough functionality for basic web automation / crawler tasks. Consider using activesoup when:

  • You’ve already checked out requests-html

  • You need to actively interact with some web-page from Python (e.g. submitting forms, downloading files)

  • You don’t control the site you need to interact with (if you do, just make an API).

  • You don’t need javascript support (you’ll need selenium or phantomjs).

Usage examples

In the example below, we’ll load a page with a simple form, enumerate the fields, and make a submission:

>>> import activesoup

>>> # Start a session
>>> d = activesoup.Driver()

>>> page = d.get("https://httpbin.org/forms/post")

>>> # conveniently access elements, inspired by BeautifulSoup
>>> form = page.form

>>> # get the power of raw xpath search too
>>> form.find('.//input[@name="size"]')
BoundTag<input>
>>> # any element, searching by attribute
>>> form.find('.//*', name="size")
BoundTag<input>
>>> # or just search by attribute
>>> form.find(name="size")
BoundTag<input>

>>> # inspect element attributes
>>> print([i['name'] for i in form.find_all('input')])
['custname', 'custtel', 'custemail', 'size', 'size', 'size', 'topping', 'topping', 'topping', 'topping', 'delivery']

>>> # work actively with objects on the page
>>> r = form.submit({"custname": "john", "size": "small"})

>>> # responses parsed and ready based on content type
>>> r.keys()
dict_keys(['args', 'data', 'files', 'form', 'headers', 'json', 'origin', 'url'])
>>> r['form']
{'custname': 'john', 'size': 'small', 'topping': 'mushroom'}

>>> # access the underlying requests.Session too
>>> d.session
<requests.sessions.Session object at 0x7f283dc95700>

>>> # log in with cookie support
>>> d.get('https://httpbin.org/cookies/set/foo/bar')
>>> d.session.cookies['foo']
'bar'

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

activesoup-0.3.2.tar.gz (12.0 kB view details)

Uploaded Source

Built Distribution

activesoup-0.3.2-py3-none-any.whl (13.2 kB view details)

Uploaded Python 3

File details

Details for the file activesoup-0.3.2.tar.gz.

File metadata

  • Download URL: activesoup-0.3.2.tar.gz
  • Upload date:
  • Size: 12.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.9.13

File hashes

Hashes for activesoup-0.3.2.tar.gz
Algorithm Hash digest
SHA256 2799b21dbddd192f05779ad405d6263617e1a202b1dcda304c7f10d02f71002f
MD5 52fb66ede86003e3205e2097e99d0cbd
BLAKE2b-256 418e367cbd0395da97fc44cced39c4fd7d5fa5a87eb85a0228262678dfdfafc4

See more details on using hashes here.

File details

Details for the file activesoup-0.3.2-py3-none-any.whl.

File metadata

  • Download URL: activesoup-0.3.2-py3-none-any.whl
  • Upload date:
  • Size: 13.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.9.13

File hashes

Hashes for activesoup-0.3.2-py3-none-any.whl
Algorithm Hash digest
SHA256 87eafc775f83adbab57782b19a44aa620565b0bd641e4cf5981e6d9cc53e964d
MD5 555340c900988b65bfba2c6db06aac77
BLAKE2b-256 ff5c0e352eb7fb7f84a8f3e6c1da0430d93950208adb2ea711389d9508407090

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page