A pure-python headless browser
Project description
A simple library for interacting with the web from python
Description
activesoup combines familiar python web capabilities for convenient headless “browsing” functionality:
Modern HTTP support with requests - connection pooling, sessions, …
Convenient access to the web page with an interface inspired by beautifulsoup - convenient HTML navigation.
Robust HTML parsing with html5lib - parse the web like browsers do.
Full documentation can be found at https://activesoup.dev.
Use cases
activesoup aims to provide just enough functionality for basic web automation / crawler tasks. Consider using activesoup when:
You’ve already checked out requests-html
You need to actively interact with some web-page from Python (e.g. submitting forms, downloading files)
You don’t control the site you need to interact with (if you do, just make an API).
You don’t need javascript support (you’ll need selenium or phantomjs).
Usage examples
In the example below, we’ll load a page with a simple form, enumerate the fields, and make a submission:
>>> import activesoup
>>> # Start a session
>>> d = activesoup.Driver()
>>> page = d.get("https://httpbin.org/forms/post")
>>> # conveniently access elements, inspired by BeautifulSoup
>>> form = page.form
>>> # get the power of raw xpath search too
>>> form.find('.//input[@name="size"]')
BoundTag<input>
>>> # any element, searching by attribute
>>> form.find('.//*', name="size")
BoundTag<input>
>>> # or just search by attribute
>>> form.find(name="size")
BoundTag<input>
>>> # inspect element attributes
>>> print([i['name'] for i in form.find_all('input')])
['custname', 'custtel', 'custemail', 'size', 'size', 'size', 'topping', 'topping', 'topping', 'topping', 'delivery']
>>> # work actively with objects on the page
>>> r = form.submit({"custname": "john", "size": "small"})
>>> # responses parsed and ready based on content type
>>> r.keys()
dict_keys(['args', 'data', 'files', 'form', 'headers', 'json', 'origin', 'url'])
>>> r['form']
{'custname': 'john', 'size': 'small', 'topping': 'mushroom'}
>>> # access the underlying requests.Session too
>>> d.session
<requests.sessions.Session object at 0x7f283dc95700>
>>> # log in with cookie support
>>> d.get('https://httpbin.org/cookies/set/foo/bar')
>>> d.session.cookies['foo']
'bar'
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file activesoup-0.3.2.tar.gz
.
File metadata
- Download URL: activesoup-0.3.2.tar.gz
- Upload date:
- Size: 12.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.9.13
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2799b21dbddd192f05779ad405d6263617e1a202b1dcda304c7f10d02f71002f |
|
MD5 | 52fb66ede86003e3205e2097e99d0cbd |
|
BLAKE2b-256 | 418e367cbd0395da97fc44cced39c4fd7d5fa5a87eb85a0228262678dfdfafc4 |
Provenance
File details
Details for the file activesoup-0.3.2-py3-none-any.whl
.
File metadata
- Download URL: activesoup-0.3.2-py3-none-any.whl
- Upload date:
- Size: 13.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.9.13
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 87eafc775f83adbab57782b19a44aa620565b0bd641e4cf5981e6d9cc53e964d |
|
MD5 | 555340c900988b65bfba2c6db06aac77 |
|
BLAKE2b-256 | ff5c0e352eb7fb7f84a8f3e6c1da0430d93950208adb2ea711389d9508407090 |