Home page
Overview
A Python library for automating interaction with websites. MechanicalSoup automatically stores and sends cookies, follows redirects, and can follow links and submit forms. It doesn’t do JavaScript.
MechanicalSoup was created by M Hickford, who was a fond user of the Mechanize library. Unfortunately, Mechanize was incompatible with Python 3 until 2019 and its development stalled for several years. MechanicalSoup provides a similar API, built on Python giants Requests (for HTTP sessions) and BeautifulSoup (for document navigation). Since 2017 it is a project actively maintained by a small team including @hemberger and @moy.
Installation
PyPy3 is also supported (and tested against).
Download and install the latest released version from PyPI:
pip install MechanicalSoup
Download and install the development version from GitHub:
pip install git+https://github.com/MechanicalSoup/MechanicalSoup
Installing from source (installs the version in the current working directory):
python setup.py install
(In all cases, add --user to the install command to install in the current user’s home directory.)
Documentation
The full documentation is available on https://mechanicalsoup.readthedocs.io/. You may want to jump directly to the automatically generated API documentation.
Example
From examples/expl_qwant.py, code to get the results from a Qwant search:
"""Example usage of MechanicalSoup to get the results from the Qwant
search engine.
"""
import re
import mechanicalsoup
import html
import urllib.parse
# Connect to duckduckgo
browser = mechanicalsoup.StatefulBrowser(user_agent='MechanicalSoup')
browser.open("https://lite.qwant.com/")
# Fill-in the search form
browser.select_form('#search-form')
browser["q"] = "MechanicalSoup"
browser.submit_selected()
# Display the results
for link in browser.page.select('.result a'):
# Qwant shows redirection links, not the actual URL, so extract
# the actual URL from the redirect link:
href = link.attrs['href']
m = re.match(r"^/redirect/[^/]*/(.*)$", href)
if m:
href = urllib.parse.unquote(m.group(1))
print(link.text, '->', href)More examples are available in examples/.
For an example with a more complex form (checkboxes, radio buttons and textareas), read tests/test_browser.py and tests/test_form.py.
Development
Instructions for building, testing and contributing to MechanicalSoup: see CONTRIBUTING.rst.
Common problems
Read the FAQ.
Release files for MechanicalSoup 1.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| MechanicalSoup-1.1.0.tar.gz | 49.3 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| MechanicalSoup-1.1.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 68.5 kB
Release files / MechanicalSoup-1.1.0.tar.gz
| Download URL | MechanicalSoup-1.1.0.tar.gz |
|---|---|
| Size | 49.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2be8ad9b7571990fce16d0f99a741779921510eb133d3a4dfdeccb2ff2cd00e5
|
|
BLAKE2b-256 checksum How to use checksums |
9f39ff8223982443db261ec8cf1ed9ba133fac3e50f7717b0e291ba781916441
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/4.3.1 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.61.0 CPython/3.9.5
|
Release files / MechanicalSoup-1.1.0-py3-none-any.whl
| Download URL | MechanicalSoup-1.1.0-py3-none-any.whl |
|---|---|
| Size | 19.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
0685465c449018f8bce9055b74cf60fbec2ada8391c03aa179586edc6b5a5ee8
|
|
BLAKE2b-256 checksum How to use checksums |
dbae244f73ee13f999811069fb4ef4b19b1ebf6b41b3ea5f1046645e51d949dc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/4.3.1 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.61.0 CPython/3.9.5
|
