MrScraper Python SDK

An async Python client for MrScraper content fetching and Google SERP. Requires Python 3.9+.

Install and authenticate

python -m pip install -e .
export MRSCRAPER_API_TOKEN="YOUR_TOKEN"

These commands install the current checkout. After version 0.4.0 is published, install it from PyPI with python -m pip install --upgrade mrscraper-sdk. The client reads MRSCRAPER_API_TOKEN automatically. You can also pass token="YOUR_TOKEN" to MrScraper. Pass http_client= if you want to reuse your own httpx.AsyncClient; otherwise the SDK creates one per request. Both methods are async and must be awaited.

Fetch content

import asyncio
from mrscraper import MrScraper

async def main():
    client = MrScraper()
    result = await client.fetch_content("https://example.com")
    print(result["data"])

asyncio.run(main())

fetch_content sends a POST request to https://api.mrscraper.com. The JSON body contains the target URL, maxRetries, and optional tokenCap. Page options and the token are query parameters. The return value has status_code, data, and headers keys. The SDK parses JSON responses into Python values and returns other responses as text. Depending on the requested options, data may contain HTML, Markdown, a screenshot, or a combination; inspect the returned value before assuming it is an HTML string.

To request HTML, Markdown, and a screenshot together:

import asyncio
from mrscraper import MrScraper

async def main():
    client = MrScraper()
    result = await client.fetch_content(
        "https://example.com",
        html=True,
        markdown=True,
        screenshot=True,
        super=True,
    )
    print(result["data"])

asyncio.run(main())

Setting screenshot=True also enables browser rendering, even if browser_rendering=False is passed.

Google SERP

import asyncio
from mrscraper import MrScraper

async def main():
    client = MrScraper()
    result = await client.google_serp(
        "python web scraping",
        raw=False,
        region="us",
        language="en",
        page=1,
        format="json",
    )
    print(result["data"])

asyncio.run(main())

google_serp sends a POST request to https://sync.scraper.mrscraper.com/api/google/serp/v2/sync. The first argument accepts query text or a full Google search URL; for a URL, the client extracts q, gl, hl, and start. Explicit options override values from the URL. The return value has status_code, data, and headers keys. raw=True is the default and requests HTML; set raw=False to use format.

Smoke test this checkout

file.py runs one live request per invocation. Set your token first, then run:

python file.py fetch-content --url "https://example.com" --super
python file.py google-serp --query "site:example.com" --format json

The fetch-content command calls fetch_content() and enables screenshot=true and warmStart=true by default. Pass --super to enable real device routing; the printed request URL should contain super=true. The google-serp command calls google_serp() and requests JSON by default.

Use python file.py --help for the available options. Add --save-dir manual_results to save a full JSON response.

Publishing a release

The repository uses PyPI Trusted Publishing through .github/workflows/publish.yml. The workflow runs when a GitHub Release is published. It checks that the release tag is v followed by the version in pyproject.toml and src/mrscraper/__init__.py, runs unit tests, builds a wheel and source distribution, checks both with Twine, then uploads them to PyPI.

Set up the publisher once, using an account that can manage the existing mrscraper-sdk PyPI project:

1. In PyPI, open the project's Publishing settings and add a GitHub Actions Trusted Publisher with owner mrscraper-com, repository sdk-python, workflow filename publish.yml, and environment pypi.
2. In the GitHub repository, create a pypi environment under Settings → Environments and configure a required reviewer for the publish job.
3. Make sure the workflow file is present on the repository's default branch before publishing a release.

For each release, choose a version that has not been published to PyPI and set it in both pyproject.toml and src/mrscraper/__init__.py. This checkout is prepared for 0.4.0; PyPI already has 0.3.0. Check the package locally:

python -m pip install build twine pytest pytest-asyncio
python -m pip install -e .
python -m pytest -m "not integration" -q
python -m build
python -m twine check dist/mrscraper_sdk-0.4.0*

After committing and pushing the release changes to the default branch, create tag v0.4.0 on that commit, push the tag, and publish a GitHub Release using it. The workflow will upload the package after the pypi environment approves the publish job. No PyPI API token is needed in GitHub secrets.

Errors

MrScraperError is the base exception. AuthenticationError indicates a missing or invalid token, APIError carries an HTTP status_code, and NetworkError indicates a connection or timeout failure.

Compliance & Legal Risk

Scraping login-protected pages can carry legal and compliance risks. Check the target site's terms and applicable law before automating access.

License

MIT © MrScraper