MrScraper Python SDK

An async Python client for MrScraper content fetching and Google SERP. Requires Python 3.9+.

Install and authenticate

python -m pip install -e .
export MRSCRAPER_API_TOKEN="YOUR_TOKEN"

These commands install the current checkout. To install version 0.5.0 from PyPI, run python -m pip install --upgrade mrscraper-sdk==0.5.0. The client reads MRSCRAPER_API_TOKEN automatically. You can also pass token="YOUR_TOKEN" to MrScraper. Pass http_client= if you want to reuse your own httpx.AsyncClient; otherwise the SDK creates one per request. Both methods are async and must be awaited.

Migrating to 0.5.0

Version 0.5.0 renames two public methods. Update existing calls as follows:

The previous method names have been removed. Arguments and response formats are unchanged from 0.4.0.

Scraper API

import asyncio
from mrscraper import MrScraper

async def main():
    client = MrScraper()
    result = await client.scraper_api("https://example.com")
    print(result["data"])

asyncio.run(main())

scraper_api sends a POST request to https://api.mrscraper.com. The JSON body contains the target URL, maxRetries, and optional tokenCap. Page options and the token are query parameters. The return value has status_code, data, and headers keys. The SDK parses JSON responses into Python values and returns other responses as text. Depending on the requested options, data may contain HTML, Markdown, a screenshot, or a combination; inspect the returned value before assuming it is an HTML string.

To request HTML, Markdown, and a screenshot together:

import asyncio
from mrscraper import MrScraper

async def main():
    client = MrScraper()
    result = await client.scraper_api(
        "https://example.com",
        html=True,
        markdown=True,
        screenshot=True,
        super=True,
    )
    print(result["data"])

asyncio.run(main())

Setting screenshot=True also enables browser rendering, even if browser_rendering=False is passed.

Web search

import asyncio
from mrscraper import MrScraper

async def main():
    client = MrScraper()
    result = await client.web_search(
        "python web scraping",
        raw=False,
        region="us",
        language="en",
        page=1,
        format="json",
    )
    print(result["data"])

asyncio.run(main())

web_search sends a POST request to https://sync.scraper.mrscraper.com/api/google/serp/v2/sync. The first argument accepts query text or a full Google search URL; for a URL, the client extracts q, gl, hl, and start. Explicit options override values from the URL. The return value has status_code, data, and headers keys. raw=True is the default and requests HTML; set raw=False to use format.

Smoke test this checkout

file.py runs one live request per invocation. Set your token first, then run:

python file.py fetch-content --url "https://example.com" --super
python file.py google-serp --query "site:example.com" --format json

The fetch-content command calls scraper_api() and enables screenshot=true and warmStart=true by default. Pass --super to enable real device routing; the printed request URL should contain super=true. The google-serp command calls web_search() and requests JSON by default.

Use python file.py --help for the available options. Add --save-dir manual_results to save a full JSON response.

Publishing a release

Publishing is manual. Choose a version that has not been published to PyPI and set it in both pyproject.toml and src/mrscraper/__init__.py. This checkout is prepared for 0.5.0. From the repository root, run:

python -m pip install build twine pytest pytest-asyncio
python -m pip install -e .
python -m pytest -m "not integration" -q
python -m build --outdir dist/manual-0.5.0
ls dist/manual-0.5.0
python -m twine check dist/manual-0.5.0/mrscraper_sdk-0.5.0*
python -m twine upload --username __token__ dist/manual-0.5.0/mrscraper_sdk-0.5.0*

Before uploading, confirm that ls shows only the intended 0.5.0 files. Twine prompts for a PyPI API token as the password. Do not upload older files from dist/.

Errors

MrScraperError is the base exception. AuthenticationError indicates a missing or invalid token, APIError carries an HTTP status_code, and NetworkError indicates a connection or timeout failure.

Compliance & Legal Risk

Scraping login-protected pages can carry legal and compliance risks. Check the target site's terms and applicable law before automating access.

License

MIT © MrScraper