Skip to main content

scrapenest

Official Python client for the ScrapeNest API — web scraping, search, AI answers, screenshots, PDFs, Google Maps reviews, gas prices, and audio transcription.

Install

pip install scrapenest

Requires Python 3.9+.

Quick start

from scrapenest import Client

client = Client("sn_your_key_here")

result = client.search("python web scraping", num_results=5)
for r in result.results:
    print(r.title, r.url)

Every method returns a typed dataclass. Pass include_usage=True (where supported) to get a usage block with the credits charged.

Options you leave out are not sent, so the API applies its own defaults.

# SERP results. depth = "fast" | "balanced" | "thorough" | "stealth"
hits = client.search("openai news", depth="balanced", num_results=10)

# Search + fetch & clean the top pages
deep = client.search_deep("what is rust lang", fetch_top=3)
for page in deep.pages:
    print(page.url, len(page.text or ""))

# Cited answer synthesized from live sources
answer = client.search_answer("who wrote the odyssey", max_sources=3)
print(answer.answer)
for c in answer.citations:
    print(c.index, c.url)

timeout_seconds caps how long search and search_deep wait for the engines. Search responses report how the engines did:

hits = client.search("rust async runtimes", timeout_seconds=3)
if hits.partial:
    print(hits.error, hits.note)
    for name, status in (hits.engine_status or {}).items():
        print(name, status.status, status.results)

Scrape a page

page = client.scrape_url(
    "https://example.com",
    render="auto",          # auto | direct | browser | stealth
    extract=True,
    return_markdown=True,
)
print(page.title)
print(page.markdown[:500])

# AI extraction
data = client.scrape_url("https://example.com/product", ai_query="Return the product name and price")
print(data.ai_extract)

Read a long page in windows with max_chars and offset, or pull out the passages that match focus:

url = "https://example.com/long-article"
part = client.scrape_url(url, max_chars=4000)
text = part.text or ""
while part.next_offset is not None:
    part = client.scrape_url(url, max_chars=4000, offset=part.next_offset)
    text += part.text or ""

passages = client.scrape_url(url, max_chars=4000, focus='"return policy"')

Newer API fields

Every response keeps the JSON it was built from in .raw, and every request method takes extra_body, merged into the request as-is, so fields the API adds before this SDK names them are still reachable:

res = client.search("query", extra_body={"some_new_option": True})
print(res.raw.get("some_new_field"))

Screenshots & PDF

shot = client.screenshot("https://example.com", full_page=True, format="png")
with open("page.png", "wb") as f:
    import base64
    f.write(base64.b64decode(shot.image_base64))

doc = client.pdf("https://example.com", format="A4", landscape=False)
with open("page.pdf", "wb") as f:
    import base64
    f.write(base64.b64decode(doc.pdf_base64))

Google Maps reviews

Pass a place URL or a 0x…:0x… FID. Use place_url to bypass FID-built URLs for tricky listings.

res = client.maps_reviews("https://www.google.com/maps/place/...", max_reviews=50, sort="newest")
print(res.name, res.rating, res.review_count)
for rev in res.reviews:
    print(rev.author_name, rev.rating, rev.text)

# Large pulls run as async jobs
run = client.maps_reviews_async("0x6b12ae665e892fdd:0x3133f8d75a1ac251", max_reviews=2000)
final = client.wait_for_run(run.run_id)
print(f"{len(final.result['reviews'])} reviews fetched")
# Need to stop a long job early? client.cancel_run(run.run_id)

Gas prices

gas = client.gas_prices("Los Angeles, CA", grade="regular", limit=10)
for s in gas.stations:
    print(s.brand, s.address, s.price)
if gas.region_stats:
    print(gas.region_stats.region, gas.region_stats.average_price)

Translate & transcribe

tr = client.translate("Hello, how are you?", "es")
print(tr.translated)

audio = client.audio_transcript("https://www.youtube.com/watch?v=...")
print(audio.text)

Async client

import asyncio
from scrapenest import AsyncClient

async def main():
    async with AsyncClient("sn_your_key_here") as client:
        result = await client.search("python web scraping")
        print(result.results[0].title)

asyncio.run(main())

AsyncClient exposes the same methods as Client.

Error handling

from scrapenest import RateLimitError, AuthenticationError, PaymentRequiredError

try:
    result = client.search("test")
except RateLimitError as e:
    print(f"Rate limited. Retry after {e.retry_after}s")
except PaymentRequiredError:
    print("Out of credits")
except AuthenticationError:
    print("Invalid API key")

License

MIT

Metadata

Release files for scrapenest 0.5.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for scrapenest 0.5.1
File Size Uploaded Provenance Actions
scrapenest-0.5.1.tar.gz 15.2 kB Without provenance Details for scrapenest-0.5.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for scrapenest 0.5.1
File Interpreter ABI Platform Actions
scrapenest-0.5.1-py3-none-any.whl Python 3 none any Details for scrapenest-0.5.1-py3-none-any.whl

Total release size: 39.7 kB

Release files / scrapenest-0.5.1.tar.gz

Download URL scrapenest-0.5.1.tar.gz
Size 15.2 kB
Tags Source
SHA-256 checksum
How to use checksums
ac8110b4320aae6c059f57f4e87d2ea6b76a032451df37581febafcac1481d6d
BLAKE2b-256 checksum
How to use checksums
9cfa13ffa5e1bcddf4c5ae183b5fde26b20641a8d61f12a182e0d889ed87f96e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.11.6

Release files / scrapenest-0.5.1-py3-none-any.whl

Download URL scrapenest-0.5.1-py3-none-any.whl
Size 24.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
c121af3f9c1d124421c48b78c8dec59be15c3ed90d0737ca5d00042367625cbb
BLAKE2b-256 checksum
How to use checksums
63019d97e7a5f5c796fb0a1bf930544429a791ec9e5c6c42aeba4a0e2a581542
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.11.6

Security for scrapenest 0.5.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page