Skip to main content

scrapenest

Official Python client for the ScrapeNest API — web scraping, search, AI answers, screenshots, PDFs, Google Maps reviews, gas prices, and audio transcription.

Install

pip install scrapenest

Requires Python 3.9+.

Quick start

from scrapenest import Client

client = Client("sn_your_key_here")

result = client.search("python web scraping", num_results=5)
for r in result.results:
    print(r.title, r.url)

Every method returns a typed dataclass. Pass include_usage=True (where supported) to get a usage block with the credits charged.

Options you leave out are not sent, so the API applies its own defaults.

# SERP results. depth = "fast" | "balanced" | "thorough" | "stealth"
hits = client.search("openai news", depth="balanced", num_results=10)

# Search + fetch & clean the top pages
deep = client.search_deep("what is rust lang", fetch_top=3)
for page in deep.pages:
    print(page.url, len(page.text or ""))

# Cited answer synthesized from live sources
answer = client.search_answer("who wrote the odyssey", max_sources=3)
print(answer.answer)
for c in answer.citations:
    print(c.index, c.url)

timeout_seconds caps how long search and search_deep wait for the engines. Search responses report how the engines did:

hits = client.search("rust async runtimes", timeout_seconds=3)
if hits.partial:
    print(hits.error, hits.note)
    for name, status in (hits.engine_status or {}).items():
        print(name, status.status, status.results)

Scrape a page

page = client.scrape_url(
    "https://example.com",
    render="auto",          # auto | direct | browser | stealth
    extract=True,
    return_markdown=True,
)
print(page.title)
print(page.markdown[:500])

# AI extraction
data = client.scrape_url("https://example.com/product", ai_query="Return the product name and price")
print(data.ai_extract)

Read a long page in windows with max_chars and offset, or pull out the passages that match focus:

url = "https://example.com/long-article"
part = client.scrape_url(url, max_chars=4000)
text = part.text or ""
while part.next_offset is not None:
    part = client.scrape_url(url, max_chars=4000, offset=part.next_offset)
    text += part.text or ""

passages = client.scrape_url(url, max_chars=4000, focus='"return policy"')

Newer API fields

Every response keeps the JSON it was built from in .raw, and every request method takes extra_body, merged into the request as-is, so fields the API adds before this SDK names them are still reachable:

res = client.search("query", extra_body={"some_new_option": True})
print(res.raw.get("some_new_field"))

Screenshots & PDF

shot = client.screenshot("https://example.com", full_page=True, format="png")
with open("page.png", "wb") as f:
    import base64
    f.write(base64.b64decode(shot.image_base64))

doc = client.pdf("https://example.com", format="A4", landscape=False)
with open("page.pdf", "wb") as f:
    import base64
    f.write(base64.b64decode(doc.pdf_base64))

Google Maps reviews

Pass the place's name and city, or any Google Maps link of it. If the name matches several places, the call fails free of charge with the error code place_ambiguous, and the message lists them with their place IDs. The place_id in a response skips the name lookup on later calls.

res = client.maps_reviews("Katz's Delicatessen, New York", max_reviews=50)
print(res.name, res.rating, res.review_count)
for rev in res.reviews:
    print(rev.author_name, rev.rating, rev.text)

# Large pulls run as async jobs
run = client.maps_reviews_async(res.place_id, max_reviews=2000)
final = client.wait_for_run(run.run_id)
print(f"{len(final.result['reviews'])} reviews fetched")
# Need to stop a long job early? client.cancel_run(run.run_id)

Gas prices

gas = client.gas_prices("Los Angeles, CA", grade="regular", limit=10)
for s in gas.stations:
    print(s.brand, s.address, s.price)
if gas.region_stats:
    print(gas.region_stats.region, gas.region_stats.average_price)

Translate & transcribe

tr = client.translate("Hello, how are you?", "es")
print(tr.translated)

audio = client.audio_transcript("https://www.youtube.com/watch?v=...")
print(audio.text)

Async client

import asyncio
from scrapenest import AsyncClient

async def main():
    async with AsyncClient("sn_your_key_here") as client:
        result = await client.search("python web scraping")
        print(result.results[0].title)

asyncio.run(main())

AsyncClient exposes the same methods as Client.

Error handling

from scrapenest import RateLimitError, AuthenticationError, PaymentRequiredError

try:
    result = client.search("test")
except RateLimitError as e:
    print(f"Rate limited. Retry after {e.retry_after}s")
except PaymentRequiredError:
    print("Out of credits")
except AuthenticationError:
    print("Invalid API key")

License

MIT

Metadata

Release files for scrapenest 0.5.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for scrapenest 0.5.2
File Size Uploaded Provenance Actions
scrapenest-0.5.2.tar.gz 14.7 kB Without provenance Details for scrapenest-0.5.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for scrapenest 0.5.2
File Interpreter ABI Platform Actions
scrapenest-0.5.2-py3-none-any.whl Python 3 none any Details for scrapenest-0.5.2-py3-none-any.whl

Total release size: 38.7 kB

Release files / scrapenest-0.5.2.tar.gz

Download URL scrapenest-0.5.2.tar.gz
Size 14.7 kB
Tags Source
SHA-256 checksum
How to use checksums
cd2768c480c25bf2a8e19de4988cbde9d71808165d3b90d26097d123f10ec27f
BLAKE2b-256 checksum
How to use checksums
cc2a925d028ab3f4c3412e116ea7df494b8d3e57a2b385e0f75f04bc223e61d6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.11.6

Release files / scrapenest-0.5.2-py3-none-any.whl

Download URL scrapenest-0.5.2-py3-none-any.whl
Size 24.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f21d7eca455a52a5f8fe396e56e8721be37c2f6c9dee097e4feb3b4f712ae14d
BLAKE2b-256 checksum
How to use checksums
795205e6d6d871bd459ba60a62299f5f6365b2514e7c942aaf313b5b1b870373
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.11.6

Security for scrapenest 0.5.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page