scrapenest
Official Python client for the ScrapeNest API — web scraping, search, AI answers, screenshots, PDFs, Google Maps reviews, gas prices, and audio transcription.
- Docs: https://scrapenest.dev/docs
- Get an API key: https://scrapenest.dev
Install
pip install scrapenest
Requires Python 3.9+.
Quick start
from scrapenest import Client
client = Client("sn_your_key_here")
result = client.search("python web scraping", num_results=5)
for r in result.results:
print(r.title, r.url)
Every method returns a typed dataclass. Pass include_usage=True (where supported) to get a usage block with the credits charged.
Options you leave out are not sent, so the API applies its own defaults.
Search
# SERP results. depth = "fast" | "balanced" | "thorough" | "stealth"
hits = client.search("openai news", depth="balanced", num_results=10)
# Search + fetch & clean the top pages
deep = client.search_deep("what is rust lang", fetch_top=3)
for page in deep.pages:
print(page.url, len(page.text or ""))
# Cited answer synthesized from live sources
answer = client.search_answer("who wrote the odyssey", max_sources=3)
print(answer.answer)
for c in answer.citations:
print(c.index, c.url)
timeout_seconds caps how long search and search_deep wait for the engines. Search responses report how the engines did:
hits = client.search("rust async runtimes", timeout_seconds=3)
if hits.partial:
print(hits.error, hits.note)
for name, status in (hits.engine_status or {}).items():
print(name, status.status, status.results)
Scrape a page
page = client.scrape_url(
"https://example.com",
render="auto", # auto | direct | browser | stealth
extract=True,
return_markdown=True,
)
print(page.title)
print(page.markdown[:500])
# AI extraction
data = client.scrape_url("https://example.com/product", ai_query="Return the product name and price")
print(data.ai_extract)
Read a long page in windows with max_chars and offset, or pull out the passages that match focus:
url = "https://example.com/long-article"
part = client.scrape_url(url, max_chars=4000)
text = part.text or ""
while part.next_offset is not None:
part = client.scrape_url(url, max_chars=4000, offset=part.next_offset)
text += part.text or ""
passages = client.scrape_url(url, max_chars=4000, focus='"return policy"')
Newer API fields
Every response keeps the JSON it was built from in .raw, and every request method takes extra_body, merged into the request as-is, so fields the API adds before this SDK names them are still reachable:
res = client.search("query", extra_body={"some_new_option": True})
print(res.raw.get("some_new_field"))
Screenshots & PDF
shot = client.screenshot("https://example.com", full_page=True, format="png")
with open("page.png", "wb") as f:
import base64
f.write(base64.b64decode(shot.image_base64))
doc = client.pdf("https://example.com", format="A4", landscape=False)
with open("page.pdf", "wb") as f:
import base64
f.write(base64.b64decode(doc.pdf_base64))
Google Maps reviews
Pass a place URL or a 0x…:0x… FID. Use place_url to bypass FID-built URLs for tricky listings.
res = client.maps_reviews("https://www.google.com/maps/place/...", max_reviews=50, sort="newest")
print(res.name, res.rating, res.review_count)
for rev in res.reviews:
print(rev.author_name, rev.rating, rev.text)
# Large pulls run as async jobs
run = client.maps_reviews_async("0x6b12ae665e892fdd:0x3133f8d75a1ac251", max_reviews=2000)
final = client.wait_for_run(run.run_id)
print(f"{len(final.result['reviews'])} reviews fetched")
# Need to stop a long job early? client.cancel_run(run.run_id)
Gas prices
gas = client.gas_prices("Los Angeles, CA", grade="regular", limit=10)
for s in gas.stations:
print(s.brand, s.address, s.price)
if gas.region_stats:
print(gas.region_stats.region, gas.region_stats.average_price)
Translate & transcribe
tr = client.translate("Hello, how are you?", "es")
print(tr.translated)
audio = client.audio_transcript("https://www.youtube.com/watch?v=...")
print(audio.text)
Async client
import asyncio
from scrapenest import AsyncClient
async def main():
async with AsyncClient("sn_your_key_here") as client:
result = await client.search("python web scraping")
print(result.results[0].title)
asyncio.run(main())
AsyncClient exposes the same methods as Client.
Error handling
from scrapenest import RateLimitError, AuthenticationError, PaymentRequiredError
try:
result = client.search("test")
except RateLimitError as e:
print(f"Rate limited. Retry after {e.retry_after}s")
except PaymentRequiredError:
print("Out of credits")
except AuthenticationError:
print("Invalid API key")
License
MIT
Metadata
Release files for scrapenest 0.5.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| scrapenest-0.5.1.tar.gz | 15.2 kB | Without provenance | Details for scrapenest-0.5.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| scrapenest-0.5.1-py3-none-any.whl | Python 3 | none | any | Details for scrapenest-0.5.1-py3-none-any.whl |
Total release size: 39.7 kB
Release files / scrapenest-0.5.1.tar.gz
| Download URL | scrapenest-0.5.1.tar.gz |
|---|---|
| Size | 15.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ac8110b4320aae6c059f57f4e87d2ea6b76a032451df37581febafcac1481d6d
|
|
BLAKE2b-256 checksum How to use checksums |
9cfa13ffa5e1bcddf4c5ae183b5fde26b20641a8d61f12a182e0d889ed87f96e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.6
|
Release files / scrapenest-0.5.1-py3-none-any.whl
| Download URL | scrapenest-0.5.1-py3-none-any.whl |
|---|---|
| Size | 24.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
c121af3f9c1d124421c48b78c8dec59be15c3ed90d0737ca5d00042367625cbb
|
|
BLAKE2b-256 checksum How to use checksums |
63019d97e7a5f5c796fb0a1bf930544429a791ec9e5c6c42aeba4a0e2a581542
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.6
|
Security for scrapenest 0.5.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.