zapfetch-py 0.1.0

Official Python SDK for the ZapFetch v2 web scraping API.

Zapfetch Python SDK

The Zapfetch Python SDK is a library that allows you to easily search, scrape, and interact with the web, and output the data in a format ready for use with language models (LLMs). It provides a simple and intuitive interface for the Zapfetch API.

Installation

To install the Zapfetch Python SDK, you can use pip:

pip install zapfetch-py

Usage

1. Get an API key from zapfetch.com
2. Set the API key as an environment variable named ZAPFETCH_API_KEY or pass it as a parameter to the Zapfetch class.

Here's an example of how to use the SDK:

from zapfetch import Zapfetch
from zapfetch.types import ScrapeOptions

zapfetch = Zapfetch(api_key="fc-YOUR_API_KEY")

# Scrape a website (v2):
data = zapfetch.scrape(
  'https://zapfetch.com', 
  formats=['markdown', 'html']
)
print(data)

# Crawl a website (v2 waiter):
crawl_status = zapfetch.crawl(
  'https://zapfetch.com', 
  limit=100, 
  scrape_options=ScrapeOptions(formats=['markdown', 'html'])
)
print(crawl_status)

Scraping a URL

To scrape a single URL, use the scrape method. It takes the URL as a parameter and returns a document with the requested formats.

# Scrape a website (v2):
scrape_result = zapfetch.scrape('https://zapfetch.com', formats=['markdown', 'html'])
print(scrape_result)

Crawling a Website

To crawl a website, use the crawl method. It takes the starting URL and optional parameters as arguments. You can control depth, limits, formats, and more.

crawl_status = zapfetch.crawl(
  'https://zapfetch.com', 
  limit=100, 
  scrape_options=ScrapeOptions(formats=['markdown', 'html']),
  poll_interval=30
)
print(crawl_status)

Asynchronous Crawling

Looking for async operations? Check out the Async Class section below.

To enqueue a crawl asynchronously, use start_crawl. It returns the crawl ID which you can use to check the status of the crawl job.

crawl_job = zapfetch.start_crawl(
  'https://zapfetch.com', 
  limit=100, 
  scrape_options=ScrapeOptions(formats=['markdown', 'html']),
)
print(crawl_job)

Checking Crawl Status

To check the status of a crawl job, use the get_crawl_status method. It takes the job ID as a parameter and returns the current status of the crawl job.

crawl_status = zapfetch.get_crawl_status("<crawl_id>")
print(crawl_status)

Manual Pagination (v2)

Crawl and batch scrape status responses may include a next URL when more data is available. The SDK auto-paginates by default; to page manually, disable auto-pagination and pass the opaque next URL back to the SDK.

from zapfetch.v2.types import PaginationConfig

# Crawl: fetch one page at a time
crawl_job = zapfetch.start_crawl("https://zapfetch.com", limit=100)
status = zapfetch.get_crawl_status(
  crawl_job.id,
  pagination_config=PaginationConfig(auto_paginate=False),
)
if status.next:
  page2 = zapfetch.get_crawl_status_page(status.next)

# Batch scrape: fetch one page at a time
batch_job = zapfetch.start_batch_scrape(["https://zapfetch.com"])
status = zapfetch.get_batch_scrape_status(
  batch_job.id,
  pagination_config=PaginationConfig(auto_paginate=False),
)
if status.next:
  page2 = zapfetch.get_batch_scrape_status_page(status.next)

Cancelling a Crawl

To cancel an asynchronous crawl job, use the cancel_crawl method. It takes the job ID of the asynchronous crawl as a parameter and returns the cancellation status.

cancel_crawl = zapfetch.cancel_crawl(id)
print(cancel_crawl)

Map a Website

Use map to generate a list of URLs from a website. Options let you customize the mapping process, including whether to use the sitemap or include subdomains.

# Map a website (v2):
map_result = zapfetch.map('https://zapfetch.com')
print(map_result)

Scrape-bound interactive browsing (v2)

Use a scrape job ID to keep interacting with the replayed browser context:

doc = zapfetch.scrape(
  "https://example.com",
  actions=[{"type": "click", "selector": "a[href='/pricing']"}],
)

scrape_job_id = doc.metadata_typed.scrape_id
if not scrape_job_id:
  raise RuntimeError("Missing scrape job id")

run = zapfetch.interact(
  scrape_job_id,
  code="print(await page.url())",
  language="python",
  timeout=60,
)
print(run.stdout)

zapfetch.stop_interaction(scrape_job_id)

{/* ### Extracting Structured Data from Websites

To extract structured data from websites, use the extract method. It takes the URLs to extract data from, a prompt, and a schema as arguments. The schema is a Pydantic model that defines the structure of the extracted data.

*/}

Crawling a Website with WebSockets

To crawl a website with WebSockets, use the crawl_url_and_watch method. It takes the starting URL and optional parameters as arguments. The params argument allows you to specify additional options for the crawl job, such as the maximum number of pages to crawl, allowed domains, and the output format.

# inside an async function...
nest_asyncio.apply()

# Define event handlers
def on_document(detail):
    print("DOC", detail)

def on_error(detail):
    print("ERR", detail['error'])

def on_done(detail):
    print("DONE", detail['status'])

    # Function to start the crawl and watch process
async def start_crawl_and_watch():
    # Initiate the crawl job and get the watcher
    watcher = app.crawl_url_and_watch('zapfetch.com', exclude_paths=['blog/*'], limit=5)

    # Add event listeners
    watcher.add_event_listener("document", on_document)
    watcher.add_event_listener("error", on_error)
    watcher.add_event_listener("done", on_done)

    # Start the watcher
    await watcher.connect()

# Run the event loop
await start_crawl_and_watch()

Error Handling

The SDK handles errors returned by the Zapfetch API and raises appropriate exceptions. If an error occurs during a request, an exception will be raised with a descriptive error message.

Async Class

For async operations, you can use the AsyncZapfetch class. Its methods mirror the Zapfetch class, but you await them.

from zapfetch import AsyncZapfetch

zapfetch = AsyncZapfetch(api_key="YOUR_API_KEY")

# Async Scrape (v2)
async def example_scrape():
  scrape_result = await zapfetch.scrape(url="https://example.com")
  print(scrape_result)

# Async Crawl (v2)
async def example_crawl():
  crawl_result = await zapfetch.crawl(url="https://example.com")
  print(crawl_result)

v1 compatibility

For legacy code paths, v1 remains available under zapfetch.v1 with the original method names.

from zapfetch import Zapfetch

zapfetch = Zapfetch(api_key="YOUR_API_KEY")

# v1 methods (feature‑frozen)
doc_v1 = zapfetch.v1.scrape_url('https://zapfetch.com', formats=['markdown', 'html'])
crawl_v1 = zapfetch.v1.crawl_url('https://zapfetch.com', limit=100)
map_v1 = zapfetch.v1.map_url('https://zapfetch.com')