evocrawl-py 4.22.1

Python SDK for Evocrawl API

EvoCrawl Python SDK

The EvoCrawl Python SDK is a library that allows you to easily scrape and crawl websites, and output the data in a format ready for use with language models (LLMs). It provides a simple and intuitive interface for interacting with the EvoCrawl API.

Installation

To install the EvoCrawl Python SDK, you can use pip:

pip install evocrawl-py

Usage

1. Get an API key from evocrawl.com
2. Set the API key as an environment variable named EVOCRAWL_API_KEY or pass it as a parameter to the EvoCrawl class.

Here's an example of how to use the SDK:

from evocrawl import EvoCrawl
from evocrawl.types import ScrapeOptions

evocrawl = EvoCrawl(api_key="fc-YOUR_API_KEY")

# Scrape a website (v2):
data = evocrawl.scrape(
  'https://evocrawl.com', 
  formats=['markdown', 'html']
)
print(data)

# Crawl a website (v2 waiter):
crawl_status = evocrawl.crawl(
  'https://evocrawl.com', 
  limit=100, 
  scrape_options=ScrapeOptions(formats=['markdown', 'html'])
)
print(crawl_status)

Scraping a URL

To scrape a single URL, use the scrape method. It takes the URL as a parameter and returns a document with the requested formats.

# Scrape a website (v2):
scrape_result = evocrawl.scrape('https://evocrawl.com', formats=['markdown', 'html'])
print(scrape_result)

Crawling a Website

To crawl a website, use the crawl method. It takes the starting URL and optional parameters as arguments. You can control depth, limits, formats, and more.

crawl_status = evocrawl.crawl(
  'https://evocrawl.com', 
  limit=100, 
  scrape_options=ScrapeOptions(formats=['markdown', 'html']),
  poll_interval=30
)
print(crawl_status)

Asynchronous Crawling

Looking for async operations? Check out the Async Class section below.

To enqueue a crawl asynchronously, use start_crawl. It returns the crawl ID which you can use to check the status of the crawl job.

crawl_job = evocrawl.start_crawl(
  'https://evocrawl.com', 
  limit=100, 
  scrape_options=ScrapeOptions(formats=['markdown', 'html']),
)
print(crawl_job)

Checking Crawl Status

To check the status of a crawl job, use the get_crawl_status method. It takes the job ID as a parameter and returns the current status of the crawl job.

crawl_status = evocrawl.get_crawl_status("<crawl_id>")
print(crawl_status)

Manual Pagination (v2)

Crawl and batch scrape status responses may include a next URL when more data is available. The SDK auto-paginates by default; to page manually, disable auto-pagination and pass the opaque next URL back to the SDK.

from evocrawl.v2.types import PaginationConfig

# Crawl: fetch one page at a time
crawl_job = evocrawl.start_crawl("https://evocrawl.com", limit=100)
status = evocrawl.get_crawl_status(
  crawl_job.id,
  pagination_config=PaginationConfig(auto_paginate=False),
)
if status.next:
  page2 = evocrawl.get_crawl_status_page(status.next)

# Batch scrape: fetch one page at a time
batch_job = evocrawl.start_batch_scrape(["https://evocrawl.com"])
status = evocrawl.get_batch_scrape_status(
  batch_job.id,
  pagination_config=PaginationConfig(auto_paginate=False),
)
if status.next:
  page2 = evocrawl.get_batch_scrape_status_page(status.next)

Cancelling a Crawl

To cancel an asynchronous crawl job, use the cancel_crawl method. It takes the job ID of the asynchronous crawl as a parameter and returns the cancellation status.

cancel_crawl = evocrawl.cancel_crawl(id)
print(cancel_crawl)

Map a Website

Use map to generate a list of URLs from a website. Options let you customize the mapping process, including whether to use the sitemap or include subdomains.

# Map a website (v2):
map_result = evocrawl.map('https://evocrawl.com')
print(map_result)

Scrape-bound interactive browsing (v2)

Use a scrape job ID to keep interacting with the replayed browser context:

doc = evocrawl.scrape(
  "https://example.com",
  actions=[{"type": "click", "selector": "a[href='/pricing']"}],
)

scrape_job_id = doc.metadata_typed.scrape_id
if not scrape_job_id:
  raise RuntimeError("Missing scrape job id")

run = evocrawl.interact(
  scrape_job_id,
  code="print(await page.url())",
  language="python",
  timeout=60,
)
print(run.stdout)

evocrawl.stop_interaction(scrape_job_id)

{/* ### Extracting Structured Data from Websites

To extract structured data from websites, use the extract method. It takes the URLs to extract data from, a prompt, and a schema as arguments. The schema is a Pydantic model that defines the structure of the extracted data.

*/}

Crawling a Website with WebSockets

To crawl a website with WebSockets, use the crawl_url_and_watch method. It takes the starting URL and optional parameters as arguments. The params argument allows you to specify additional options for the crawl job, such as the maximum number of pages to crawl, allowed domains, and the output format.

# inside an async function...
nest_asyncio.apply()

# Define event handlers
def on_document(detail):
    print("DOC", detail)

def on_error(detail):
    print("ERR", detail['error'])

def on_done(detail):
    print("DONE", detail['status'])

    # Function to start the crawl and watch process
async def start_crawl_and_watch():
    # Initiate the crawl job and get the watcher
    watcher = app.crawl_url_and_watch('evocrawl.com', exclude_paths=['blog/*'], limit=5)

    # Add event listeners
    watcher.add_event_listener("document", on_document)
    watcher.add_event_listener("error", on_error)
    watcher.add_event_listener("done", on_done)

    # Start the watcher
    await watcher.connect()

# Run the event loop
await start_crawl_and_watch()

Error Handling

The SDK handles errors returned by the EvoCrawl API and raises appropriate exceptions. If an error occurs during a request, an exception will be raised with a descriptive error message.

Async Class

For async operations, you can use the AsyncEvoCrawl class. Its methods mirror the EvoCrawl class, but you await them.

from evocrawl import AsyncEvoCrawl

evocrawl = AsyncEvoCrawl(api_key="YOUR_API_KEY")

# Async Scrape (v2)
async def example_scrape():
  scrape_result = await evocrawl.scrape(url="https://example.com")
  print(scrape_result)

# AsyncEvoCrawl (v2)
async def example_crawl():
  crawl_result = await evocrawl.crawl(url="https://example.com")
  print(crawl_result)

v1 compatibility

For legacy code paths, v1 remains available under evocrawl.v1 with the original method names.

from evocrawl import EvoCrawl

evocrawl = EvoCrawl(api_key="YOUR_API_KEY")

# v1 methods (feature‑frozen)
doc_v1 = evocrawl.v1.scrape_url('https://evocrawl.com', formats=['markdown', 'html'])
crawl_v1 = evocrawl.v1.crawl_url('https://evocrawl.com', limit=100)
map_v1 = evocrawl.v1.map_url('https://evocrawl.com')