hej-ai-crawler 0.1.0

Scrape any webpage into clean markdown

hej-ai-crawler

Scrape any webpage into clean markdown.

Install

pip install hej-ai-crawler

Usage

import asyncio
from hej_ai_crawler import crawler

scrape = crawler("your-api-key")

async def main():
    page = await scrape("https://example.com")
    print(page.title)    # "Example Domain"
    print(page.markdown) # clean markdown content
    print(page.links)    # internal links found on the page

asyncio.run(main())

Scraping modes

# Fast HTTP fetch + Readability extraction
await scrape("https://example.com", "static")

# Full headless browser render (default)
await scrape("https://example.com", "playwright")

# Tries static first, uses browser if content differs significantly
await scrape("https://example.com", "scout")

Response

ScrapedPage dataclass with fields:

• url — final URL after redirects
• title — page title
• markdown — page content as markdown
• links — internal links found
• mode — mode used to produce the result

License

MIT