UnWeb Python SDK

Python SDK for the UnWeb API — convert HTML to clean, LLM-ready Markdown for RAG pipelines, AI agents, and documentation ingestion.

Installation

pip install unweb

Quick Start

from unweb import UnWebClient

client = UnWebClient(api_key="unweb_your_key_here")

# Convert HTML to Markdown
result = client.convert.paste("<h1>Hello World</h1><p>Clean markdown output.</p>")
print(result.markdown)       # "# Hello World\n\nClean markdown output."
print(result.quality_score)  # 100

# Convert a webpage
result = client.convert.url("https://example.com/article")
print(result.markdown)

# Upload an HTML file
result = client.convert.upload("page.html")
print(result.markdown)

Get your free API key at app.unweb.info (500 credits/month, no credit card required).

Features

• Conversions - Paste HTML, fetch URLs, or upload files. Returns clean CommonMark with quality scores.
• Web Crawler - Crawl entire documentation sites with BFS traversal. Export as raw Markdown, LangChain JSONL, or LlamaIndex JSON.
• Webhook Notifications - Get notified when crawl jobs complete via HTTPS webhooks.
• Dashboard Access - Manage API keys, view usage, and handle subscriptions programmatically.
• Quality Scores - Every conversion returns a 0-100 quality score detecting JS-rendered pages and content extraction issues.

API Reference

Conversions

All conversion methods return a ConversionResult with markdown, warnings, and quality_score.

from unweb import UnWebClient

client = UnWebClient(api_key="unweb_...")

# Paste raw HTML
result = client.convert.paste("<h1>Title</h1><p>Content</p>")
result.markdown       # "# Title\n\nContent"
result.quality_score  # 0-100
result.warnings       # ["Content auto-detected using: <main> element"]

# Convert from URL (fetches and converts server-side)
result = client.convert.url("https://docs.python.org/3/tutorial/index.html")

# Upload an HTML file
result = client.convert.upload("./downloaded-page.html")

Web Crawler

Crawl documentation sites and download results as a ZIP archive.

import time

# Start a crawl job
job = client.crawl.start(
    "https://docs.example.com",
    allowed_path="/docs/",      # Only crawl URLs under this path
    max_pages=100,               # Page limit
    export_format="raw-md",      # "raw-md", "langchain", or "llamaindex"
    webhook_url="https://your-app.com/hooks/crawl",  # Optional completion webhook
)
print(f"Job started: {job.job_id}")  # Job ID for polling

# Poll until complete
while not job.is_complete:
    time.sleep(5)
    job = client.crawl.status(job.job_id)
    print(f"  {job.status}: {job.pages_crawled} pages crawled")

# Download results
if job.status == "Completed":
    download = client.crawl.download(job.job_id)
    print(f"Download ZIP: {download.download_url}")
    print(f"Size: {download.size_bytes} bytes")

# List all your crawl jobs
jobs = client.crawl.list(status="Completed")
for j in jobs.jobs:
    print(f"  {j.job_id}: {j.pages_crawled} pages")

# Cancel a running job
client.crawl.cancel(job.job_id)

Export formats:

Authentication

The SDK uses API keys for conversion and crawler endpoints (set once in the constructor). For dashboard endpoints (usage, keys, subscription), authenticate with email/password to get a JWT:

# API key auth (conversions + crawler) - set in constructor
client = UnWebClient(api_key="unweb_...")

# JWT auth (dashboard endpoints) - login first
client.auth.login("you@example.com", "your-password")

# Now dashboard endpoints work
usage = client.usage.current()
keys = client.keys.list()

# Register a new account
token = client.auth.register("new@example.com", "password", "First", "Last")

# Get current user profile
profile = client.auth.me()
print(f"{profile.first_name} ({profile.email})")

# Update profile
client.auth.update_profile(first_name="NewName")

# Change password
client.auth.change_password("old-password", "new-password")

API Key Management

Requires JWT auth (client.auth.login(...) first).

# List API keys
keys = client.keys.list()
for key in keys:
    print(f"  {key.name}: {key.key_prefix}...")

# Create a new API key (full key only shown once)
new_key = client.keys.create("Production Key")
print(f"Key: {new_key.key}")  # Save this — not retrievable later

# Revoke an API key
client.keys.revoke(key_id="...")

Usage Tracking

Requires JWT auth.

usage = client.usage.current()
print(f"Credits used: {usage.credits_used}/{usage.credits_limit}")
print(f"Overage: {usage.overage_credits_used}")
print(f"Billing cycle: {usage.billing_cycle_start} - {usage.billing_cycle_end}")

# Detailed stats and history (returns raw dict)
stats = client.usage.stats()
history = client.usage.history()

Subscription

Requires JWT auth.

sub = client.subscription.get()
print(f"Tier: {sub.tier}")            # Free, Starter, Pro, Scale
print(f"Credits: {sub.credits_used}/{sub.monthly_credits}")
print(f"Overage: {sub.allows_overage}")

# Get a checkout URL to upgrade
url = client.subscription.checkout("Pro")
print(f"Upgrade: {url}")

# Cancel subscription
client.subscription.cancel()

Error Handling

The SDK raises typed exceptions for API errors:

from unweb import UnWebClient, UnWebError, AuthError, QuotaExceededError, ValidationError, NotFoundError

client = UnWebClient(api_key="unweb_...")

try:
    result = client.convert.paste("")
except ValidationError as e:
    print(f"Bad request: {e}")           # 400
except AuthError as e:
    print(f"Auth failed: {e}")           # 401/403
except QuotaExceededError as e:
    print(f"Quota exceeded: {e}")        # 429
except NotFoundError as e:
    print(f"Not found: {e}")             # 404
except UnWebError as e:
    print(f"API error ({e.status_code}): {e}")

# All exceptions have:
# e.status_code  - HTTP status code
# e.response     - Raw response body dict

Configuration

client = UnWebClient(
    api_key="unweb_...",                       # API key for conversions/crawler
    base_url="https://api.unweb.info",         # Default API URL
    timeout=30.0,                              # Request timeout in seconds
)

# Use as context manager for automatic cleanup
with UnWebClient(api_key="unweb_...") as client:
    result = client.convert.paste("<h1>Hello</h1>")

Pricing

Different operations cost different credits. Paid plans include overage billing so your pipelines never stop. See unweb.info for details.

Links

• UnWeb Homepage
• API Documentation
• Dashboard (get your API key)
• Report Issues

License

MIT - see LICENSE for details.