product-dl / product-ui / list-dl

This repository contains three packages:

• product-dl — "youtube-dl but for product pages". Given a URL it detects the right provider, scrapes the product page, and returns a normalised ParsedProduct as JSON. Exposes a CLI and a FastAPI JSON API (default port 7001).
• product-ui — UI companion. Calls the product-dl API, persists crawls to disk, and serves a Bootstrap web interface for browsing and triggering crawls (default port 7002).
• list-dl — Scrapes order histories and wishlists on demand. Like product-dl but for lists (Amazon orders, public wishlists, etc.). Returns structured Order objects. Exposes a CLI and a FastAPI JSON API (default port 7003).

---

product-dl

Supported sites

Provider	Domains	ID format
kleinanzeigen	kleinanzeigen.de	/s-anzeige/…/<id>
ebay	ebay.de, ebay.com, ebay.co.uk, …	/itm/…/<id>
amazon	amazon.de, amazon.com, amazon.co.uk, …	/dp/<ASIN>, /gp/product/<ASIN>
aliexpress	aliexpress.com, aliexpress.us	/item/….html
thomann	thomann.de, thomannmusic.com	/<slug>.htm

Install

pip install product-dl        # or: uv add product-dl

CLI

# Fetch a product page and print JSON
product-dl fetch <url>

# Extra options
product-dl fetch <url> \
  --headers '{"Accept-Language": "de-DE"}' \
  --cookies path/to/cookies.json \
  --data-dir ./data \
  --save-example my-label \
  --indent 4

# Parse a locally saved HTML file
product-dl parse amazon path/to/product.html --url https://www.amazon.de/dp/B08N5WRWNW

# Start the API server
product-dl serve --host 0.0.0.0 --port 7001

# Export the OpenAPI schema (default: YAML)
product-dl openapi
product-dl openapi --format json --output ./openapi.json

# List discovered providers
product-dl providers
product-dl providers --format json

--data-dir / PRODUCT_DL_DATA_DIR

When set, the CLI (and API server via env var) will:

• Auto-load <data-dir>/cookies/<provider>.json and <data-dir>/headers/<provider>.json before each request.
• Detect Akamai CDN bot-detection and switch to httpx-curl-cffi (Chrome TLS impersonation) automatically:
  • If any Akamai cookie key (ak_bmsc, _abck, bm_sz, bm_sv, RT) is present in the merged cookies.
  • Or if a 503 response body contains Akamai signatures (errors.edgesuite.net / Zero size object).
• Save Akamai cookies received in curl_cffi responses back to <data-dir>/cookies/<provider>.json for reuse.
• Save reusable Shein CAPTCHA session material after a solved remote challenge:
  • <data-dir>/cookies/shein.json
  • <data-dir>/headers/shein.json
  • <data-dir>/sessions/shein.playwright.json
  • <data-dir>/sessions/shein.json

# CLI
product-dl fetch https://www.ebay.de/itm/123 --data-dir ./data

# env var (also used by the API server)
export PRODUCT_DL_DATA_DIR=./data
product-dl fetch https://www.ebay.de/itm/123

API

product-dl serve          # http://127.0.0.1:7001
# or
uvicorn product_dl.api.app:app

GET  /fetch?url=<url>
POST /fetch  {"url": "<url>"}
POST /fetch  {"url": "<url>", "session": {"provider": "shein", ...}}

Both return a ParsedProduct JSON object. PRODUCT_DL_DATA_DIR is read server-side from the environment — it is not an API parameter.

For Shein URLs, the API may return 202 Accepted with {"challenge_id": "..."} when a remote CAPTCHA challenge must be solved. Poll /challenge/{challenge_id}/status; once solved, the payload contains result and may contain a reusable session object:

{
  "status": "solved",
  "result": {"parser": "shein"},
  "session": {
    "provider": "shein",
    "version": 1,
    "cookies": {"...": "..."},
    "headers": {"User-Agent": "..."},
    "storage_state": {"cookies": [], "origins": []}
  }
}

If PRODUCT_DL_DATA_DIR is configured, that session is persisted automatically for later crawls. API clients can also pass the returned session back in future POST /fetch requests; its cookies and headers are applied to that crawl and persisted for Shein reuse.

Latest generated API descriptions are published to the repository gh-pages branch as:

• product_dl/openapi.json
• product_dl/openapi.yml
• product_dl/providers.json
• list_dl/openapi.json
• list_dl/openapi.yml
• list_dl/providers.json
• list_dl/login_providers.json
• catalog/commit_map.json

Output schema (ParsedProduct)

ParsedProduct
├── id: str | int | None
├── title: str
├── description: str | None
├── gallery: list[Image | Video | PDF] | None
│     Image / Video ── url, mime, thumb, width, height
│     Video         ── length (seconds / frames)
│     PDF           ── url, name, page_count
├── pricing: Pricing | None
│     ├── price: Price | None        (value, unit, raw, symbol, suffix)
│     ├── shipping: list[Shipping]   (name, price)
│     ├── original: Price | None     (pre-discount list price)
│     └── reduction: list[Reduction] (percentage or price discount)
├── canonical_url: str | None
├── fetch_url: str
├── parser: str                      e.g. "kleinanzeigen", "amazon"
├── website_variant: str | None       e.g. "de", "co.uk", "gb"
├── version: int                     schema version (starts at 1)
├── available: bool | None
├── seller: Seller | None            (name, id, url, picture)
├── brand: Brand | None              (name, url, logo)
├── category: Category | None        (name, id, url, parent)
├── variation_types: list[VariationType] | None
├── tables: dict[str, dict[str, str]] | None
├── meta: PageMeta | None            (meta tags, LD+JSON, tracking)
└── extra: <ProviderExtra>           provider-specific fields

Adding a provider

Built-in — create src/product_dl/providers/<name>/__init__.py exposing:

PROVIDER_NAME: str
def can_provide(url: str, parsed_url: ParseResult) -> T | None: ...
async def provide(
    url: str,
    parsed_url: ParseResult,
    provide_check: T,
    *,
    save_files: SaveFiles | None = None,
    local_html: str | None = None,
    extra_headers: dict[str, str] | None = None,
    extra_cookies: dict[str, str] | None = None,
    data_dir: Path | None = None,
) -> ParsedProduct: ...

Auto-discovered via pkgutil — no pyproject.toml changes needed. Use product_dl.utils.http.fetch_html for HTTP requests so Akamai handling and cookie persistence work automatically.

Third-party — publish a package with a product_dl.providers entry point. No changes to this project needed.

---

product-ui

Install

pip install product-dl        # product-ui is bundled in the same package

Quick start

# 1. Start the product-dl API backend
product-dl serve

# 2. Start the UI (in another terminal)
product-ui serve

# 3. Open http://127.0.0.1:7002

CLI

# Start the web UI
product-ui serve [--host 127.0.0.1] [--port 7002] [--reload] \
                 [--data-dir ./crawl_data/product_dl] [--product-dl-api http://localhost:7001]

# Fetch a URL and save to disk (product-dl must be running)
product-ui fetch <url> [--data-dir ./crawl_data/product_dl] [--product-dl-api http://localhost:7001]

Environment variables

Variable	Default	Description
PRODUCT_UI_DATA_DIR	./crawl_data/product_dl	Directory where crawls are stored
PRODUCT_DL_API	http://localhost:7001	Base URL of the product-dl JSON API

Disk layout

{data_dir}/{provider}/{YYYY}/{MM}/{DD}/{HH_MM_SS}/{id}/
    crawl.json                  ← {"information": {url, provider, variant, time}, "crawl": <ParsedProduct>}
    files/full/{nnn}.{ext}      ← base64 gallery files decoded to disk
    files/thumb/{nnn}.{ext}     ← thumbnails (same index number as matching full)
{data_dir}/index.jsonl          ← append-only JSON-Lines: {provider, url, id, title, date, path}

Web UI routes

Route	Description
GET /	Filterable list of crawls; includes new-crawl form (filters via GET → shareable URLs)
GET /crawl?url=<path>	Detail view: gallery, pricing, variations, tables, provider extras
POST /fetch	Trigger a new crawl (form field url); saves to disk, redirects to detail
GET /data/…	Static serving of saved crawl files (images, PDFs, etc.)

---

list-dl

Scrapes order histories and wishlists from e-commerce sites on demand. Mirror of product-dl architecture but for lists instead of individual products.

Supported providers

Provider	Status	Notes
amazon_orders	Active	Amazon order history scraper (pagination, delivery states, items). Requires authentication.

Install

# Bundled with product-dl
pip install product-dl

Login & Credentials

list-dl uses a pluggable login provider system. Each provider handles site-specific authentication and stores credentials locally for reuse.

Login Command

# Interactive login (prompts for password)
list-dl login amazon --username user@example.com

# Non-interactive login (provide all credentials)
list-dl login amazon --username user@example.com --password mypassword

# Login to a specific flavor (e.g., amazon.de)
list-dl login amazon --flavor de --username user@example.com --password mypassword

Credentials are stored in ./crawl_data/list_dl/login/{site}/login.json.

Environment Variables

Credentials can be loaded from environment variables. Precedence (highest to lowest):

1. CLI arguments (--username, --password, --code)
2. LOGIN_{SITE}_{FLAVOR}_{FIELD} (e.g., LOGIN_AMAZON_DE_USERNAME)
3. LOGIN_{SITE}_{FIELD} (e.g., LOGIN_AMAZON_USERNAME)
4. Generic field name (e.g., USERNAME, PASSWORD, CODE) — Warning: generic USERNAME defaults to the Linux user if not explicitly set

# Load credentials from environment
export LOGIN_AMAZON_USERNAME=user@example.com
export LOGIN_AMAZON_PASSWORD=mypassword
list-dl login amazon

# Or with flavor
export LOGIN_AMAZON_DE_USERNAME=user@example.com
export LOGIN_AMAZON_DE_PASSWORD=mypassword
list-dl login amazon --flavor de

# Non-interactive (no TTY)
echo mypassword | list-dl login amazon --username user@example.com

.env File Support

The login command reads from .env.local (highest priority) and .env files:

# .env.local (excluded from git)
LOGIN_AMAZON_USERNAME=user@example.com
LOGIN_AMAZON_PASSWORD=mypassword

Then:

list-dl login amazon

Two-Factor Authentication (2FA / TOTP)

When a site requires 2FA, list-dl supports three workflows:

Workflow 1: Single command (all credentials at once)

list-dl login amazon --username user@example.com --password mypass --code 123456

Automatically chains login → 2FA → stores final credentials.

Workflow 2: Two separate commands

# First run: login, save partial credentials with TOTP form
list-dl login amazon --username user@example.com --password mypass
# Output: ⚠ TOTP code required. Run: list-dl login amazon --code <code> --flavor com

# Second run: submit 2FA code using stored credentials
list-dl login amazon --code 123456

Workflow 3: Interactive mode (detects TTY)

# Prompts for missing credentials, including 2FA code when required
list-dl login amazon
# Email address: user@example.com
# Password: (hidden input)
# 2FA Code (6 digits): 123456

2FA codes can also be provided via the CODE environment variable:

export CODE=123456
list-dl login amazon

CLI

# Fetch orders from Amazon (requires prior login)
list-dl fetch amazon orders --mode my_orders

# Fetch a public wishlist (no login required)
list-dl fetch amazon wishlist --url https://www.amazon.de/hz/wishlist/ls/ABC123

# Fetch from a different flavor (amazon.de)
list-dl fetch amazon orders --flavor de --mode my_orders

# Start the API server
list-dl serve --host 0.0.0.0 --port 7003

# Export the OpenAPI schema (default: YAML)
list-dl openapi
list-dl openapi --format json --output ./openapi.json

# List discovered providers
list-dl providers
list-dl providers --format json
list-dl login-providers --format json

API

list-dl serve          # http://127.0.0.1:7003

Credentials can be supplied in the request:

POST /fetch
{
  "site": "amazon",
  "category": "orders",
  "mode": "my_orders",
  "credentials": {"username": "...", "password": "...", "cookies": {...}},
  "fetch_details": true
}

Or, if already logged in, credentials are auto-loaded from disk:

curl -X POST http://localhost:7003/fetch \
  -H "Content-Type: application/json" \
  -d '{"site": "amazon", "category": "orders", "mode": "my_orders"}'

Output schema (Order)

Order
├── order_number: str                    unique order ID
├── provider_name: str                   e.g. "amazon_orders"
├── url: str | None                      link to order detail page
├── order_date: date | None              when order was placed
├── total: Price | None                  {value, unit, raw, symbol, suffix}
├── dispatch_to: Address | None          shipping address
├── delivery: DeliveryStatus | None      {state, substate}
│     state: "delivered" | "on_the_way_late" | "cancelled" | ...
│     substate: Optional freeform substate (e.g. "Parcel left in letterbox")
├── delivered_at: datetime | None
├── return_until: date | None
├── documents: list[Attachment] | None   invoices, receipts, etc.
├── summary: OrderSummary | None         price breakdown
│     ├── items_subtotal, postage, total_before_vat, estimated_vat, total, grand_total
├── payment: PaymentMethod | None        how order was paid
│     ├── type, bank, masked_number
├── items: list[OrderItem] | None        products in order
│     ├── link: URL
│     ├── product: ParsedProduct | None  (fetched via product-dl if available)
│     └── extra: <ProviderItemExtra>
└── extra: <ProviderOrderExtra>          provider-specific fields

Adding a provider

Built-in — create src/list_dl/providers/<name>/__init__.py with:

from list_dl.providers.base import ListProviderBase, register

@register
class MyOrdersProvider(ListProviderBase[CHECK_TYPE, MyOrderExtra, MyItemExtra]):
    PROVIDER_NAME = "my_orders"

    @classmethod
    def can(cls, provider_name, mode, url) -> CHECK_TYPE | None:
        """Return truthy if this provider can handle the request."""
        ...

    async def init(self) -> None:
        """Initialize (e.g., open HTTP client)."""
        ...

    async def fetch_orders(self) -> AsyncIterator[Order]:
        """Yield Order objects."""
        ...

    async def fetch_order_detail(self, order: Order) -> Order:
        """Enrich order with detail page data (items, summary, payment)."""
        return order  # or enriched order

Auto-discovered via pkgutil — no pyproject.toml changes needed. Use httpx with optional Akamai bypass via curl_cffi (same pattern as product-dl).

Deploy

1. Remember to bump the version in pyproject.toml:3

2. export PYPI_TOKEN=''
   

3. uv build
   uv publish --token "$PYPI_TOKEN"