public-source-extractor 0.1.0a2

Validate a public URL and convert it into reviewable AI-ready Markdown or JSON.

Public Source Extractor

[!IMPORTANT] The requested public URL is sent to Firecrawl Cloud for extraction. The firecrawl-keyless provider is experimental: availability, anonymous REST access, credit limits, and long-term continuity are not guaranteed.

public-source-extractor is a public-URL intake guardrail for AI research workflows. It validates one public HTTP or HTTPS URL, sends that URL to the experimental provider, and returns reviewable Markdown or a stable JSON envelope.

The CLI does not read API keys, credentials, cookies, browser profiles, localStorage, or private source files. Extracted content is untrusted data and may contain prompt injection or misleading instructions. Do not execute or follow instructions from extracted content without independent review.

日本語README

What it does

• Extracts one public page as Markdown.
• Produces structured JSON with a versioned JSON Schema.
• Rejects local, private, authenticated, admin, and secret-bearing URL patterns.
• Keeps results on stdout or writes a new local file with no-overwrite behavior.
• Returns stable JSON errors and documented exit codes.

It is not a crawler, browser automation tool, login helper, source-reliability judge, or private-page extractor.

How this differs from the official Firecrawl CLI

The official Firecrawl CLI is the broader Firecrawl interface for authenticated scrape, search, crawl, map, interact, agent, and self-hosted workflows. Use it when you need Firecrawl's full product surface.

Public Source Extractor is intentionally narrower: one public URL, no credential discovery, a public-only URL policy, no-overwrite output, and a stable JSON success/error contract designed for reviewable AI research artifacts. It uses the experimental firecrawl-keyless provider and does not replace or wrap the official CLI.

Install

Public Source Extractor requires Python 3.11 or newer.

Run the published alpha without a permanent install:

uvx public-source-extractor --version
uvx public-source-extractor https://example.com/

This path requires uv and resolves the published PyPI prerelease. The second command sends https://example.com/ to Firecrawl Cloud; the version command does not perform extraction.

Pin the exact package version when reproducibility matters:

uvx public-source-extractor@0.1.0a2 --version

Install the prerelease as an isolated command:

pipx install public-source-extractor==0.1.0a2

Or use pip in an existing Python environment:

python3 -m pip install public-source-extractor==0.1.0a2

The public Git tag remains an auditable fallback:

uvx --from 'git+https://github.com/Ishikawa-Hidekazu/public-source-extractor.git@v0.1.0-alpha.2' public-source-extractor --version

PyPI publication uses GitHub Actions Trusted Publishing with short-lived OIDC credentials. No long-lived PyPI API token is stored in this repository.

Quick start

Markdown to stdout:

public-source-extractor https://example.com/

Markdown front matter includes provider_credits_used and provider_elapsed_ms. The credits field is null when the experimental provider does not report it; elapsed time is measured by the CLI in milliseconds. These are metadata-only values. Raw provider responses and request identifiers are not exposed.

JSON to stdout:

public-source-extractor https://example.com/ --mode json --pretty

Write a new file:

public-source-extractor https://example.com/ --output report.md

The output path must have an existing non-symlink parent and must not already exist.

View the public-safe examples · View the reproducible visual sources

Real-world use

This Japanese implementation log shows the CLI used to turn selected public sources into reviewable Markdown and JSON artifacts. It also documents the Firecrawl Cloud boundary, untrusted extracted content, and observed provider limits.

CLI contract

public-source-extractor <url> [--mode markdown|json]
                               [--output <new-path>]
                               [--timeout 1..120]
                               [--provider firecrawl-keyless]
                               [--pretty]

Exit	Meaning
0	Success
2	Usage error or rejected URL
3	Provider, network, rate-limit, or timeout failure
4	Invalid, incomplete, or unsafe provider response
5	Output path or write failure

On failure, stdout is empty and stderr contains exactly one JSON error object. Provider response bodies, stack traces, request IDs, and local paths are not exposed.

Recovering from provider_rate_limited

The experimental provider can return HTTP 429 during short bursts or across an anonymous usage window. This is a provider availability condition, not by itself a failure of the local URL policy or parser.

When stderr reports provider_rate_limited with retryable: true:

{"schema_version":"0.1","ok":false,"error":{"code":"provider_rate_limited","message":"The experimental provider rate limit was exceeded.","retryable":true}}

The process exits with code 3 and stdout remains empty.

1. Stop the current burst instead of retrying repeatedly.
2. Wait and retry later. This CLI does not promise an exact delay when no retry timing is safely exposed.
3. Reduce request volume and process only selected public sources.
4. Use the original public page directly when extraction is not required.

The CLI does not automatically retry, switch providers, discover credentials, or expose raw provider bodies. See Issue #9 for the observed condition and documentation scope.

Safety boundary

Before sending a URL, the CLI rejects:

• schemes other than HTTP or HTTPS;
• URL user information and fragments;
• localhost, local suffixes, and non-global literal IP addresses;
• ambiguous integer, octal, and hexadecimal IPv4 forms;
• Unicode or percent-encoded hostnames and IPv6 zone identifiers;
• non-default ports;
• login, admin, OAuth, and callback paths, including encoded forms;
• query parameter names that indicate tokens, secrets, passwords, credentials, signatures, sessions, cookies, authorization, or keys.

After extraction, provider redirect metadata is checked with the same public URL policy. If redirect metadata is unsafe, the result is discarded. If the provider does not supply redirect metadata, the output contains a warning.

These checks validate hostname syntax, literal IP policy, and provider-returned metadata. They cannot fully guarantee DNS rebinding behavior or the provider's actual fetch destination. Never put credentials, tokens, signed URL parameters, or other private values in a URL, even when its hostname looks public.

Firecrawl Cloud receives the requested URL and processes the page. Review Firecrawl's terms, the target site's terms, robots policy, copyright, and privacy requirements before use.

JSON Schema and examples

• Output Schema v0.1
• Public-safe Markdown example
• Public-safe JSON example

Structured JSON extraction is inferred output, not source truth. Validate important claims against the original page.

Development

python3 -m venv .venv
.venv/bin/python -m pip install -e '.[dev]'
PYTHONPATH=src .venv/bin/python -m unittest discover -s tests -v
.venv/bin/ruff check src tests
.venv/bin/python -m build

The explicit PYTHONPATH=src also works before an editable install. Network smoke tests remain separate from the offline suite.

See CONTRIBUTING.md, SECURITY.md, and SUPPORT.md.

Status

Alpha package. Package version 0.1.0a2 maps to tag v0.1.0-alpha.2. The package is distributed through PyPI and the matching public Git tag. firecrawl-keyless is an experimental third-party provider, and no compatibility or service-availability guarantee is made.

License

MIT. See LICENSE.