Scraplet DSL

Scraplet DSL is a safe, data-driven adapter engine for resolving URLs into structured outputs without executing arbitrary code.

It is designed as a reusable package that can be embedded in other applications (news readers, crawlers, ETL tools, feed resolvers).

Features

• URL trigger matching by domain and path regex
• Deterministic step pipeline with schema validation
• HTML extraction with CSS selectors (lxml + cssselect)
• Regex extraction and replacement operations
• JSON extraction with dot-path and array wildcard support
• Date helper steps (datetime, parse_date)
• Pluggable HTTP fetcher and HTML parser interfaces for testability
• Lazy HTML parser construction: ScriptEngine() does not import lxml/cssselect until a select step actually runs

Step Types

• fetch: GET a URL from url or url_var (exactly one is required at execution), saving the body to save_body_as; optional save_status_as, save_final_url_as, retry, retry_backoff, and timeout
• select: CSS-select from HTML using html_var, selector, and output; extract chooses text (default), html, or attr (with attr naming the attribute), and mode chooses first (default) or all matches
• regex: search/findall over variable content, with optional numeric or named capture-group selection and optional flags (i/m/s); without group, search returns the first capture group when the pattern has one, otherwise the full match
• replace: regex-replace an input variable, with optional flags (i/m/s) and non-negative count (0 means replace all)
• assign: assign a literal or templated value; template defaults to true
• assert: fail if a variable is missing/empty, with an optional custom message
• set_url: rewrite input_url using value (and optional template), keeping derived input_host/input_path/input_query in sync
• datetime: write the current UTC datetime to var, using an optional format (default %Y-%m-%d)
• parse_date: parse human dates from input_var into output using an optional format (default %Y-%m-%d); optional month_names for locale-specific month names bypasses dateparser
• json: parse JSON and extract by path (items[0].name, items[*].name, or the list-pluck path items.name.first); scalar values (int/float/bool) are preserved, and invalid array access on non-lists resolves to ""
• output: produce final output map

URL rewriting

set_url and replace (when output = "input_url") keep the derived URL parts in sync. After rewriting input_url, the variables input_host, input_path, and input_query are re-derived from the new URL, so later steps and output templates see consistent values.

set_url is URL-aware:

• A value containing ${input_url} is treated as path manipulation: the prefix/suffix around the reference is appended to the base path, the base query string is preserved, and the fragment is dropped. ${input_url}/rss against https://example.com/news?c=1 becomes https://example.com/news/rss?c=1. archive${input_url}/rss becomes https://example.com/archive/news/rss?c=1.
• A value with a scheme (e.g. https://other.example/feed) is treated as an absolute override.
• A relative value without ${input_url} (e.g. /feed) is joined onto the current input_url via urljoin, with the base's query and fragment stripped.

Regex safety

Adapter regexes are compiled with a conservative safety guard.

• regex.pattern, replace.pattern, and url_triggers.path_patterns reject nested repeated subpatterns such as ^(a+)+$, which are a common source of catastrophic backtracking in Python's re engine. Possessive quantifiers (a++) and atomic groups ((?>...)) suppress backtracking and are accepted.
• Repeated groups with clearly ambiguous alternatives, such as (a|a)+ or (a|aa)+, are rejected as well. The guard remains conservative and may reject some patterns whose overlap is difficult to determine statically.
• The guard is enforced during adapter load for bundle-defined adapters, and on first execution for step instances created directly in Python.
• This is a heuristic safe-subset check, not a complete ReDoS analyser. Keep adapter regexes simple and treat third-party-sourced adapters as untrusted input unless you review them.

JSON extraction typing

json step results preserve the underlying JSON scalar types. A path like items[*].id returns a list of int/float/bool values as they appear in the source, not their stringified form. Nested arrays are preserved as nested lists.

When a path applies [index] or [*] to a non-list value, the step returns an empty string rather than serializing the object at that path.

When a typed list is used inside a partial template such as "IDs=${ids}", its elements are stringified and joined with commas. An exact ${ids} template value remains the original typed list.

Datetime step timezone

The datetime step emits timestamps from datetime.now(datetime.UTC), i.e. a tz-aware UTC value. This avoids the cross-timezone correctness footgun that came from the previous naive datetime.now() (which silently used the host's local time). To force a specific offset in the formatted string, use the %z/%Z directives — e.g. format = "%Y-%m-%dT%H:%M:%S%z" produces a trailing +0000 for UTC.

Installation

Scraplet DSL supports Python 3.11 through 3.14. The 3.11 floor is intentional: the loader uses the stdlib tomllib module. CI runs format, lint, tests, build, and installed-wheel smoke checks on Python 3.11, 3.12, 3.13, and 3.14.

From PyPI

pip install scraplet-dsl

This installs the library with the default httpx-based HTTP fetcher and the lxml/cssselect HTML parser, plus the dateparser dependency used by parse_date when custom month names are not supplied. These are runtime dependencies, so a fresh pip install is enough to start resolving URLs.

Local editable install (for development)

git clone https://github.com/Baksalyar/Scraplet-DSL.git
cd Scraplet-DSL
pip install -e .

If you use Poetry, poetry install works the same way.

For Library Users

Once installed, the minimal end-to-end flow is:

from scraplet_dsl import ScriptEngine, load_adapter_bundle
from scraplet_dsl.engine import select_adapter

bundle = load_adapter_bundle(...)  # see "Adapter Bundle Example" below
adapter = select_adapter(bundle.adapters, url)
if adapter is None:
    raise RuntimeError("No adapter matched the URL")

result = ScriptEngine().resolve(adapter, url)
print(result.output)

Pass an on_step callback to opt into execution tracing. The callback receives immutable StepTrace objects with the step type, input/output variable names, duration, errors, and step details. The same events are available as result.trace; tracing is disabled and result.trace is None when no callback is supplied. Fetch steps include response status/final URL, and select steps include their match count.

These snippets use the default fetcher for brevity. When ScriptEngine() constructs its default HttpxFetcher, the fetcher warns that allow_domains is unset. Configure HttpxFetcher(allow_domains=...) for adapters or input URLs that are not fully trusted (see Network Hardening).

The in-memory bundle is a plain Python dict; its schema is described in Adapter Schema. A TOML file with the same top-level schema_version and adapters keys can be parsed with tomllib and passed to load_adapter_bundle. For directory-based adapter loading, load_adapters_from_dir treats each *.toml file as one adapter table.

Adapter Bundle Example

A runnable end-to-end example, defining one adapter in a Python dict bundle and resolving a URL through it:

from scraplet_dsl import load_adapter_bundle, ScriptEngine
from scraplet_dsl.engine import select_adapter

bundle = load_adapter_bundle(
    {
        "schema_version": 1,
        "adapters": [
            {
                "name": "example_article",
                "priority": 10,
                "url_triggers": {"domains": ["example.com"], "path_patterns": [r"^/news/"]},
                "steps": [
                    {"type": "fetch", "url_var": "input_url", "save_body_as": "html", "retry": 2, "retry_backoff": 0.5, "timeout": 20},
                    {"type": "select", "html_var": "html", "selector": "h1", "output": "title"},
                    {"type": "output", "output": {"title": "${title}"}},
                ],
            }
        ],
    }
)

url = "https://example.com/news/123"
adapter = select_adapter(bundle.adapters, url)
if adapter is None:
    raise RuntimeError("No adapter matched the URL")

result = ScriptEngine().resolve(adapter, url)
print(result.output)

Adapter Schema (bundle mode)

Top-level keys:

• schema_version: must be integer 1 (true/false are rejected)
• adapters: list of up to 200 adapter definitions

Adapter keys:

• name: unique adapter name (duplicates are rejected at load time)
• priority: integer priority (defaults to 100); lower value wins when multiple adapters match, with adapter name as the deterministic alphabetical tie-breaker
• url_triggers.domains: non-empty list of domains; each domain matches its exact host and any subdomain
• url_triggers.path_patterns: optional list of valid regex filters matched against the URL path (compiled during load and rejected if they use blocked nested-repeat forms)
• steps: ordered list of 1 to 50 step tables
• headers: optional table of HTTP headers attached to every fetch the adapter issues (validated as a str -> str map; engine-level fetcher headers take precedence on conflicts)

ScriptEngine(headers=...) sets default headers for its default HttpxFetcher; HttpxFetcher(headers=...) can be used when constructing a custom fetcher. Those host-configured headers override adapter headers with the same case-insensitive name, preventing an adapter from replacing credentials.

Selected numeric validation rules:

• fetch.retry: integer from 0 through 10 (inclusive)
• fetch.retry_backoff: finite number >= 0 (defaults to 0.5; retries sleep retry_backoff * attempt_number seconds)
• fetch.timeout: finite number > 0; when omitted, the active fetcher's default is used (HttpxFetcher defaults to 20.0 seconds). This caps a single HTTP request and is further capped by the remaining adapter-wide resolution budget (see Resolution Deadline).
• replace.count: integer >= 0

The loader validates structure, field types, numeric bounds, select options, and regex syntax/safety. Some conditional requirements are checked when a step runs: a fetch with neither url nor url_var fails at execution (specifying both is rejected during load), select requires attr when extract = "attr", and regex.mode must be search or findall.

Error Model

• ScrapletError: base class for all errors below
• ScriptValidationError: invalid schema or step declaration
• ExecutionError: runtime step failure
• MissingDependencyError: optional dependency missing at runtime

Runtime errors include adapter and step context through ScriptEngine.resolve.

Resolution Deadline

ScriptEngine.resolve accepts a timeout= keyword argument that sets a monotonic deadline for work that honors the resolution context:

result = engine.resolve(adapter, url, timeout=10.0)

When timeout is set, an internal monotonic deadline is propagated through ExecutionContext:

• The engine checks the deadline before each step. Once control returns from a step after the deadline, the next operation aborts with an ExecutionError referencing the step that was skipped.
• FetchStep checks the deadline before each fetch attempt, passes the smaller of fetch.timeout and the remaining budget to the fetcher, and rejects a result that returns after the deadline. It refuses to start another attempt if no budget is left for its retry backoff, and the retry-backoff sleep is also capped to the remaining budget. A custom fetcher must honor its timeout argument to stop in-flight work promptly; a late result is never accepted.
• A custom step that does not check the deadline can still run past it. Any subsequent step or retry will see the deadline already exceeded and abort.

timeout must be finite and > 0 when given and defaults to 60 seconds so adapter work is bounded when callers omit it. Trusted callers can pass None explicitly to disable the deadline.

Network Hardening (HttpxFetcher)

The default HttpxFetcher enforces a small security policy on every request:

• Scheme allowlist. Only http:// and https:// URLs are accepted. Unsupported schemes (ftp://, file://, ...) are rejected before the client opens a socket. The same check is applied to every redirect Location.
• Domain allowlist. When allow_domains=... is configured, the host of the initial URL and every redirect target is validated against the allowlist before the request is issued. Disallowed hosts are never contacted.
• Private-network blocking. Private, loopback, link-local, multicast, reserved, and unspecified addresses are blocked by default. Set block_private_networks=False only when an explicitly trusted integration needs to access those address ranges. IP literals and hostnames are checked before any request is issued.
• Redirect cap. Up to max_redirects (default 5) redirects are followed. Beyond that, the fetch raises httpx.HTTPError with a message that includes the configured cap (for example, too many redirects (>5)).
• Streaming size cap. Requests are issued in streaming mode, and response bodies are read in chunks via iter_bytes(). The fetch is aborted as soon as more than max_bytes (default 2_000_000) bytes are accumulated, so oversized responses cannot be fully buffered into memory.
• UTF-8 only. Bodies that fail UTF-8 decoding are rejected rather than silently producing mojibake.
• HTTP status codes are data, not errors. A non-2xx response (404, 410, 500, ...) is returned as a normal FetchResult carrying its status, body, and headers. Only transport-level and policy failures (unsupported scheme, disallowed host, redirect cap, oversized body, invalid UTF-8, timeout/connect error) raise. Use fetch.save_status_as plus an assert or output template if an adapter needs to branch on the status.

These guarantees apply to the final response (after redirects) as well as every intermediate hop.

Important: without allow_domains=..., HttpxFetcher allows any valid public http:// or https:// host and emits a warning at construction time. If adapters or input URLs can come from untrusted third parties, configure allow_domains:

from scraplet_dsl.http import HttpxFetcher

fetcher = HttpxFetcher(
    allow_domains=("example.com",),
)

Private-network blocking resolves hostnames before each request and redirect. This is a useful SSRF guard, but it is not a complete network sandbox; deploy network-level egress controls for high-risk untrusted adapter execution.

Development

poetry install
poetry run ruff format --check .
poetry run ruff check .
poetry run pytest
poetry build

The CI workflow runs these checks on Python 3.11, 3.12, 3.13, and 3.14, then installs the built wheel in a clean virtual environment for a public-API and typing-marker smoke test.

License

Scraplet DSL is distributed under the Apache License 2.0. See LICENSE for details.

External contributions are accepted under the contribution terms in CONTRIBUTING.md.

Compatibility Policy

• scraplet-dsl is pre-1.0 and may take breaking package/API changes in minor releases when that helps the project move faster
• Adapter-schema breaking changes must be deliberate and must bump schema_version
• User-visible changes and breakages should be recorded in CHANGELOG.md

Project Layout

• src/scraplet_dsl/engine.py: adapter matching and execution
• src/scraplet_dsl/loader.py: schema parsing and validation
• src/scraplet_dsl/steps.py: step implementations
• src/scraplet_dsl/http.py: fetcher protocol and default httpx fetcher
• src/scraplet_dsl/html.py: parser protocol and lxml implementation
• src/scraplet_dsl/variables.py: variable store and template helpers
• src/scraplet_dsl/types.py: shared types (Value, Variables, ResolutionResult)
• src/scraplet_dsl/errors.py: error hierarchy
• src/scraplet_dsl/regex_utils.py: regex flag parsing and the nested-repeat safety guard