Scraplet DSL

Scraplet DSL is a safe, data-driven adapter engine for resolving URLs into structured outputs without executing arbitrary code.

It is designed as a reusable package that can be embedded in other applications (news readers, crawlers, ETL tools, feed resolvers).

Features

• URL trigger matching by domain and path regex
• Deterministic step pipeline with strict schema validation
• HTML extraction with CSS selectors (lxml + cssselect)
• Regex extraction and replacement operations
• JSON extraction with dot-path and array wildcard support
• Date helper steps (datetime, parse_date)
• Pluggable HTTP fetcher and HTML parser interfaces for testability
• Lazy HTML parser construction: ScriptEngine() does not import lxml/cssselect until a select step actually runs

Step Types

• fetch: GET a URL from url or url_var (specify exactly one), saving the body to save_body_as; optional save_status_as, save_final_url_as, retry, retry_backoff, and timeout
• select: CSS-select from HTML using html_var, selector, and output; extract chooses text (default), html, or attr (with attr naming the attribute), and mode chooses first (default) or all matches
• regex: search/findall over variable content, with optional numeric or named capture-group selection and optional flags (i/m/s)
• replace: regex-replace an input variable, with optional flags (i/m/s) and non-negative count (0 means replace all)
• assign: assign a literal or templated value; template defaults to true
• assert: fail if a variable is missing/empty, with an optional custom message
• set_url: rewrite input_url using value (and optional template), keeping derived input_host/input_path/input_query in sync
• datetime: write the current UTC datetime to var, using an optional format (default %Y-%m-%d)
• parse_date: parse human dates from input_var into output using an optional format (default %Y-%m-%d); optional month_names for locale-specific month names bypasses dateparser
• json: parse JSON and extract by path (items[0].name, items[*].name); scalar values (int/float/bool) are preserved, and invalid array access on non-lists resolves to ""
• output: produce final output map

URL rewriting

set_url and replace (when output = "input_url") keep the derived URL parts in sync. After rewriting input_url, the variables input_host, input_path, and input_query are re-derived from the new URL, so later steps and output templates see consistent values.

set_url is URL-aware:

• A value containing ${input_url} is treated as path manipulation: the prefix/suffix around the reference is appended to the base path, the base query string is preserved, and the fragment is dropped. ${input_url}/rss against https://example.com/news?c=1 becomes https://example.com/news/rss?c=1. archive${input_url}/rss becomes https://example.com/archive/news/rss?c=1.
• A value with a scheme (e.g. https://other.example/feed) is treated as an absolute override.
• A relative value without ${input_url} (e.g. /feed) is joined onto the current input_url via urljoin, with the base's query and fragment stripped.

Regex safety

Adapter regexes are compiled with a conservative safety guard.

• regex.pattern, replace.pattern, and url_triggers.path_patterns reject nested repeated subpatterns such as ^(a+)+$, which are a common source of catastrophic backtracking in Python's re engine. Possessive quantifiers (a++) and atomic groups ((?>...)) suppress backtracking and are accepted.
• The guard is enforced during adapter load for bundle-defined adapters, and on first execution for step instances created directly in Python.
• This is a heuristic safe-subset check, not a complete ReDoS analyser: it targets the nested-quantifier shape and does not catch every backtracking bomb (e.g. ambiguous alternation like (a|a)+). Keep adapter regexes simple and treat third-party-sourced adapters as untrusted input unless you review them.

JSON extraction typing

json step results preserve the underlying JSON scalar types. A path like items[*].id returns a list of int/float/bool values as they appear in the source, not their stringified form. Nested arrays are preserved as nested lists.

When a path applies [index] or [*] to a non-list value, the step returns an empty string rather than serializing the object at that path.

Datetime step timezone

The datetime step emits timestamps from datetime.now(datetime.UTC), i.e. a tz-aware UTC value. This avoids the cross-timezone correctness footgun that came from the previous naive datetime.now() (which silently used the host's local time). To force a specific offset in the formatted string, use the %z/%Z directives — e.g. format = "%Y-%m-%dT%H:%M:%S%z" produces a trailing +0000 for UTC.

Installation

Scraplet DSL supports Python 3.11, 3.12, and 3.13. The 3.11 floor is intentional: the loader uses the stdlib tomllib module. CI runs format, lint, tests, build, and installed-wheel smoke checks on all supported Python versions.

From PyPI

pip install scraplet-dsl

This installs the library with the default httpx-based HTTP fetcher and the lxml/cssselect HTML parser, plus the dateparser dependency used by parse_date when custom month names are not supplied. These are runtime dependencies, so a fresh pip install is enough to start resolving URLs.

Local editable install (for development)

git clone https://github.com/Baksalyar/Scraplet-DSL.git
cd Scraplet-DSL
pip install -e .

If you use Poetry, poetry install works the same way.

For Library Users

Once installed, the minimal end-to-end flow is:

from scraplet_dsl import ScriptEngine, load_adapter_bundle
from scraplet_dsl.engine import select_adapter

bundle = load_adapter_bundle(...)  # see "Adapter Bundle Example" below
adapter = select_adapter(bundle.adapters, url)
if adapter is None:
    raise RuntimeError("No adapter matched the URL")

result = ScriptEngine().resolve(adapter, url)
print(result.output)

These snippets use the default fetcher for brevity. Constructing ScriptEngine() without an allow_domains policy emits a warning; configure HttpxFetcher(allow_domains=...) for adapters or input URLs that are not fully trusted (see Network Hardening).

The in-memory bundle is a plain Python dict; its schema is described in Adapter Schema. A TOML file with the same top-level schema_version and adapters keys can be parsed with tomllib and passed to load_adapter_bundle. For directory-based adapter loading, load_adapters_from_dir treats each *.toml file as one adapter table.

Adapter Bundle Example

A runnable end-to-end example, defining one adapter in a Python dict bundle and resolving a URL through it:

from scraplet_dsl import load_adapter_bundle, ScriptEngine
from scraplet_dsl.engine import select_adapter

bundle = load_adapter_bundle(
    {
        "schema_version": 1,
        "adapters": [
            {
                "name": "example_article",
                "priority": 10,
                "url_triggers": {"domains": ["example.com"], "path_patterns": [r"^/news/"]},
                "steps": [
                    {"type": "fetch", "url_var": "input_url", "save_body_as": "html", "retry": 2, "retry_backoff": 0.5, "timeout": 20},
                    {"type": "select", "html_var": "html", "selector": "h1", "output": "title"},
                    {"type": "output", "output": {"title": "${title}"}},
                ],
            }
        ],
    }
)

url = "https://example.com/news/123"
adapter = select_adapter(bundle.adapters, url)
if adapter is None:
    raise RuntimeError("No adapter matched the URL")

result = ScriptEngine().resolve(adapter, url)
print(result.output)

Adapter Schema (bundle mode)

Top-level keys:

• schema_version: must be integer 1 (true/false are rejected)
• adapters: list of adapter definitions

Adapter keys:

• name: unique adapter name (duplicates are rejected at load time)
• priority: integer priority (defaults to 100); lower value wins when multiple adapters match, with adapter name as the deterministic alphabetical tie-breaker
• url_triggers.domains: non-empty list of domains; each domain matches its exact host and any subdomain
• url_triggers.path_patterns: optional list of valid regex filters matched against the URL path (compiled during load and rejected if they use blocked nested-repeat forms)
• steps: ordered list of step tables
• headers: optional table of HTTP headers attached to every fetch the adapter issues (validated as a str -> str map; values override the engine-level fetcher headers)

ScriptEngine(headers=...) sets default headers for its default HttpxFetcher; HttpxFetcher(headers=...) can be used when constructing a custom fetcher. Adapter headers override those defaults for their requests.

Selected numeric validation rules:

• fetch.retry: integer from 0 through 10 (inclusive)
• fetch.retry_backoff: number >= 0 (defaults to 0.5; retries sleep retry_backoff * attempt_number seconds)
• fetch.timeout: number > 0; when omitted, the active fetcher's default is used (HttpxFetcher defaults to 20.0 seconds). This caps a single HTTP request and is separate from the adapter-wide resolution deadline (see Resolution Deadline).
• replace.count: integer >= 0

Error Model

• ScrapletError: base class for all errors below
• ScriptValidationError: invalid schema or step declaration
• ExecutionError: runtime step failure
• MissingDependencyError: optional dependency missing at runtime

Runtime errors include adapter and step context through ScriptEngine.resolve.

Resolution Deadline

ScriptEngine.resolve accepts an optional timeout= keyword argument that sets a monotonic deadline for work that honors the resolution context:

result = engine.resolve(adapter, url, timeout=10.0)

When timeout is set, an internal monotonic deadline is propagated through ExecutionContext:

• The engine checks the deadline before each step. Once control returns from a step after the deadline, the next operation aborts with an ExecutionError referencing the step that was skipped.
• FetchStep checks the deadline before each fetch attempt and refuses to start another attempt if no budget is left for its retry backoff. The retry-backoff sleep is also capped to the remaining budget, so a slow or wedged fetch cannot exhaust the budget while sleeping between retries. The deadline does not interrupt an HTTP request that is already in progress; use fetch.timeout to cap that individual request.
• A custom step that does not check the deadline can still run past it. Any subsequent step or retry will see the deadline already exceeded and abort.

timeout must be > 0 when given. The default (None) preserves the prior behavior — no deadline is propagated and ExecutionContext.deadline stays unset.

Network Hardening (HttpxFetcher)

The default HttpxFetcher enforces a small security policy on every request:

• Scheme allowlist. Only http:// and https:// URLs are accepted. Unsupported schemes (ftp://, file://, ...) are rejected before the client opens a socket. The same check is applied to every redirect Location.
• Domain allowlist. When allow_domains=... is configured, the host of the initial URL and every redirect target is validated against the allowlist before the request is issued. Disallowed hosts are never contacted.
• Private-network blocking. Private, loopback, link-local, multicast, reserved, and unspecified addresses are blocked by default. Set block_private_networks=False only when an explicitly trusted integration needs to access those address ranges. IP literals and hostnames are checked before any request is issued.
• Redirect cap. Up to max_redirects (default 5) redirects are followed. Beyond that, the fetch fails with httpx.HTTPError("too many redirects").
• Streaming size cap. Requests are issued in streaming mode, and response bodies are read in chunks via iter_bytes(). The fetch is aborted as soon as more than max_bytes (default 2_000_000) bytes are accumulated, so oversized responses cannot be fully buffered into memory.
• UTF-8 only. Bodies that fail UTF-8 decoding are rejected rather than silently producing mojibake.
• HTTP status codes are data, not errors. A non-2xx response (404, 410, 500, ...) is returned as a normal FetchResult carrying its status, body, and headers. Only transport-level and policy failures (unsupported scheme, disallowed host, redirect cap, oversized body, invalid UTF-8, timeout/connect error) raise. Use fetch.save_status_as plus an assert or output template if an adapter needs to branch on the status.

These guarantees apply to the final response (after redirects) as well as every intermediate hop.

Important: without allow_domains=..., HttpxFetcher allows any valid public http:// or https:// host and emits a warning at construction time. If adapters or input URLs can come from untrusted third parties, configure allow_domains:

from scraplet_dsl.http import HttpxFetcher

fetcher = HttpxFetcher(
    allow_domains=("example.com",),
)

Private-network blocking resolves hostnames before each request and redirect. This is a useful SSRF guard, but it is not a complete network sandbox; deploy network-level egress controls for high-risk untrusted adapter execution.

If a fetch step omits timeout, the active fetcher's default is used. The default HttpxFetcher uses 20.0 seconds per request. This caps an individual HTTP request and is distinct from the adapter-wide resolution deadline passed to ScriptEngine.resolve(timeout=...) (see Resolution Deadline).

Development

poetry install
poetry run ruff format --check .
poetry run ruff check .
poetry run pytest
poetry build

The CI workflow runs these checks across Python 3.11, 3.12, and 3.13, then installs the built wheel in a clean virtual environment for a smoke test.

License

Scraplet DSL is distributed under the Apache License 2.0. See LICENSE for details.

External contributions are accepted under the contribution terms in CONTRIBUTING.md.

Compatibility Policy

• scraplet-dsl is pre-1.0 and may take breaking package/API changes in minor releases when that helps the project move faster
• Adapter-schema breaking changes must be deliberate and must bump schema_version
• User-visible changes and breakages should be recorded in CHANGELOG.md

Project Layout

• src/scraplet_dsl/engine.py: adapter matching and execution
• src/scraplet_dsl/loader.py: schema parsing and validation
• src/scraplet_dsl/steps.py: step implementations
• src/scraplet_dsl/http.py: fetcher protocol and default httpx fetcher
• src/scraplet_dsl/html.py: parser protocol and lxml implementation
• src/scraplet_dsl/variables.py: variable store and template helpers
• src/scraplet_dsl/types.py: shared types (Value, Variables, ResolutionResult)
• src/scraplet_dsl/errors.py: error hierarchy
• src/scraplet_dsl/regex_utils.py: regex flag parsing and the nested-repeat safety guard