Skip to main content

sekoia-hwl

sekoia-hwl is a library and command-line tool for the Sekoia HTTP Workflow Language (HWL). It validates five kinds of YAML documents, and runs workflows and connectors, so that an integration can be written and tested locally before it is deployed on Sekoia.io:

Kind Purpose
workflow A pipeline of HTTP, GraphQL and output stages (a document without kind is a workflow)
connection An API: its account fields (config), base URL, auth method, HTTP options and query engine
table A table queryable with SOL, backed by an API (filters, pagination, hydration, aggregations) or by a query in another language (KQL, SPL, PowerQuery, ES|QL...)
connector A collector of events or assets (checkpoint, window, events pipeline, asset mapping)
dialect How SOL commands are written in a query language

Table and dialect documents are validated here; they are queried on Sekoia.io. The library has no connection to Sekoia.io: collected events and workflow outputs are printed on stdout, accounts come from a local file and connector state lives in a local directory.

Installation

pip install sekoia-hwl

The hwl command is installed with the package. The older shwl-runner and shwl-validator commands are installed too, as aliases of hwl run and hwl validate.

Command line

run, validate and connector run share their exit codes: 0 success, 1 invalid documents, 2 file not found, 3 invalid file (not readable, not YAML, an invalid accounts or state file), 4 runtime error. Usage errors (an unknown option, a missing argument) exit with 2, as in every click command. connector run prints records only on stdout, one JSON object per line; logs and warnings go to stderr.

Validate documents

hwl validate workflow.yml                         # one workflow, human-readable summary
hwl validate --json workflow.yml                  # {"valid": ..., "errors": [...]}
hwl validate documents/ more/connection.yml       # several files and directories, validated as one set
hwl validate --jsonl documents/                   # one diagnostic per line; nothing at all when valid

Run a workflow

hwl run workflow.yml -i query=sekoia.io -i limit=10
hwl run workflow.yml --inputs inputs.yaml -i limit=10   # --input values win over the file

The workflow outputs are printed as one JSON object. Validation diagnostics go to stderr; --no-validate skips validation and --strict-validation also reports W101 warnings.

Run a connector

hwl connector run delinea_pra_events --documents documents/ --accounts accounts.yml
hwl connector run sentinelone_assets --documents documents/ --accounts accounts.yml --account eu --follow

The connector's checkpoint, deduplication keys and rotated secrets are stored in <state-dir>/<connector>/<account>.json (--state-dir, default .hwl-state). With --follow, a run that emitted something is followed immediately by another one; otherwise the next run waits for schedule.every.

Documents

A connection declares the fields of an account and how to authenticate:

hwl: "1.1"
kind: connection
name: sentinelone
config:
  base_url:  {type: string, format: uri}
  api_token: {type: string, secret: true}
base_url: "{{ config.base_url }}"
auth:
  bearer: {token: "{{ config.api_token }}", scheme: ApiToken}

Tables and connectors reference their connection by name. The JSON Schema of each kind, shipped in sekoia_hwl/schemas/ (connection.json, table.json, connector.json, dialect.json), describes every key; editors that read JSON Schemas complete and check documents with them.

In the new kinds, a string that is exactly one {{ expression }} evaluates to the expression's native value (a datetime, a list, a number...), other templated strings are rendered to text, and request parameters, headers, cookies and JSON body members whose value is null are omitted. Paths such as $.data[*].id are RFC 9535 JSONPath.

Accounts file

connections:
  sentinelone:
    - name: eu
      config: {base_url: "https://eu1.sentinelone.net", api_token: "${S1_EU_TOKEN}"}
    - name: us
      config: {base_url: "https://usea1.sentinelone.net", api_token: "${S1_US_TOKEN}"}

${VAR} is replaced from the environment (a missing variable is an error), and each config is validated against the connection's config. Fields declared with secret: true, workflow inputs declared with secret: true and all credentials are masked (***) in logs, errors, warnings and command output.

Python API

Run a workflow:

from sekoia_hwl.workflow.configuration import Configuration
from sekoia_hwl.workflow.runner import WorkflowRunner

runner = WorkflowRunner(definition, Configuration())
outputs = runner.run({"query": "sekoia.io"})  # from synchronous code
outputs = await runner.arun({"query": "sekoia.io"})  # from a running event loop

Validate documents, then run a connector once:

import sys
from pathlib import Path

from sekoia_hwl.accounts import load_accounts, resolve_config
from sekoia_hwl.connectors import ConnectorRunner, JsonLinesSink, StateStore
from sekoia_hwl.validation.documents import load_document_set

# Raises InvalidDocumentsError (with the diagnostics of every file) when a document is invalid
documents = load_document_set([Path("documents")])
connector = documents.connectors["delinea_pra_events"]
connection = documents.connection_of(connector)
account = load_accounts(Path("accounts.yml")).account(connection.name, "default")
runner = ConnectorRunner(
    connector,
    connection,
    account.name,
    resolve_config(connection, account),
    sink=JsonLinesSink(sys.stdout.buffer),
    state=StateStore.for_account(Path(".hwl-state"), connector.name, account.name),
)
result = await runner.run_once()  # RunResult(emitted=..., warnings=[...])

sekoia_hwl.validation.documents.validate_documents returns per-file diagnostics without raising, and python -m sekoia_hwl.schemas regenerates the JSON Schemas of the new kinds in sekoia_hwl/schemas/.

Embedding the runtime

The hwl command runs documents its user wrote, with generous limits. A service that runs documents written by others (and calls APIs that may be hostile) bounds every execution and supplies its own network access. Everything below lives in context variables: a scope applies to the code in its block, to the tasks it starts and to the threads they start.

Limits

RuntimeLimits (sekoia_hwl.limits) is a frozen dataclass of every bound: document size, YAML depth and nodes, response bytes (as received and decompressed), zip members, decoded depth and lines, requests per execution, request timeout, retries, backoff, rate-limit interval, pages, rows, job polling interval and timeout, hydration concurrency, template fuel and output, template cache size, regex length and compile memory, convert_value literal size, RSA key bits and PEM size. RuntimeLimits.service_defaults() are the limits of a multi-tenant service, and apply by default: the hwl command runs a document within the limits it will have once deployed. Values written in documents (limits.max_pages, job.poll.every, options.timeout, retry.total, ...) are clamped to them.

from dataclasses import replace

from sekoia_hwl.limits import RuntimeLimits, limits_scope, set_default_limits

set_default_limits(RuntimeLimits.service_defaults())  # the whole process, every thread
with limits_scope(replace(RuntimeLimits.service_defaults(), max_rows=10_000)):
    ...  # this block only

Template fuel only bounds loops and calls: an expression such as 'x' * 10**8 builds its value before the output cap (max_template_output_bytes) rejects it, so the process memory must also be bounded (e.g. RLIMIT_AS).

Request budget

A RequestBudget(max_requests, timeout=None) counts every request of an execution (retries, token and login requests included) and refuses those past its deadline (RequestBudgetExceededError, ExecutionTimeoutError, both LimitExceededError). WorkflowRunner.arun and connector runs use the current budget, or create one from the limits (max_requests, execution_timeout).

from sekoia_hwl.limits import RequestBudget, budget_scope

with budget_scope(RequestBudget(limits.max_requests, timeout=seconds_until_deadline)):
    outputs = await WorkflowRunner(workflow, configuration).arun(inputs)

Network

Every HTTP client is created by sekoia_hwl.http.client.create_client: it never follows redirects, bounds bodies and decompression, gives each request a deadline (a server that sends its body byte by byte cannot keep it alive) and counts it in the budget. The network transports at the bottom of each client, and the proxies they go through, come from the current HttpEnvironment. Its proxies default to the standard variables of the process (HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, NO_PROXY, read when the environment is created); the rest of the process environment is ignored:

import httpx2

from sekoia_hwl.http.client import HttpEnvironment, Proxies, http_environment_scope, set_http_environment


def transport(verify: bool, pool_limits: httpx2.Limits, proxy: httpx2.Proxy | None) -> httpx2.AsyncBaseTransport:
    return httpx2.AsyncHTTPTransport(verify=verify, limits=pool_limits, proxy=proxy, trust_env=False)


set_http_environment(HttpEnvironment(transport_factory=transport, pool_limits=httpx2.Limits(max_connections=10)))
# Without proxies, whatever the process environment: HttpEnvironment(proxies=Proxies())

Credentials are only sent to the origin of the connection's base_url, which a connection with credentials must have (E122): a request to another origin goes without the auth and without the headers, cookies and parameters that contain a secret of the account, and is refused (CrossOriginSecretError) when its URL or body contains one; a next_url or hydration URL on another origin is requested without the headers of the request either, with warning HWL_CROSS_ORIGIN_NEXT_URL.

Caches

Auth objects (and their tokens) are kept by a process-wide bounded cache, which a host configures; cache_scope partitions it, and the rate limiters, per key (e.g. a tenant). A scope opened inside another one partitions it further:

from sekoia_hwl.caching import cache_scope
from sekoia_hwl.sources.session import auth_cache

auth_cache.configure(max_entries=64, ttl=3600)
with cache_scope(community_uuid):
    ...

An auth object is bound to the secret store of the sessions that built it: rotated refresh tokens always go to that store. Sessions created without a store share the in-memory store of their cache scope.

Documents without files

from sekoia_hwl.documents.loader import DocumentSet, SourceDocument
from sekoia_hwl.validation.documents import validate_documents

sources = [SourceDocument.from_text(text, path) for path, text in uploaded]  # hardened YAML, source positions
sources.append(SourceDocument.from_data(canonical_json, "tables/agents.yml"))  # parsed content, no positions
result = validate_documents(sources)  # per-file diagnostics
documents = DocumentSet.from_sources(sources)  # once the result is valid

The YAML loader refuses aliases (and merge keys) and bounds the size, depth and nodes of a document; InvalidDocumentFileError reports the reason.

What templates see

Templates get read-only views of their context: mappings, lists, bytes and datetimes keep their values, fields, comparisons and the read-only items(), keys(), values() and get() of mappings, but no other method (config.pop(...), items.extend(...) and now.replace(...) fail). Values returned by expressions are copies. Template errors never include the context (minijinja's debug report is off), and secrets stay masked in errors, warnings, logs and results.

Extensions

Other distributions add hwl commands through the sekoia_hwl.commands entry point group (a click command, named after its entry point) and rules across a document set through sekoia_hwl.set_rules (a callable taking a sekoia_hwl.validation.documents.SetContext and yielding (SourceDocument, ValidationError) pairs).

Development

uv sync --all-extras --dev
uv run pytest --keepalive tests
uv run ruff check && uv run ruff format --check
uv run mypy sekoia_hwl tests

License

Apache-2.0, see LICENSE.

Metadata

Release files for sekoia-hwl 0.11.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for sekoia-hwl 0.11.0
File Size Uploaded Provenance Actions
sekoia_hwl-0.11.0.tar.gz 164.2 kB Without provenance Details for sekoia_hwl-0.11.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for sekoia-hwl 0.11.0
File Interpreter ABI Platform Actions
sekoia_hwl-0.11.0-py3-none-any.whl Python 3 none any Details for sekoia_hwl-0.11.0-py3-none-any.whl

Total release size: 384.8 kB

Release files / sekoia_hwl-0.11.0.tar.gz

Download URL sekoia_hwl-0.11.0.tar.gz
Size 164.2 kB
Tags Source
SHA-256 checksum
How to use checksums
c2cb37511cc9c1170b85368a3fa307f9aafdb53633fe4e7b5771867ba579374f
BLAKE2b-256 checksum
How to use checksums
7645d7c3f9d48d6c4a5e4f342ea89ce69904626664b3310216cf6da4d03cda0b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / sekoia_hwl-0.11.0-py3-none-any.whl

Download URL sekoia_hwl-0.11.0-py3-none-any.whl
Size 220.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
dbe86de1f3b0a3b8425813435f7e19b98566405161b6c8918d9a6d3535811762
BLAKE2b-256 checksum
How to use checksums
425aa4a0f5d2cb154dc552d5ee96c729325bf63505e83fc41953668506e2b55c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for sekoia-hwl 0.11.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.11.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page