ssc-codegen

Code generator for web scraping parsers. Describe HTML extraction rules in a declarative KDL 2.0 DSL, then generate ready-to-use parser code for multiple languages and libraries.

.kdl schema --> [kdl parser] --> AST --> [linter] --> [converter] --> output code

Features

Declarative DSL based on KDL 2.0 syntax
Static type checking and linting before code generation
Multiple output targets: Python (bs4, lxml, parsel, selectolax), JavaScript (DOM API), Go (goquery)
Typed user extensions with lazy target imports and runtime helpers
Struct types: item, list, dict, table, flat
LLM-friendly: system prompt + linter loop for AI-assisted schema generation

Install

uv tool install ssc_codegen

Quick example

books.kdl:

(list)struct Book {
    @split-doc { css-all ".product-card" }

    title { css ".title"; text }
    price { css ".price"; text; re #"(\d+\.\d+)"#; to-float }
    url   { css "a[href]"; attr "href"; fallback #null }
}

Generate Python parser:

ssc-gen generate python books.kdl -L bs4 -o ./output

Usage

Generate code

# single file (Python + bs4)
ssc-gen generate python schema.kdl -L bs4 -o ./output

# all .kdl files in a directory (JavaScript)
ssc-gen generate js examples/ -o ./output

# Go (goquery + net/http)
ssc-gen generate go schema.kdl -o ./output

# Go package defaults to main; override explicitly when generating a library
ssc-gen generate go schema.kdl -o ./output --package scraper

# with custom package name
ssc-gen generate python schema.kdl -L bs4 -o ./parsers --package scraper

# with @request support (REST/HTTP codegen)
ssc-gen generate python schema.kdl -L bs4 -o ./out --http-client httpx

# extract helper functions into a separate runtime module
ssc-gen generate python schema.kdl -L bs4 -o ./out -R

Languages: generate python, generate js, generate go. HTML libraries (--lib / -L, Python only): bs4 (default), lxml, parsel, slax.

Lint schemas

# human-readable output
ssc-gen check schema.kdl

# JSON output (for LLM pipelines)
ssc-gen check schema.kdl -f json

# check all files in a directory
ssc-gen check examples/

Test schema against HTML

# from file
ssc-gen run examples/booksToScrape.kdl:MainCatalogue -L bs4 -i page.html

# from stdin
curl https://books.toscrape.com/ | ssc-gen run examples/booksToScrape.kdl:MainCatalogue -L bs4

ssc-gen run executes generated Python in-process. Run only trusted schema files.

Health check (verify selectors match elements)

# from file
ssc-gen health examples/booksToScrape.kdl:MainCatalogue -i page.html

# from stdin
curl https://books.toscrape.com/ | ssc-gen health examples/booksToScrape.kdl:MainCatalogue

Documentation

Quick start
Syntax and file structure
Type system
Pipeline operations
Predicates and logic
JSON schemas and jsonify
@request — HTTP constructor + REST clients
LLM-compact reference -- full DSL spec in one file for LLM context
Examples

LLM integration

LLM agents can generate and validate .kdl schemas automatically using the linter feedback loop.

In AI-powered IDEs (Claude Code, Cursor, opencode, etc.)

Project works through agents + skills. Use skills under .agents/skills/:

sscgen-dsl — HTML scraping schemas (css, text, (item)struct, (list)struct, (table)struct, predicates, jsonify).
sscgen-rest — REST/JSON API clients ((rest)struct, @request, @error, typed placeholders).

Development

uv sync                  # install dependencies
uv build --wheel         # build wheel
uv run ruff check ssc_codegen/
uv run mypy ssc_codegen/

Running tests

# Run full test suite
uv run pytest

# Run full test suite in parallel (faster, recommended)
uv run pytest -n auto

# Fast test run: skip external toolchains (Rust cargo, Go, Node.js)
uv run pytest -m "not toolchain"

# Run only toolchain tests (Go, Rust, JavaScript integration)
uv run pytest -m toolchain

Test dependencies

Python and parser tests require only uv sync.

Integration smoke tests for external compilers are skipped automatically if their toolchains are missing:

JavaScript: Requires Node.js in PATH and npm install (jsdom in package.json).
Go: Requires go in PATH.
Rust: Requires cargo in PATH.

Metadata

Release files for ssc_codegen 0.43.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ssc_codegen 0.43.1
File	Size	Uploaded	Provenance
ssc_codegen-0.43.1.tar.gz	249.1 kB	Oct 22, 2026	Without provenance	Details

Built distribution (wheel)

Table of built distributions (wheels) for ssc_codegen 0.43.1
File	Interpreter	ABI	Platform	Reset
ssc_codegen-0.43.1-py3-none-any.whl 296.8 kB Oct 22, 2026 Without provenance	Python 3	none	any	Details

Total release size: 546.0 kB

Release files / ssc_codegen-0.43.1.tar.gz

Download URL	ssc_codegen-0.43.1.tar.gz
Size	249.1 kB
Tags	Source
SHA-256 checksum How to use checksums	`86a01f8631fc50e0bbbdd3aee8aa491b27b7c0af39915ef5915a5a1f8be1b26c`
BLAKE2b-256 checksum How to use checksums	`bb39edd27135f7614b8d54a444822ba9918c0b1ff68d1f3b5f4b90386481fdf3`
Upload date	Oct 22, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.11.14 {"installer":{"name":"uv","version":"0.11.14","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}`

Release files / ssc_codegen-0.43.1-py3-none-any.whl

Download URL	ssc_codegen-0.43.1-py3-none-any.whl
Size	296.8 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`2a360d79e70f77fecc435ba63bcb77840b4b5cd444a43b579bdc6162466c3bbc`
BLAKE2b-256 checksum How to use checksums	`604b8656ddb4eaa22e551e09f8c5f813403cdd8d9ab97f1cf7415ed93a1a8a04`
Upload date	Oct 22, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.11.14 {"installer":{"name":"uv","version":"0.11.14","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}`

Security for ssc_codegen 0.43.1

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.