ssc-codegen

Code generator for web scraping parsers. Describe HTML extraction rules in a declarative KDL 2.0 DSL, then generate ready-to-use parser code for multiple languages and libraries.

.kdl schema --> [kdl parser] --> AST --> [linter] --> [converter] --> output code

Features

Declarative DSL based on KDL 2.0 syntax
Static type checking and linting before code generation
Multiple output targets: Python (bs4, lxml, parsel, selectolax), JavaScript (DOM API), Go (goquery)
Typed user extensions with lazy target imports and runtime helpers
Struct types: item, list, dict, table, flat
LLM-friendly: system prompt + linter loop for AI-assisted schema generation

Install

uv tool install ssc_codegen

Quick example

books.kdl:

(list)struct Book {
    @split-doc { css-all ".product-card" }

    title { css ".title"; text }
    price { css ".price"; text; re #"(\d+\.\d+)"#; to-float }
    url   { css "a[href]"; attr "href"; fallback #null }
}

Generate Python parser:

ssc-gen generate python books.kdl -L bs4 -o ./output

Usage

Generate code

# single file (Python + bs4)
ssc-gen generate python schema.kdl -L bs4 -o ./output

# all .kdl files in a directory (JavaScript)
ssc-gen generate js examples/ -o ./output

# Go (goquery + net/http)
ssc-gen generate go schema.kdl -o ./output

# Go package defaults to main; override explicitly when generating a library
ssc-gen generate go schema.kdl -o ./output --package scraper

# with custom package name
ssc-gen generate python schema.kdl -L bs4 -o ./parsers --package scraper

# with @request support (REST/HTTP codegen)
ssc-gen generate python schema.kdl -L bs4 -o ./out --http-client httpx

# extract helper functions into a separate runtime module
ssc-gen generate python schema.kdl -L bs4 -o ./out -R

Languages: generate python, generate js, generate go. HTML libraries (--lib / -L, Python only): bs4 (default), lxml, parsel, slax.

Lint schemas

# human-readable output
ssc-gen check schema.kdl

# JSON output (for LLM pipelines)
ssc-gen check schema.kdl -f json

# check all files in a directory
ssc-gen check examples/

Test schema against HTML

# from file
ssc-gen run examples/booksToScrape.kdl:MainCatalogue -L bs4 -i page.html

# from stdin
curl https://books.toscrape.com/ | ssc-gen run examples/booksToScrape.kdl:MainCatalogue -L bs4

ssc-gen run executes generated Python in-process. Run only trusted schema files.

Health check (verify selectors match elements)

# from file
ssc-gen health examples/booksToScrape.kdl:MainCatalogue -i page.html

# from stdin
curl https://books.toscrape.com/ | ssc-gen health examples/booksToScrape.kdl:MainCatalogue

Documentation

Quick start
Syntax and file structure
Type system
Pipeline operations
Predicates and logic
JSON schemas and jsonify
@request — HTTP constructor + REST clients
LLM-compact reference -- full DSL spec in one file for LLM context
Examples

LLM integration

LLM agents can generate and validate .kdl schemas automatically using the linter feedback loop.

In AI-powered IDEs (Claude Code, Cursor, opencode, etc.)

Project works through agents + skills. Use skills under .agents/skills/:

sscgen-dsl — HTML scraping schemas (css, text, (item)struct, (list)struct, (table)struct, predicates, jsonify).
sscgen-rest — REST/JSON API clients ((rest)struct, @request, @error, typed placeholders).

Development

uv sync                  # install dependencies
uv build --wheel         # build wheel
uv run ruff check ssc_codegen/
uv run mypy ssc_codegen/

Running tests

# Run full test suite
uv run pytest

# Run full test suite in parallel (faster, recommended)
uv run pytest -n auto

# Fast test run: skip external toolchains (Rust cargo, Go, Node.js)
uv run pytest -m "not toolchain"

# Run only toolchain tests (Go, Rust, JavaScript integration)
uv run pytest -m toolchain

Test dependencies

Python and parser tests require only uv sync.

Integration smoke tests for external compilers are skipped automatically if their toolchains are missing:

JavaScript: Requires Node.js in PATH and npm install (jsdom in package.json).
Go: Requires go in PATH.
Rust: Requires cargo in PATH.

Metadata

Release files for ssc_codegen 0.42.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ssc_codegen 0.42.0
File	Size	Uploaded
ssc_codegen-0.42.0.tar.gz	242.0 kB	Oct 10, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for ssc_codegen 0.42.0
File	Interpreter	ABI	Platform	Reset
ssc_codegen-0.42.0-py3-none-any.whl 290.0 kB Oct 10, 2026	Python 3	none	any	Details

Total release size: 531.9 kB

Release files / ssc_codegen-0.42.0.tar.gz

Download URL	ssc_codegen-0.42.0.tar.gz
Size	242.0 kB
Tags	Source
SHA-256 checksum How to use checksums	`1b8d2c70b5e9d61be0f404b7881f6750f7cd1a4315389bcfaed02e816368b82a`
BLAKE2b-256 checksum How to use checksums	`f253d6714b84c2f4b876f0fb35c048e0396186b3828e21a03af46d8123f3d2fb`
Upload date	Oct 10, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.11.14 {"installer":{"name":"uv","version":"0.11.14","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}`

Release files / ssc_codegen-0.42.0-py3-none-any.whl

Download URL	ssc_codegen-0.42.0-py3-none-any.whl
Size	290.0 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`cc40912c08ab9d5bd75a086597cabceaccf66eac3be5ad779ba49b61d2e439c9`
BLAKE2b-256 checksum How to use checksums	`6835e89eb2bb1c67d93a4a5c137070ff588b10680dec5fd182ddabc49cdc2234`
Upload date	Oct 10, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.11.14 {"installer":{"name":"uv","version":"0.11.14","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}`