ssc-codegen

Code generator for web scraping parsers. Describe HTML extraction rules in a declarative KDL 2.0 DSL, then generate ready-to-use parser code for multiple languages and libraries.

.kdl schema --> [kdl parser] --> AST --> [linter] --> [converter] --> output code

Features

Declarative DSL based on KDL 2.0 syntax
Static type checking and linting before code generation
Multiple output targets: Python (bs4, lxml, parsel, selectolax), JavaScript (DOM API), Go (goquery)
Typed user extensions with lazy target imports and runtime helpers
Struct types: item, list, dict, table, flat
LLM-friendly: system prompt + linter loop for AI-assisted schema generation

Install

uv tool install ssc_codegen

Quick example

books.kdl:

(list)struct Book {
    @split-doc { css-all ".product-card" }

    title { css ".title"; text }
    price { css ".price"; text; re #"(\d+\.\d+)"#; to-float }
    url   { css "a[href]"; attr "href"; fallback #null }
}

Generate Python parser:

ssc-gen generate python books.kdl -L bs4 -o ./output

Usage

Generate code

# single file (Python + bs4)
ssc-gen generate python schema.kdl -L bs4 -o ./output

# all .kdl files in a directory (JavaScript)
ssc-gen generate js examples/ -o ./output

# Go (goquery + net/http)
ssc-gen generate go schema.kdl -o ./output

# Go package defaults to main; override explicitly when generating a library
ssc-gen generate go schema.kdl -o ./output --package scraper

# with custom package name
ssc-gen generate python schema.kdl -L bs4 -o ./parsers --package scraper

# with @request support (REST/HTTP codegen)
ssc-gen generate python schema.kdl -L bs4 -o ./out --http-client httpx

# extract helper functions into a separate runtime module
ssc-gen generate python schema.kdl -L bs4 -o ./out -R

Languages: generate python, generate js, generate go. HTML libraries (--lib / -L, Python only): bs4 (default), lxml, parsel, slax.

Lint schemas

# human-readable output
ssc-gen check schema.kdl

# JSON output (for LLM pipelines)
ssc-gen check schema.kdl -f json

# check all files in a directory
ssc-gen check examples/

Test schema against HTML

# from file
ssc-gen run examples/booksToScrape.kdl:MainCatalogue -L bs4 -i page.html

# from stdin
curl https://books.toscrape.com/ | ssc-gen run examples/booksToScrape.kdl:MainCatalogue -L bs4

ssc-gen run executes generated Python in-process. Run only trusted schema files.

Health check (verify selectors match elements)

# from file
ssc-gen health examples/booksToScrape.kdl:MainCatalogue -i page.html

# from stdin
curl https://books.toscrape.com/ | ssc-gen health examples/booksToScrape.kdl:MainCatalogue

Documentation

Quick start
Syntax and file structure
Type system
Pipeline operations
Predicates and logic
JSON schemas and jsonify
@request — HTTP constructor + REST clients
LLM-compact reference -- full DSL spec in one file for LLM context
Examples

LLM integration

LLM agents can generate and validate .kdl schemas automatically using the linter feedback loop.

In AI-powered IDEs (Claude Code, Cursor, opencode, etc.)

Project works through agents + skills. Use skills under .agents/skills/:

sscgen-dsl — HTML scraping schemas (css, text, (item)struct, (list)struct, (table)struct, predicates, jsonify).
sscgen-rest — REST/JSON API clients ((rest)struct, @request, @error, typed placeholders).

Development

uv sync                  # install dependencies
uv build --wheel         # build wheel
uv run ruff check ssc_codegen/
uv run mypy ssc_codegen/

Running tests

# Run full test suite
uv run pytest

# Run full test suite in parallel (faster, recommended)
uv run pytest -n auto

# Fast test run: skip external toolchains (Rust cargo, Go, Node.js)
uv run pytest -m "not toolchain"

# Run only toolchain tests (Go, Rust, JavaScript integration)
uv run pytest -m toolchain

Test dependencies

Python and parser tests require only uv sync.

Integration smoke tests for external compilers are skipped automatically if their toolchains are missing:

JavaScript: Requires Node.js in PATH and npm install (jsdom in package.json).
Go: Requires go in PATH.
Rust: Requires cargo in PATH.

Metadata

Release files for ssc_codegen 0.39.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ssc_codegen 0.39.1
File	Size	Uploaded
ssc_codegen-0.39.1.tar.gz	237.4 kB	Oct 2, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for ssc_codegen 0.39.1
File	Interpreter	ABI	Platform	Reset
ssc_codegen-0.39.1-py3-none-any.whl 284.4 kB Oct 2, 2026	Python 3	none	any	Details

Total release size: 521.8 kB

Release files / ssc_codegen-0.39.1.tar.gz

Download URL	ssc_codegen-0.39.1.tar.gz
Size	237.4 kB
Tags	Source
SHA-256 checksum How to use checksums	`02d75d3f1574d933dcd31ff629da22e37f65512d1149d977c6731097f99176c4`
BLAKE2b-256 checksum How to use checksums	`124d77754ab508a17071444ebf2fe39ca013c6fb84b74a47de6d0803ca898154`
Upload date	Oct 2, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.11.14 {"installer":{"name":"uv","version":"0.11.14","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}`

Release files / ssc_codegen-0.39.1-py3-none-any.whl

Download URL	ssc_codegen-0.39.1-py3-none-any.whl
Size	284.4 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`edfc396605dc0cbead352b3045a77996aae96fea35f1d80046c884c6d3765f21`
BLAKE2b-256 checksum How to use checksums	`12214ea21f3558603fd739e1b49bef886624586b5cdd67036caf735650a67d89`
Upload date	Oct 2, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.11.14 {"installer":{"name":"uv","version":"0.11.14","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}`