ssc-codegen
Code generator for web scraping parsers. Describe HTML extraction rules in a declarative KDL 2.0 DSL, then generate ready-to-use parser code for multiple languages and libraries.
.kdl schema --> [kdl parser] --> AST --> [linter] --> [converter] --> output code
Features
- Declarative DSL based on KDL 2.0 syntax
- Static type checking and linting before code generation
- Multiple output targets: Python (bs4, lxml, parsel, selectolax), JavaScript (DOM API), Go (goquery)
- Typed user extensions with lazy target imports and runtime helpers
- Struct types:
item,list,dict,table,flat - LLM-friendly: system prompt + linter loop for AI-assisted schema generation
Install
uv tool install ssc_codegen
Quick example
books.kdl:
(list)struct Book {
@split-doc { css-all ".product-card" }
title { css ".title"; text }
price { css ".price"; text; re #"(\d+\.\d+)"#; to-float }
url { css "a[href]"; attr "href"; fallback #null }
}
Generate Python parser:
ssc-gen generate python books.kdl -L bs4 -o ./output
Usage
Generate code
# single file (Python + bs4)
ssc-gen generate python schema.kdl -L bs4 -o ./output
# all .kdl files in a directory (JavaScript)
ssc-gen generate js examples/ -o ./output
# Go (goquery + net/http)
ssc-gen generate go schema.kdl -o ./output
# Go package defaults to main; override explicitly when generating a library
ssc-gen generate go schema.kdl -o ./output --package scraper
# with custom package name
ssc-gen generate python schema.kdl -L bs4 -o ./parsers --package scraper
# with @request support (REST/HTTP codegen)
ssc-gen generate python schema.kdl -L bs4 -o ./out --http-client httpx
# extract helper functions into a separate runtime module
ssc-gen generate python schema.kdl -L bs4 -o ./out -R
Languages: generate python, generate js, generate go.
HTML libraries (--lib / -L, Python only): bs4 (default), lxml, parsel, slax.
Lint schemas
# human-readable output
ssc-gen check schema.kdl
# JSON output (for LLM pipelines)
ssc-gen check schema.kdl -f json
# check all files in a directory
ssc-gen check examples/
Test schema against HTML
# from file
ssc-gen run examples/booksToScrape.kdl:MainCatalogue -L bs4 -i page.html
# from stdin
curl https://books.toscrape.com/ | ssc-gen run examples/booksToScrape.kdl:MainCatalogue -L bs4
ssc-gen run executes generated Python in-process. Run only trusted schema files.
Health check (verify selectors match elements)
# from file
ssc-gen health examples/booksToScrape.kdl:MainCatalogue -i page.html
# from stdin
curl https://books.toscrape.com/ | ssc-gen health examples/booksToScrape.kdl:MainCatalogue
Documentation
- Quick start
- Syntax and file structure
- Type system
- Pipeline operations
- Predicates and logic
- JSON schemas and jsonify
- @request — HTTP constructor + REST clients
- LLM-compact reference -- full DSL spec in one file for LLM context
- Examples
LLM integration
LLM agents can generate and validate .kdl schemas automatically using the
linter feedback loop.
In AI-powered IDEs (Claude Code, Cursor, opencode, etc.)
Project works through agents + skills. Use skills under .agents/skills/:
sscgen-dsl— HTML scraping schemas (css,text,(item)struct,(list)struct,(table)struct, predicates,jsonify).sscgen-rest— REST/JSON API clients ((rest)struct,@request,@error, typed placeholders).
Development
uv sync # install dependencies
uv build --wheel # build wheel
uv run ruff check ssc_codegen/
uv run mypy ssc_codegen/
Running tests
# Run full test suite
uv run pytest
# Run full test suite in parallel (faster, recommended)
uv run pytest -n auto
# Fast test run: skip external toolchains (Rust cargo, Go, Node.js)
uv run pytest -m "not toolchain"
# Run only toolchain tests (Go, Rust, JavaScript integration)
uv run pytest -m toolchain
Test dependencies
Python and parser tests require only uv sync.
Integration smoke tests for external compilers are skipped automatically if their toolchains are missing:
- JavaScript: Requires Node.js in PATH and
npm install(jsdominpackage.json). - Go: Requires
goin PATH. - Rust: Requires
cargoin PATH.
1 maintainer
Metadata
1 maintainer
Release files for ssc_codegen 0.40.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| ssc_codegen-0.40.0.tar.gz | 239.9 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| ssc_codegen-0.40.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 527.0 kB
Release files / ssc_codegen-0.40.0.tar.gz
| Download URL | ssc_codegen-0.40.0.tar.gz |
|---|---|
| Size | 239.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
bcddc2e67cab5956dfc003ca16d9a66dec6701b0ff06053d51d55ed4ddb59c1c
|
|
BLAKE2b-256 checksum How to use checksums |
3813df6754592244a206b4dc81cc4b25ddee4edb159128839d5e2b31b54c8a78
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.14 {"installer":{"name":"uv","version":"0.11.14","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / ssc_codegen-0.40.0-py3-none-any.whl
| Download URL | ssc_codegen-0.40.0-py3-none-any.whl |
|---|---|
| Size | 287.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
47091f50a25f3a1f8d8478e2206674ce924943efc985762b98ba363c53d609d3
|
|
BLAKE2b-256 checksum How to use checksums |
3351f95dd98f2bf42a1a340bca3961381afaffd161c5b21dd7b00bfe8cf00bdd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.14 {"installer":{"name":"uv","version":"0.11.14","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
