ssc-codegen
Code generator for web scraping parsers. Describe HTML extraction rules in a declarative KDL 2.0 DSL, then generate ready-to-use parser code for multiple languages and libraries.
.kdl schema --> [kdl parser] --> AST --> [linter] --> [converter] --> output code
Features
- Declarative DSL based on KDL 2.0 syntax
- Static type checking and linting before code generation
- Multiple output targets: Python (bs4, lxml, parsel, selectolax), JavaScript (DOM API)
- Struct types:
item,list,dict,table,flat - LLM-friendly: system prompt + linter loop for AI-assisted schema generation
Install
uv tool install ssc_codegen
Quick example
books.kdl:
(list)struct Book {
@split-doc { css-all ".product-card" }
title { css ".title"; text }
price { css ".price"; text; re #"(\d+\.\d+)"#; to-float }
url { css "a[href]"; attr "href"; fallback #null }
}
Generate Python parser:
ssc-gen generate python books.kdl -L bs4 -o ./output
Usage
Generate code
# single file (Python + bs4)
ssc-gen generate python schema.kdl -L bs4 -o ./output
# all .kdl files in a directory (JavaScript)
ssc-gen generate js examples/ -o ./output
# Go (goquery + net/http)
ssc-gen generate go schema.kdl -o ./output
# Go package defaults to main; override explicitly when generating a library
ssc-gen generate go schema.kdl -o ./output --package scraper
# with custom package name
ssc-gen generate python schema.kdl -L bs4 -o ./parsers --package scraper
# with @request support (REST/HTTP codegen)
ssc-gen generate python schema.kdl -L bs4 -o ./out --http-client httpx
# extract helper functions into a separate runtime module
ssc-gen generate python schema.kdl -L bs4 -o ./out -R
Languages: generate python, generate js, generate go.
HTML libraries (--lib / -L, Python only): bs4 (default), lxml, parsel, slax.
Lint schemas
# human-readable output
ssc-gen check schema.kdl
# JSON output (for LLM pipelines)
ssc-gen check schema.kdl -f json
# check all files in a directory
ssc-gen check examples/
Test schema against HTML
# from file
ssc-gen run examples/booksToScrape.kdl:MainCatalogue -L bs4 -i page.html
# from stdin
curl https://books.toscrape.com/ | ssc-gen run examples/booksToScrape.kdl:MainCatalogue -L bs4
ssc-gen run executes generated Python in-process. Run only trusted schema files.
Health check (verify selectors match elements)
# from file
ssc-gen health examples/booksToScrape.kdl:MainCatalogue -i page.html
# from stdin
curl https://books.toscrape.com/ | ssc-gen health examples/booksToScrape.kdl:MainCatalogue
Documentation
- Quick start
- Syntax and file structure
- Type system
- Pipeline operations
- Predicates and logic
- JSON schemas and jsonify
- @request — HTTP constructor + REST clients
- LLM-compact reference -- full DSL spec in one file for LLM context
- Examples
LLM integration
LLM agents can generate and validate .kdl schemas automatically using the
linter feedback loop.
In chats (ChatGPT, Claude, etc.)
Use SYSTEM_PROMPT.md as system prompt. After generation,
run ssc-gen check -f json and send errors back to the LLM for correction.
In AI-powered IDEs (Claude Code, Cursor, opencode, etc.)
Use the skills under .agents/skills/:
sscgen-dsl— HTML scraping schemas (css,text,(item)struct,(list)struct,(table)struct, predicates,jsonify).sscgen-rest— REST/JSON API clients ((rest)struct,@request,@error, typed placeholders).sscgen-openapi— convert OpenAPI/Swagger specs into.kdl.
Development
uv sync # install dependencies
uv build --wheel # build wheel
uv run pytest # run tests
uv run ruff check ssc_codegen/
Test dependencies
Python tests require only uv sync. JS integration tests additionally need:
npm install # installs jsdom (dev dependency in package.json)
Node.js must be installed and available as node in PATH. JS tests are automatically skipped if Node.js is not found.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file ssc_codegen-0.37.2.tar.gz.
File metadata
- Download URL: ssc_codegen-0.37.2.tar.gz
- Upload date:
- Size: 157.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.11.14 {"installer":{"name":"uv","version":"0.11.14","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
f3f5462a340bf92fc40304751ab9de13054f371255ecd8764b649772411ee5a8
|
|
| MD5 |
7d28e5e39bf3f756c493c67794e610b0
|
|
| BLAKE2b-256 |
0c219deb14439f2add2ecfb9427a1b18486e0f2ee78abe5e43f9828f8bf8e6f2
|
File details
Details for the file ssc_codegen-0.37.2-py3-none-any.whl.
File metadata
- Download URL: ssc_codegen-0.37.2-py3-none-any.whl
- Upload date:
- Size: 194.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.11.14 {"installer":{"name":"uv","version":"0.11.14","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
acb324d83b4d2b03fd66bdb9cb1816f8203242a8c9e88bb0d10e0dab98224e29
|
|
| MD5 |
9c8af652a7de16e6ba6731d4a5697863
|
|
| BLAKE2b-256 |
67daeb8d4e57140f73d819f5dee5433a433a8709c5597462ac6e9645f8b91cae
|
