ssc-codegen
Code generator for web scraping parsers. Describe HTML extraction rules in a declarative KDL 2.0 DSL, then generate ready-to-use parser code for multiple languages and libraries.
.kdl schema --> [kdl parser] --> AST --> [linter] --> [converter] --> output code
Features
- Declarative DSL based on KDL 2.0 syntax
- Static type checking and linting before code generation
- Multiple output targets: Python (bs4, lxml, parsel, selectolax), JavaScript (DOM API), Go (goquery)
- Typed user extensions with lazy target imports and runtime helpers
- Struct types:
item,list,dict,table,flat - LLM-friendly: system prompt + linter loop for AI-assisted schema generation
Install
uv tool install ssc_codegen
Quick example
books.kdl:
(list)struct Book {
@split-doc { css-all ".product-card" }
title { css ".title"; text }
price { css ".price"; text; re #"(\d+\.\d+)"#; to-float }
url { css "a[href]"; attr "href"; fallback #null }
}
Generate Python parser:
ssc-gen generate python books.kdl -L bs4 -o ./output
Usage
Generate code
# single file (Python + bs4)
ssc-gen generate python schema.kdl -L bs4 -o ./output
# all .kdl files in a directory (JavaScript)
ssc-gen generate js examples/ -o ./output
# Go (goquery + net/http)
ssc-gen generate go schema.kdl -o ./output
# Go package defaults to main; override explicitly when generating a library
ssc-gen generate go schema.kdl -o ./output --package scraper
# with custom package name
ssc-gen generate python schema.kdl -L bs4 -o ./parsers --package scraper
# with @request support (REST/HTTP codegen)
ssc-gen generate python schema.kdl -L bs4 -o ./out --http-client httpx
# extract helper functions into a separate runtime module
ssc-gen generate python schema.kdl -L bs4 -o ./out -R
Languages: generate python, generate js, generate go.
HTML libraries (--lib / -L, Python only): bs4 (default), lxml, parsel, slax.
Lint schemas
# human-readable output
ssc-gen check schema.kdl
# JSON output (for LLM pipelines)
ssc-gen check schema.kdl -f json
# check all files in a directory
ssc-gen check examples/
Test schema against HTML
# from file
ssc-gen run examples/booksToScrape.kdl:MainCatalogue -L bs4 -i page.html
# from stdin
curl https://books.toscrape.com/ | ssc-gen run examples/booksToScrape.kdl:MainCatalogue -L bs4
ssc-gen run executes generated Python in-process. Run only trusted schema files.
Health check (verify selectors match elements)
# from file
ssc-gen health examples/booksToScrape.kdl:MainCatalogue -i page.html
# from stdin
curl https://books.toscrape.com/ | ssc-gen health examples/booksToScrape.kdl:MainCatalogue
Documentation
- Quick start
- Syntax and file structure
- Type system
- Pipeline operations
- Predicates and logic
- JSON schemas and jsonify
- @request — HTTP constructor + REST clients
- LLM-compact reference -- full DSL spec in one file for LLM context
- Examples
LLM integration
LLM agents can generate and validate .kdl schemas automatically using the
linter feedback loop.
In AI-powered IDEs (Claude Code, Cursor, opencode, etc.)
Project works through agents + skills. Use skills under .agents/skills/:
sscgen-dsl— HTML scraping schemas (css,text,(item)struct,(list)struct,(table)struct, predicates,jsonify).sscgen-rest— REST/JSON API clients ((rest)struct,@request,@error, typed placeholders).
Development
uv sync # install dependencies
uv build --wheel # build wheel
uv run ruff check ssc_codegen/
uv run mypy ssc_codegen/
Running tests
# Run full test suite
uv run pytest
# Run full test suite in parallel (faster, recommended)
uv run pytest -n auto
# Fast test run: skip external toolchains (Rust cargo, Go, Node.js)
uv run pytest -m "not toolchain"
# Run only toolchain tests (Go, Rust, JavaScript integration)
uv run pytest -m toolchain
Test dependencies
Python and parser tests require only uv sync.
Integration smoke tests for external compilers are skipped automatically if their toolchains are missing:
- JavaScript: Requires Node.js in PATH and
npm install(jsdominpackage.json). - Go: Requires
goin PATH. - Rust: Requires
cargoin PATH.
1 maintainer
Metadata
1 maintainer
Release files for ssc_codegen 0.44.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| ssc_codegen-0.44.0.tar.gz | 246.7 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| ssc_codegen-0.44.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 541.1 kB
Release files / ssc_codegen-0.44.0.tar.gz
| Download URL | ssc_codegen-0.44.0.tar.gz |
|---|---|
| Size | 246.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
90e53ccd414508fe82a07452e6b3eafdbc8e49be38efa6e69f17083492886200
|
|
BLAKE2b-256 checksum How to use checksums |
d2d1f2889be0a1be7a01f61080fdc377055ecb752cff1ce280d050b05f08995f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.14 {"installer":{"name":"uv","version":"0.11.14","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / ssc_codegen-0.44.0-py3-none-any.whl
| Download URL | ssc_codegen-0.44.0-py3-none-any.whl |
|---|---|
| Size | 294.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
d8a5d0d242087718614861d7d2fbde6c8d73b8a80d1ed376e824f3d05f87cf40
|
|
BLAKE2b-256 checksum How to use checksums |
5ef2f15ac4fc7ef33037a8986753b242d12e293efb64a36a0c94ffda4441c5a3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.14 {"installer":{"name":"uv","version":"0.11.14","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for ssc_codegen 0.44.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
