ssc-codegen
Code generator for web scraping parsers. Describe HTML extraction rules in a declarative KDL 2.0 DSL, then generate ready-to-use parser code for multiple languages and libraries.
.kdl schema --> [kdl parser] --> AST --> [linter] --> [converter] --> output code
Features
- Declarative DSL based on KDL 2.0 syntax
- Static type checking and linting before code generation
- Multiple output targets: Python (bs4, lxml, parsel, selectolax), JavaScript (DOM API), Go (goquery)
- Typed user extensions with lazy target imports and runtime helpers
- Struct types:
item,list,dict,table,flat - LLM-friendly: system prompt + linter loop for AI-assisted schema generation
Install
uv tool install ssc_codegen
Quick example
books.kdl:
(list)struct Book {
@split-doc { css-all ".product-card" }
title { css ".title"; text }
price { css ".price"; text; re #"(\d+\.\d+)"#; to-float }
url { css "a[href]"; attr "href"; fallback #null }
}
Generate Python parser:
ssc-gen generate python books.kdl -L bs4 -o ./output
Usage
Generate code
# single file (Python + bs4)
ssc-gen generate python schema.kdl -L bs4 -o ./output
# all .kdl files in a directory (JavaScript)
ssc-gen generate js examples/ -o ./output
# Go (goquery + net/http)
ssc-gen generate go schema.kdl -o ./output
# Go package defaults to main; override explicitly when generating a library
ssc-gen generate go schema.kdl -o ./output --package scraper
# with custom package name
ssc-gen generate python schema.kdl -L bs4 -o ./parsers --package scraper
# with @request support (REST/HTTP codegen)
ssc-gen generate python schema.kdl -L bs4 -o ./out --http-client httpx
# extract helper functions into a separate runtime module
ssc-gen generate python schema.kdl -L bs4 -o ./out -R
Languages: generate python, generate js, generate go.
HTML libraries (--lib / -L, Python only): bs4 (default), lxml, parsel, slax.
Lint schemas
# human-readable output
ssc-gen check schema.kdl
# JSON output (for LLM pipelines)
ssc-gen check schema.kdl -f json
# check all files in a directory
ssc-gen check examples/
Test schema against HTML
# from file
ssc-gen run examples/booksToScrape.kdl:MainCatalogue -L bs4 -i page.html
# from stdin
curl https://books.toscrape.com/ | ssc-gen run examples/booksToScrape.kdl:MainCatalogue -L bs4
ssc-gen run executes generated Python in-process. Run only trusted schema files.
Health check (verify selectors match elements)
# from file
ssc-gen health examples/booksToScrape.kdl:MainCatalogue -i page.html
# from stdin
curl https://books.toscrape.com/ | ssc-gen health examples/booksToScrape.kdl:MainCatalogue
Documentation
- Quick start
- Syntax and file structure
- Type system
- Pipeline operations
- Predicates and logic
- JSON schemas and jsonify
- @request — HTTP constructor + REST clients
- LLM-compact reference -- full DSL spec in one file for LLM context
- Examples
LLM integration
LLM agents can generate and validate .kdl schemas automatically using the
linter feedback loop.
In AI-powered IDEs (Claude Code, Cursor, opencode, etc.)
Project works through agents + skills. Use skills under .agents/skills/:
sscgen-dsl— HTML scraping schemas (css,text,(item)struct,(list)struct,(table)struct, predicates,jsonify).sscgen-rest— REST/JSON API clients ((rest)struct,@request,@error, typed placeholders).
Development
uv sync # install dependencies
uv build --wheel # build wheel
uv run ruff check ssc_codegen/
uv run mypy ssc_codegen/
Running tests
# Run full test suite
uv run pytest
# Run full test suite in parallel (faster, recommended)
uv run pytest -n auto
# Fast test run: skip external toolchains (Rust cargo, Go, Node.js)
uv run pytest -m "not toolchain"
# Run only toolchain tests (Go, Rust, JavaScript integration)
uv run pytest -m toolchain
Test dependencies
Python and parser tests require only uv sync.
Integration smoke tests for external compilers are skipped automatically if their toolchains are missing:
- JavaScript: Requires Node.js in PATH and
npm install(jsdominpackage.json). - Go: Requires
goin PATH. - Rust: Requires
cargoin PATH.
1 maintainer
Metadata
1 maintainer
Release files for ssc_codegen 0.43.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| ssc_codegen-0.43.0.tar.gz | 249.1 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| ssc_codegen-0.43.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 546.0 kB
Release files / ssc_codegen-0.43.0.tar.gz
| Download URL | ssc_codegen-0.43.0.tar.gz |
|---|---|
| Size | 249.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d28ec7ea799060572d1ba90f1e0c6dc69da9d12e603e08f86627d59125270193
|
|
BLAKE2b-256 checksum How to use checksums |
56d04a94441456f388d81d431d8a5eebdc4fc90b5854671d9a4dc92124459feb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.14 {"installer":{"name":"uv","version":"0.11.14","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / ssc_codegen-0.43.0-py3-none-any.whl
| Download URL | ssc_codegen-0.43.0-py3-none-any.whl |
|---|---|
| Size | 296.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
3c72c2bfd2359229bd8321c414a61535dd092d1a1f61469f1dc8c22d8e81aeb8
|
|
BLAKE2b-256 checksum How to use checksums |
ce3f910ec59ec67a11420f06fc946318b9183fb321ad9d519359f63a976c6ec9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.14 {"installer":{"name":"uv","version":"0.11.14","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for ssc_codegen 0.43.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
