ssc_codegen 0.31.0

Python-dsl code converter to html parser for web scraping

ssc-codegen

Code generator for web scraping parsers. Describe HTML extraction rules in a declarative KDL 2.0 DSL, then generate ready-to-use parser code for multiple languages and libraries.

.kdl schema --> [kdl parser] --> AST --> [linter] --> [converter] --> output code

Features

• Declarative DSL based on KDL 2.0 syntax
• Static type checking and linting before code generation
• Multiple output targets: Python (bs4, lxml, parsel, selectolax), JavaScript (DOM API)
• Struct types: item, list, dict, table, flat
• LLM-friendly: system prompt + linter loop for AI-assisted schema generation

Install

uv tool install ssc_codegen

Quick example

books.kdl:

(list)struct Book {
    @split-doc { css-all ".product-card" }

    title { css ".title"; text }
    price { css ".price"; text; re #"(\d+\.\d+)"#; to-float }
    url   { css "a[href]"; attr "href"; fallback #null }
}

Generate Python parser:

ssc-gen generate books.kdl -t py-bs4 -o ./output

Usage

Generate code

# single file
ssc-gen generate schema.kdl -t py-bs4 -o ./output

# all .kdl files in a directory
ssc-gen generate examples/ -t js-pure -o ./output

# with custom package name
ssc-gen generate schema.kdl -t py-bs4 -o ./parsers --package scraper

Targets: py-bs4, py-lxml, py-parsel, py-slax, js-pure

Lint schemas

# human-readable output
ssc-gen check schema.kdl

# JSON output (for LLM pipelines)
ssc-gen check schema.kdl -f json

# check all files in a directory
ssc-gen check examples/

Test schema against HTML

# from file
ssc-gen run examples/booksToScrape.kdl:MainCatalogue -t py-bs4 -i page.html

# from stdin
curl https://books.toscrape.com/ | ssc-gen run examples/booksToScrape.kdl:MainCatalogue -t py-bs4

Health check (verify selectors match elements)

# from file
ssc-gen health examples/booksToScrape.kdl:MainCatalogue -i page.html

# from stdin
curl https://books.toscrape.com/ | ssc-gen health examples/booksToScrape.kdl:MainCatalogue

Documentation

• Quick start
• Syntax and file structure
• Type system
• Pipeline operations
• Predicates and logic
• JSON schemas and jsonify
• Transforms and dsl blocks
• LLM-compact reference -- full DSL spec in one file for LLM context
• Examples

LLM integration

LLM agents can generate and validate .kdl schemas automatically using the linter feedback loop.

In chats (ChatGPT, Claude, etc.)

Use SYSTEM_PROMPT.md as system prompt. After generation, run ssc-gen check -f json and send errors back to the LLM for correction.

In AI-powered IDEs (Claude Code, Cursor, etc.)

Use the kdl-schema-dsl skill for automatic generation, validation, and iteration.

Development

uv sync                  # install dependencies
uv build --wheel         # build wheel
uv run pytest            # run tests
uv run ruff check ssc_codegen/

Test dependencies

Python tests require only uv sync. JS integration tests additionally need:

npm install      # installs jsdom (dev dependency in package.json)

Node.js must be installed and available as node in PATH. JS tests are automatically skipped if Node.js is not found.