asciidoctrine 0.2.0a4

= AsciiDoctrine
Michael R. Bernstein <zopemaven@gmail.com>
v0.1.0
:toc: left
:icons: font
:idprefix:
:idseparator: -
:sectanchors:
:sectlinks:
:source-highlighter: highlight.js

image:https://img.shields.io/pypi/v/asciidoctrine.svg[PyPI Version, link=https://pypi.org/project/asciidoctrine/]
image:https://img.shields.io/badge/License-Apache_2.0-blue.svg[License, link=https://opensource.org/licenses/Apache-2.0]
image:https://img.shields.io/badge/python-3.10--3.14-blue.svg[Python Version]
image:https://img.shields.io/badge/coverage-82%25-green.svg[Coverage, link=https://github.com/webmaven/asciidoctrine/actions/workflows/ci.yml]
image:https://img.shields.io/badge/status-alpha-orange.svg[Development Status]
image:https://img.shields.io/badge/docs-GitHub_Pages-blue[Documentation, link=https://webmaven.github.io/asciidoctrine/]

A high-performance, pure-Python AsciiDoc parser built with Lark, designed for compatibility with the official AsciiDoc specification and the TCK. 

* *Documentation*: link:https://webmaven.github.io/asciidoctrine/[https://webmaven.github.io/asciidoctrine/]
* *PyPI Package*: link:https://pypi.org/project/asciidoctrine/[https://pypi.org/project/asciidoctrine/]

== Motivation

The Python ecosystem has long lacked a modern, maintainable, and specification-compliant AsciiDoc parser. Existing tools are often port-based or rely on regex-heavy implementations that struggle with the complex, context-sensitive nature of AsciiDoc.

AsciiDoctrine is built from the ground up to provide:

1. **Spec Alignment**: Strict adherence to the upcoming official AsciiDoc Language Specification.
2. **First-Class AST**: A structured, type-safe Abstract Syntax Tree that makes building renderers and static analysis tools a breeze.
3. **Performance**: Leveraging the Lark parsing engine for efficient processing of large documents.

== Architecture

The parser operates in a multi-pass pipeline to handle the inherent complexity of AsciiDoc:

[source,mermaid]
....
graph LR
    A[Source] --> B(Lark Parser)
    B --> C[Concrete Syntax Tree]
    C --> D(Transformer)
    D --> E[Structured AST]
    E --> F(Semantic Passes)
    F --> G[Resolved ASG]
....

* **AST (Abstract Syntax Tree)**: Represented in `nodes.py`, this is a structural tree of the document elements.
* **ASG (Abstract Semantic Graph)**: The final resolved state where attributes, cross-references, and includes are fully processed.

== Technical Choices

*   **Lark Parsing Engine**: We use Lark because it supports multiple parsing algorithms (Earley, LALR) and has an experimental PEG mode. This allow us to handle the context-sensitive nature of AsciiDoc without the maintenance nightmare of large regex collections.
*   **Two-Pass Pipeline**: Handling attributes and includes requires knowing the state of the whole document. Our multi-pass approach ensures that we can resolve semantic details (like cross-references) correctly.
*   **Pure Python**: Zero C-extensions means easy installation on all platforms including Pyodide and WebAssembly.

== Installation & Setup

End-users can install the official, stable release of AsciiDoctrine from link:https://pypi.org/project/asciidoctrine/[PyPI]:

[source,bash]
----
pip install asciidoctrine
----

For developers and contributors, the recommended "happy path" is to set up a local virtual environment and install AsciiDoctrine in editable development mode:

[source,bash]
----
# 1. Clone the repository
git clone https://github.com/webmaven/asciidoctrine.git
cd asciidoctrine

# 2. Create and activate a Python virtual environment
python3 -m venv venv
source venv/bin/activate

# 3. Install in development mode with test and documentation dependencies
pip install -e ".[test,docs]"
----

> [!NOTE]
> This package is currently under active development in tandem with the official TCK integration.

== Quick Start

AsciiDoctrine implements a two-pass resolution pipeline. First, parse the raw source to a syntax-level AST. Then, resolve the AST to a semantic, queryable Abstract Semantic Graph (ASG):

[source,python]
----
from asciidoctrine import parse_to_ast
from asciidoctrine.resolver import ASGResolver

source = """
== Section Title
This is a *bold* word in a paragraph.
"""

# 1. Parse raw source to syntax-level AST
ast = parse_to_ast(source)

# 2. Resolve to semantic ASG (resolves attributes, includes, and filters comments)
resolver = ASGResolver(ast)
asg = resolver.resolve(ast)

# Iterate through sections in the ASG blocks list
for block in asg.get("blocks", []):
    if block.get("name") == "section":
        # The title is a list of inline nodes in the ASG schema
        title_nodes = block.get("title", [])
        title_text = "".join(node.get("value", "") for node in title_nodes if node.get("name") == "text")
        print(f"Found section: {title_text}")
----

=== ASG Representation

Calling `asg.to_dict()` yields a structured, spec-compliant representation matching the official AsciiDoc Language ASG schema:

[source,json]
----
{
  "name": "document",
  "type": "block",
  "blocks": [
    {
      "name": "section",
      "type": "block",
      "level": 1,
      "title": {
        "name": "title",
        "type": "inline",
        "inlines": [
          { "name": "text", "type": "string", "value": "Section Title" }
        ]
      },
      "blocks": [
        {
          "name": "paragraph",
          "type": "block",
          "inlines": [
            { "name": "text", "type": "string", "value": "This is a " },
            {
              "name": "span",
              "type": "inline",
              "variant": "strong",
              "form": "constrained",
              "inlines": [
                { "name": "text", "type": "string", "value": "bold" }
              ]
            },
            { "name": "text", "type": "string", "value": " word in a paragraph." }
          ]
        }
      ]
    }
  ],
  "attributes": {}
}
----

== Roadmap to Syntax Parity

The path to 1:1 syntactic specification compliance is tracked through the following phases:

[cols="1,3,1"]
|===
| Phase | Focus | Status

| **0** | **Foundations**: PEG/Earley grammar, structured AST, and TCK test harness. | ✅
| **1** | **Advanced Blocks**: Admonitions ✅, Sidebars ✅, Source blocks ✅, Example blocks ✅, and Open blocks ✅. | ✅
| **2** | **Document Infra**: Headers ✅, dynamic attribute resolution ✅, and multi-file includes ✅. | ✅
| **3** | **Tables & Description Lists**: Nested/mixed description lists, checklists ✅, and advanced table formatting/spans ✅. | ✅
| **4** | **Testing & Developer Experience**: Callout stripping, strict AST syntax auditing ✅, and precise source location coordinate tracking ✅. | ✅
| **5** | **Structural Resolution & SSG**: TOC outline extraction ✅, cross-references (`xref`) ✅, and bare URL/email autolinks ✅. | ✅
| **6** | **Sphinx Extension Support**: 100% Docutils node visitor coverage ✅, metadata alignment ✅, and permissive error recovery ✅. | ✅
| **7** | **Spec & TCK Conformance**: Authoring language specification modules, ASG schema convergence, and upstream TCK contributions. | 🔄
|===

== Project Structure

[source,text]
----
asciidoctrine/
├── src/
│   └── asciidoctrine/      # Core parser logic
│       ├── grammar.lark      # EBNF Grammar
│       ├── lark_parser.py    # Transformer and Parser entry point
│       ├── nodes.py          # AST Node definitions
│       └── __init__.py       # Public API
├── examples/                 # Real-world usage samples
├── tests/                    # Unit and integration tests
├── pyproject.toml            # Build configuration
└── README.adoc               # This file
----

== Testing & Compliance

We prioritize correctness by testing against three fronts:
1. **Unit & Integration Tests**: Granular tests for grammar and semantic components.
2. **External Corpus (DocTest)**: Real-world examples from the `asciidoctor-doctest` corpus.
3. **TCK (Technology Compatibility Kit)**: Direct compliance with the official AsciiDoc Language TCK suite.

Run the stable Pytest suite locally with:
[source,bash]
----
pytest -k "not functional"
----

Run the official TCK test suite with:
[source,bash]
----
./run-tck.sh
----

== Self-Hosted Documentation

This project's documentation is authored entirely in AsciiDoc, compiled using Sphinx along with the dedicated `sphinx_asciidoctrine` plugin, and published online at link:https://webmaven.github.io/asciidoctrine/[https://webmaven.github.io/asciidoctrine/]!

Build and view the documentation locally:
[source,bash]
----
# Compile HTML documentation
sphinx-build -b html docs/ docs/_build/html

# Open in your browser (macOS example)
open docs/_build/html/index.html
----

== Contributing

We welcome contributions! Please see link:CONTRIBUTING.adoc[CONTRIBUTING.adoc] for detailed development workflow guidelines, TCK compliance processes, and code style.

== License

Distributed under the **Apache License 2.0**. See link:LICENSE[LICENSE] for details.