pyregtab 0.2.0

pyRegTab: pattern-based extraction of recordsets from tables (RTL / ATP / ITM)

pyRegTab

RegTab: pattern-driven data extraction from document tables with regular structure — the Python port of jRegTab with a native Rust core.

pyRegTab compiles RTL (Regular Table Language) patterns into abstract table patterns (ATP), matches them against a table's syntactic layer (ITM), and interprets the match into a relational recordset:

TableSyntax → RtlCompiler/TablePattern → AtpMatcher → TableInterpreter → Recordset

pyRegTab 0.1.x ≙ jRegTab 0.4.0 (same API, same semantics, same test corpus). 0.2.0 adds the embedded RTL DSL (pyregtab.dsl), mirroring jRegTab's ru.icc.regtab.dsl.

Installation

pip install pyregtab

Binary wheels are published for Windows, Linux and macOS (x86-64 / arm64), CPython ≥ 3.10 (one abi3 wheel per platform). Building from the sdist requires a Rust toolchain.

Example

from pyregtab import TableSyntax, RtlCompiler, AtpMatcher, TableInterpreter

syntax = TableSyntax(3, 3)
syntax.cell(0, 1).set_text("CA");  syntax.cell(0, 2).set_text("HU")
syntax.cell(1, 0).set_text("IKT"); syntax.cell(1, 1).set_text("0 Jan"); syntax.cell(1, 2).set_text("8 Feb")
syntax.cell(2, 0).set_text("SVO"); syntax.cell(2, 1).set_text("31 Jan"); syntax.cell(2, 2).set_text("40 Feb")

pattern = RtlCompiler.compile("""
    [ [] [VAL : 'AIRLINE'->AVP]+ ]
    [ [VAL : 'AIRPORT'->AVP]
      [VAL : (COL, ROW, CL)->REC, 'ND'->AVP " " VAL : 'MON'->AVP]+ ]+
""")

itm = AtpMatcher.match(pattern, syntax)     # InterpretableTable | None
rs = TableInterpreter().interpret(itm)      # Recordset
rs.schema.attributes                        # ['ND', 'AIRLINE', 'AIRPORT', 'MON']
rs[0]["ND"]                                 # '0'
df = rs.to_pandas()                         # extras: pip install pyregtab[pandas]

Patterns can also be built without RTL, via the fluent spec API (TablePattern.of(SubtablePattern.of(...)) — same factories as in Java, snake_case method names), and serialized back to RTL with AtpToRtlSerializer.serialize(pattern).

For a terser, RTL-like way to build patterns in code, use the embedded RTL DSL (pyregtab.dsl) — see Embedded RTL below.

Named Python predicates are attached to RTL via EXT('name'):

from pyregtab import Bindings

p = RtlCompiler.compile(
    "{ [ [EXT('isTotal') ? VAL : ST*->REC] []+ ] }+",
    Bindings.of().cell("isTotal", lambda cell: cell.text.startswith("Total")),
)

Embedded RTL

The pyregtab.dsl module is a fluent DSL that reads almost like RTL but is ordinary Python — with IDE completion, structural typing, pattern composition via plain variables, and Python callables as escape-hatch constraints. It builds the same TablePattern objects as the compiler (verified byte-for-byte against RtlCompiler.compile for a representative set of tasks in tests/test_dsl.py).

from pyregtab.dsl import *

# RTL: { [ [VAL : ST*->REC] [VAL]{2} []+ ]
#        [ []               [VAL]{4} []+ ] }+
p = table(
    subtable(
        row(cell(VAL, rec(ST.unbounded())), cell(VAL).exactly(2), skip().one_or_more()),
        row(skip(),                         cell(VAL).exactly(4), skip().one_or_more()),
    ).one_or_more())

Method names are snake_case (.one_or_more(), .and_(), .split_by()); the vocabulary constants (VAL, ST, COL, C(n), …) match RTL. See the Embedded RTL guide for the full mapping and the where(...) escape hatch.

API mapping (Java → Python)

Java	Python
RtlCompiler.compile(String)	RtlCompiler.compile(str) / pyregtab.compile(...)
AtpMatcher.match(p, s) → Optional<InterpretableTable>	AtpMatcher.match(p, s) → InterpretableTable | None
Quantifier.oneOrMore()	Quantifier.one_or_more()
new TableInterpreter().withStrategy(s).interpret(itm)	TableInterpreter().with_strategy(s).interpret(itm)
rs.records().get(0).get("Name")	rs[0]["Name"], rs.records, record.get("Name")
cell.text() / cell.setText(t)	property cell.text (get/set); cell.set_text(t) also works
RtlCompileException	RtlCompileError

Architecture

Everything after the Python call boundary runs in a native core written in Rust (pyregtab._core, built with PyO3 and maturin); the Python layer is a thin re-export.

• grammar/RTL.g4 — the normative specification of the RTL language (a verbatim copy from jRegTab; the upstream commit and the grammar's SHA-256 are recorded in grammar/UPSTREAM). The core's parser is a hand-written lexer + recursive descent that structurally follows the grammar rules. A CI job (tools/check_grammar_sync.py) fails the build if the copy drifts from the pinned hash, and — when a jRegTab read token is available — cross-checks it byte-for-byte against the upstream commit.
• conformance/ — the shared RTL conformance corpus (also pinned from jRegTab, see conformance/UPSTREAM and conformance/README.md). Both implementations must compile every positive case to the same canonical form and reject every negative case; the corpus runs in CI of both projects. Any RTL language change flows: RTL.g4 in jregtab → corpus extension → both parsers → green corpus in both CIs.
• Regular expressions in RTL constraints are executed by the Rust regex crate (linear-time). The reference fixture corpus uses no lookaround/backreferences (audited), so the dialect is compatible with java.util.regex on this corpus. Documented divergences from Java: \d/\s/\w are Unicode-aware in regex (ASCII in Java), and SUBSTR indices count code points (UTF-16 units in Java) — identical behavior on the entire reference corpus.

Testing

pytest tests runs (1 904 tests):

• the full benchmark suite — tasks 001–150 (Foofah, RegTab, Baikal), every fixture variant, both via RTL patterns and via ATP patterns built with the Python spec API (1 500 task variants in total; fixtures are copied verbatim from jRegTab into tests/fixtures/tasks, ATP builders are mechanically translated from the Java tests by tools/translate_atp.py);
• embedded RTL DSL parity — 26 representative tasks/constructs built with pyregtab.dsl produce byte-identical ATP to RtlCompiler.compile (tests/test_dsl.py);
• the RTL conformance corpus (positive canonical forms, fixed points, negative rejections);
• RTL↔ATP round-trip for tasks 001–050;
• API unit tests (syntax layer, extractors, EXT bindings, custom predicates, transformations, interpreter options, GIL-released batch matching from a thread pool).

cargo test additionally runs the conformance corpus and an end-to-end smoke test against the native core alone. Differential testing against the Java reference (tools/differential.py + tools/RecordsetDumpMain.java) compares recordsets cell-by-cell on all 750 task variants — zero mismatches against jRegTab v0.4.0.

IDE support

ide/vscode/ is a VS Code extension (and IntelliJ/PyCharm TextMate bundle) that highlights .rtl files and RTL embedded in Python strings passed to RtlCompiler.compile(...). See ide/README.md. RTL is also validated at compile time: RtlCompiler.compile(...) raises RtlCompileError with a line:col position on an invalid pattern.

Development

python -m venv .venv && . .venv/bin/activate   # or .venv\Scripts\activate
pip install maturin pytest
maturin develop --release
pytest tests -q

License

MIT