pyyudit

Cross-platform Python binding to Yudit's Unicode shaping engine, with per-step trace output.

What is this?

pyyudit extracts the shaping engine from the Yudit Unicode editor into a standalone shared library and wraps it with Python ctypes bindings. It provides:

• Indic script shaping — Devanagari, Bengali, Gujarati, Gurmukhi, Oriya, Telugu, Kannada, Malayalam, Sinhala
• Arabic / Syriac joining — isolated, initial, medial and final forms, plus required ligatures (lam-alef, Allah)
• Hebrew, Thaana, N'Ko, Mandaic, Mongolian — right-to-left ordering and joining
• Thai, Lao, Tibetan text processing
• Hangul Jamo processed through ccmp/ljmo/vjmo/tjmo (no Unicode precomposition)
• OpenType GSUB/GPOS feature application, including Extension lookups (GSUB type 7 / GPOS type 9) used by Arial, Segoe UI, Noto, …
• BiDi (bidirectional text) base-direction detection and RTL visual ordering
• Per-step trace — see every decompose → reorder → gsub → gpos step

How Yudit shapes text

Yudit has two shaping layers, and pyyudit routes text through the right one:

Both paths share the same OpenType engine. Mark positioning (mark, mkmk) and required/contextual ligatures (rlig, calt) are what Yudit implements; pair kerning (kern) is not one of them - see Limitations.

Architecture

pyyudit (Python)
├── YuditFont          — load font from bytes/path
│   ├── shape()        — one-shot shaping
│   └── shape_trace()  — shaping with full trace
├── _binding.py        — ctypes declarations for C API
├── _loader.py         — cross-platform native lib discovery
└── _lib/              — bundled native library (DLL/SO/dylib)

yudit_shaper (C/C++)
├── yudit_shaper.h/cpp — C API wrapper
├── YuditParagraphShaper.h/cpp — paragraph layer (Arabic/Syriac joining, RTL)
├── YuditScriptProcessor.h/cpp — script processor trace (Indic, Thai, Lao, …)
├── stoolkit/          — Yudit core (BiDi, Unicode, glyphs, paragraphs)
├── swindow/           — Font handling (SFontTTF) + SScriptProcessor
└── CMakeLists.txt     — build system

Installation

From source (with native build)

# Prerequisites: CMake 3.15+, C++ compiler
git clone https://github.com/Kushim-Jiang/pyyudit.git
cd pyyudit

# Download Yudit source (for the shaping engine)
curl -L https://www.yudit.org/download/yudit-3.1.0.tar.gz | tar xz

# Build the native library
python scripts/build_native.py

# Install the Python package
pip install -e ".[dev]"

Cross-platform wheel

# Build wheel (includes the native library)
pip install hatchling build
python -m build
pip install dist/pyyudit-0.3.0-py3-none-any.whl

Usage

import pyyudit

# One-shot shaping
font_bytes = open("NotoSansDevanagari.ttf", "rb").read()
glyphs = pyyudit.shape(font_bytes, "नमस्ते")
for g in glyphs:
    print(f"gid={g.g} cluster={g.cl} adv={g.ax}")

# With full trace
with pyyudit.YuditFont.from_path("NotoSansDevanagari.ttf") as f:
    result = f.shape_trace("नमस्ते")
    for stage in result.stages:
        print(f"{stage.m}: {len(stage.glyphs)} glyphs")
        if stage.effective:
            print("  (buffer changed)")

# Reusable font object
font = pyyudit.YuditFont.from_path("NotoSansArabic.ttf")
result = font.shape("مرحبا")
print(f"Advance: {result.advance_x} units")
font.close()

Trace stages

The trace captures every step of Yudit's shaping pipeline. Which stages you see depends on the script:

Each stage carries the full glyph buffer at that point, so stage.glyphs is a snapshot you can diff against the previous stage. stage.effective is True when that step actually changed the buffer — a substitute that found nothing, or a reorder that was already in order, reports False.

Every GSUB/GPOS feature stage is followed by one group of steps per lookup the feature ran, using the same message vocabulary as the reference trace:

start lookup 114 feature 'half'
skipped lookup 114 feature 'half' because no glyph matches
end lookup 114 feature 'half'

skipped appears only when no glyph matched. Each start/end pair holds the glyph snapshot as it is after the whole feature ran — the engine does not expose a buffer per lookup, so the per-lookup steps share the feature's snapshot. effective sits on the end step and marks the lookup that changed the buffer. Yudit retries a feature at every position in the run; the repeats are collapsed so each lookup is listed once per feature.

g holds the Unicode codepoint in the stages before glyph lookup (input, decompose, reorder, split, join) and the glyph index afterwards, which is how the reference trace reports an unmapped buffer. cl is the index into the original text.

Glyph positions use the reference format's own model, so the same layout code can consume both this and a HarfBuzz buffer:

pen = 0
for glyph in run:
    draw_at(pen + glyph.dx, glyph.dy)     # dx/dy are offsets
    pen += glyph.ax                       # ax is the advance

ax is never negative: Yudit reports a negative width for glyphs with a negative left side bearing, as a marker for "align to the end of the previous character", and the magnitude is what it advances by. dx is the offset from the pen position, not the step from the previous glyph, so an unpositioned glyph has dx == 0 and a mark carries its attachment offset. The run's advance_x is the sum of the reported advances.

In the joining path the glyphs are returned in visual order (reversed for RTL runs, like HarfBuzz), so pen accumulates in output order.

tests/compare_trace.py lays the trace next to a HarfBuzz trace of the same text, so the lookup count, the feature sequence and the resulting buffer can be compared step by step.

Pure-Python helpers

These work without the native library:

from pyyudit import upem_from_ttf, has_table, read_font_bytes

data = read_font_bytes("font.ttf")
upem = upem_from_ttf(data)
has_gsub = has_table(data, "GSUB")

Configuration

# Set explicit library path
pyyudit.configure("/path/to/libyudit_shaper.so")

# Or via environment variable
# export PYYUDIT_LIBRARY_PATH=/path/to/libyudit_shaper.so

# Check availability
print(pyyudit.is_available())
print(pyyudit.library_info())

Supported scripts

Yudit's shaping engine supports these scripts (with OpenType features):

Joining / right-to-left scripts go through the paragraph layer instead:

Kerning

Yudit does not kern. Pair adjustment (kern, GPOS lookup type 2) exists as a function in the engine but only checks coverage and always returns false, and the legacy kern table code sits behind #if 0. The kern stage therefore shows up in the trace as not effective for every font. This is reported as-is rather than implemented: the point of the trace is to show what Yudit does.

Limitations

• Pair kerning (kern) and legacy kern tables are absent, as described above.
• GPOS cursive attachment (curs, lookup type 3) is not implemented. Fonts that encode Arabic joining adjustments there (Dubai, for example) will miss those deltas.
• Chained contextual positioning (lookup type 8) is only partially implemented.
• Hangul jamo are not precomposed: Yudit runs ljmo/vjmo/tjmo and leaves the jamo as separate glyphs when the font does not carry those features.
• Yudit only knows the first-edition Indic script tags (deva, beng, taml, …). A font that registers only the second-edition tags (dev2, bng2, tml2, …) — Nirmala UI, for example — has no matching features at all: the run comes back unshaped and the trace shows zero lookups. This is reported, not worked around: the second-edition model applies its features in a different order, so driving it with Yudit's first-edition reordering would produce wrong glyphs. Where a font carries both editions the glyphs can still differ from HarfBuzz, which prefers the second-edition tags.
• Reordering is per run: the base direction is detected with UAX#9 P2/P3 and the run is reversed for RTL. Multi-run BiDi is the caller's job, as it is with HarfBuzz.
• face_index on yudit_font_open is currently ignored and font collections (TTC) are not supported.

License

• pyyudit Python code: MIT
• Yudit shaping engine: GPL-2.0-or-later (Gaspar Sinai)
• See COPYING.TXT for details.