pygx 0.2.0

PyGX: A library for manipulating Python objects.

An À La Carte Data Model — from Dataclass to Symbolic Programs

Documentation | Why PyGX | Performance | When to use | Installation | What's in the box

Why PyGX

PyGX is a data model with a slider. At one end it is a plain, typed, fast dataclass; at the other it is a full symbolic object model that algorithms can inspect, rewrite, and search over. Between the two you slide notch by notch — and the capabilities are à la carte: order exactly the ones you need, and what you don't order costs nothing. Every capability you don't turn on is structurally absent, not a skipped branch, and the hot paths run in a native Rust core that keeps the bottom of the slider at dataclass-class performance while beating Pydantic v2 on construction (see Performance).

[!IMPORTANT] Breaking change (pre-1.0): the pg.Object default flipped from sym=True to sym=False. A plain class C(pg.Object) is now a flat, validated dataclass — construction and assignment still validate, JSON round-trip / equality / hashing / diffing / hooks all still work — but it is not a symbolic-tree node. The semantic change to watch for: nested assignment of an already-parented object used to clone it into the new location (the single-parent invariant); at the new default it holds a plain reference, and tree operations (sym_rebind, sym_setparent) raise SymbolicModeError. Add sym=True to your class statement to get the previous behavior — it inherits to subclasses. PyGX's own tree-based classes (functors, hyper primitives, DNA, ...) pin sym=True internally and are unaffected.

Notch 0 — a typed struct, at dataclass speed

Declare fields with annotations, exactly as you would with @dataclass or Pydantic. This is the default posture — plain reference semantics: no tree, no tracking, just a validated value object:

import pygx as pg

class Rect(pg.Object):
    width: int
    height: int

r = Rect(width=2, height=3)          # ~290 ns — faster than pydantic v2
Rect(width="2", height=3)            # TypeError — validated at construction
pg.eq(r, Rect(width=2, height=3))    # True — value equality
r.to_json_str()                      # '{"_type": "...Rect", "width": 2, "height": 3}'

Equality, hashing, cloning, diffing, readable repr, and typed JSON round-trip all come with the struct — no boilerplate, no separate schema language.

Notch 1 — richer validation, same struct

When an annotation can't express the constraint, attach a value spec. It is enforced at construction and on every later assignment:

from typing import Annotated

class Rect(pg.Object):
    width: Annotated[int, pg.field(value_spec=pg.typing.Int(min_value=1))]
    height: Annotated[int, pg.field(value_spec=pg.typing.Int(min_value=1))]

Rect(width=-1, height=3)   # ValueError: Value -1 is out of range (min=1).

Notch 2 — the symbolic tree

Opt in with sym=True and the construction arguments are no longer thrown away: the object becomes a node in a mutable, parent-aware, self-validating tree. Nested objects know their path; assigning an object into a tree adopts it (cloning it if it already has a parent, preserving the single-parent invariant); mutation re-validates and fires change events; objects can be partial, sealed, or contextual. sym=True inherits to subclasses, so one symbolic base is enough for a whole hierarchy:

class Rect(pg.Object, sym=True):
    width: Annotated[int, pg.field(value_spec=pg.typing.Int(min_value=1))]
    height: Annotated[int, pg.field(value_spec=pg.typing.Int(min_value=1))]

    def on_sym_ready(self):              # derived state, recomputed on change
        super().on_sym_ready()
        self.area = self.width * self.height

r = Rect(width=2, height=3)              # r.area == 6
r.sym_rebind(width=5)                     # in-place, re-validated, area -> 15

p = Rect.partial(width=2)                 # late binding: height unbound
p.sym_rebind(height=3)                    # now complete -> on_sym_ready fires
Rect(width=2, height=3).sym_seal()        # immutable from here on

cfg = pg.Dict(a=r, b=Rect(width=4, height=5))
cfg.a.sym_path                            # 'a' — every node knows where it lives
cfg.sym_rebind({'a.height': 6})           # deep-path edits, batched + one notify
pg.query(cfg, where=lambda v: isinstance(v, Rect))

Notch 3 — symbolic programs

Because the object is a tree, programs become data. Swap any field for a space of values and enumerate, sample, or evolve it; patch trees declaratively; detour construction in code you don't own; bind functions partially and rewrite the binding later:

space = Rect(width=pg.oneof([2, 4]), height=pg.oneof([3, 5]))
list(pg.iter(space))                      # 4 concrete Rects
pg.patch_on_key(cfg, "width", 1)          # every width in the tree -> 1

@pg.functor()
def scale(rect, factor=2):
    return rect.area * factor

f = scale(factor=3)                       # a symbolic, rebind-able call
f.sym_rebind(factor=10)

This is the notch AutoML, program search, and evolutionary computation live on — and it is the same object model, not a separate spec language.

Capabilities are à la carte

Each capability is a class keyword, orthogonal to the rest — sym, slots (inline-slot storage for memory-critical classes), frozen, eq, order, validate, attr_read / attr_write — and an option you don't enable costs nothing at runtime: no tree state on a sym=False (the default) leaf, no validation machinery on validate=False fields, no per-instance __dict__ on a slots=True record. Context-aware fields (pg.ContextualObject), URI-based construction (pg.from_uri), and schema-driven meta-programming (every class carries a queryable __schema__) layer on the same model.

More meta-programming toolkits

Beyond the data model, PyGX treats program construction itself as something you can reach into:

• Functors — advanced function binding. pg.functor turns a function into a symbolic, partially-bindable call: bind some arguments now, supply or sym_rebind the rest later, and inspect or serialize the call before it ever runs.

  @pg.functor()
  def scale(rect, factor=2):
      return rect.width * rect.height * factor
  
  f = scale(factor=3)              # bind factor now; rect supplied at call time
  f(Rect(width=2, height=3))       # 18
  f.sym_rebind(factor=10)              # sym_rebind a bound argument later
  

• Detour the logic of an existing class. pg.detour redirects construction to another class inside a context manager — intercept object creation in code you don't own. (pg.symbolize / pg.wrap / pg.wrap_module similarly opt plain classes, functions, and whole modules into the symbolic world without a rewrite.)

  with pg.detour([(ResNet, MobileNet)]):
      model = build_model()          # internal ResNet(...) now yields a MobileNet
  

• Patch existing objects. pg.patch, pg.patcher, and the pg.patch_on_* family compose declarative rewrites by key, path, value, type, or member.

  cfg = pg.Dict(a=Rect(width=2, height=3), b=Rect(width=4, height=5))
  pg.patch_on_key(cfg, "width", 1)   # every width in the tree -> 1
  

Handy utilities

• Portable logging, metrics, and IO. Project-scoped logging (pg.logging), metric collection (pg.monitoring), and lightweight timing (pg.timeit), plus an fsspec-backed IO layer so pg.save / pg.load / pg.open_jsonl work transparently over local, GCS, S3, and friends — rounded out by parallel pg.concurrent.execute / map (retries, timeouts, progress bar) and pg.reload, which follows the module graph instead of leaving stale references behind.

  pg.concurrent.execute(lambda r: r.area, [Rect(width=2, height=3)], max_workers=8)
  pg.save(Rect(width=2, height=3), "gs://bucket/rect.json")   # local, GCS, S3, ...
  

• Code generation with permission control. pg.coding compiles and runs Python source under a declared permission set, with optional subprocess sandboxing.

  pg.coding.evaluate("1 + 1")        # 2
  pg.coding.evaluate("import os", permission=pg.coding.CodePermission.BASIC)
  # CodeError: `import` is not allowed.
  

Performance

The symbolic model used to be a tax; it isn't anymore. PyGX's hot paths — construction, attribute access, storage, and per-node state — run in a native Rust core (pygx-core, installed automatically), while the pure-Python implementation remains the executable specification: the full test suite runs against both cores in CI, so they cannot diverge.

Median ns/op on a 3-field object (Apple Silicon; read ratios, not absolutes — the full report covers ~50 operations across scales):

operation	pygx sym=False (default)	pygx sym=True	@dataclass	pydantic v2
construct (kwargs)	291	274	189	508
attr get	43	44	39	45
clone (deep)	808	813	2,210	1,620
to dict/json	379	404	652	527
from dict/json	634	622	200	659

Construction is ~1.8× faster than Pydantic v2 with the whole symbolic model attached; attribute reads are at dataclass speed; deep clone and serialization beat both. (A few cold ops — eq, hash — still trail a bare dataclass; the report tracks them honestly.) Wheels ship for Linux (glibc + musl, x86_64 + aarch64), macOS (Intel + Apple Silicon), and Windows on CPython 3.12–3.14 — plus, from pygx-core ≥ 0.1.2, a wheel for the free-threaded 3.14t build that is genuinely free-threaded (the GIL stays off; crash-freedom and per-operation — per-node, per-field — atomicity per the threading contract in docs/design/gil-free.md §3); on anything else PyGX falls back to the pure-Python core with identical behavior.

When PyGX is useful

Reach for PyGX when your problem has any of these shapes:

• You need extensive Python meta-programming. Symbolizing existing classes, functions, or whole modules; redirecting __new__ with pg.detour; reloading transitive module graphs with pg.reload; controlled parse / evaluate / run of generated source — PyGX treats program construction itself as a first-class activity.
• You need a mutable data model. Symbolic objects are deeply mutable, parent-aware, and self-validating. sym_rebind propagates change events up and down the tree; on_sym_change / on_sym_ready hooks let derived state recompute incrementally; partial / late binding, declarative patching, and deep diffing fall out for free.
• You need to manipulate objects with algorithms. Once a program is a symbolic tree, algorithms can operate on it directly: enumerate or sample a search space (pg.iter, pg.random_sample); drive distributed search (pg.sample); evolve populations with composable selectors and mutators (pg.algo.evolution); synthesize functions from scratch (pg.algo.mutfun); or just pg.traverse / pg.query / pg.transform over the tree yourself.

Concrete domains where these properties cash out:

• Configurable, composable applications. Replace ad-hoc config systems (YAML/JSON/argparse glue) with typed, validated, serializable Python objects that are the configuration.
• Machine learning at team scale. Models, pipelines, and experiments expressed as symbolic trees are easy to log, diff, share, and recombine — with no separate spec language.
• AutoML, program search, and evolutionary computing. Drop search into an existing Python program by replacing scalars with hyper primitives, then reach for pg.sample + pg.algo.evolution to drive distributed search. Populations are just lists of symbolic programs; mutations are direct rewrites; selectors operate on object metadata.
• Daily Python work. Late/partial binding, deep direct manipulation, context-aware components, declarative patching, and an interactive HTML inspector — useful well outside ML.

Install

pip install pygx

Nightly build:

pip install pygx --pre

Optional extras:

pip install "pygx[io]"          # fsspec-backed remote IO (GCS, S3, ...)
pip install "pygx[concurrent]"  # parallel execute/map with retries + progress

Requires Python 3.12+.

What's in the box

PyGX is a small core plus focused sub-packages — and every one of them is a consumer of the same symbolic model: pg.hyper turns fields into search spaces, pg.geno/pg.tuning drive distributed search over them, pg.algo.evolution mutates and recombines symbolic programs directly, pg.patching/pg.detouring rewrite trees and construction, and pg.views renders any node as an interactive HTML tree. Nothing downstream needs a second representation of your program. The most common entry points are re-exported at the top level as pg.*.

Symbolic object model — pg.symbolic

• pg.Object — schema-driven, dataclass-like classes with a synthesized keyword-only __init__, PEP 681 dataclass_transform support (pyright/Pylance/mypy understand it), and runtime field validation.
• pg.Dict, pg.List — mutable symbolic containers that participate in the same tree as pg.Object.
• pg.symbolize, pg.functor, pg.wrap, pg.wrap_module — opt existing classes, functions, or whole modules into the symbolic world without rewriting them.
• pg.field — dataclass-style field descriptor for defaults, factories, per-field flags (init=False, repr=False, compare=False, ...), and inline value-spec overrides.
• Symbolic operations: pg.eq, pg.ne, pg.lt, pg.gt, pg.hash, pg.clone, pg.diff, pg.traverse, pg.query, pg.contains.
• Serialization: pg.to_json / pg.from_json (and _str variants), pg.save / pg.load, pg.open_jsonl.
• Lifecycle hooks: on_sym_ready, on_sym_bound, on_sym_change, on_sym_parent_change, on_sym_path_change for incremental update and parent-aware behavior.
• Per-class behavioral knobs via class-statement kwargs (immutability, freezing, equality semantics, registry membership).

Runtime typing — pg.typing

• pg.typing.Int, Float, Str, List, Dict, Object, Union, Callable, ... — composable value specifications.
• Bounds, regex, default values, custom converters via pg.typing.register_converter.
• Annotations are automatically lowered to value specs, so the common case needs no explicit pg.typing.* usage. Reach for it when you need constraints the annotation can't express.

Search & program generation — pg.hyper, pg.geno, pg.tuning

• Hyper primitives turn any symbolic value into a space of values: pg.oneof, pg.manyof, pg.permutate, pg.floatv, pg.evolve.
• Genotypes (DNA/DNASpec) decouple the algorithms that generate programs from the user code that consumes them.
• pg.iter, pg.random_sample, pg.materialize — local enumeration and sampling.
• pg.sample — distributed search with a pluggable backend, work groups, failover, and result polling.

Algorithms — pg.algo

• pg.algo.evolution — compositional evolutionary algorithms built from selectors, mutators, and recombinators wired with the >> operator. Ready-made constructors: hill_climb, neat, nsga2, regularized_evolution.
• pg.algo.mutfun — mutable symbolic programs (Function, Assign, Var, ...) for symbolic regression and from-scratch program synthesis.
• pg.algo.scalars — scalar schedules.
• pg.algo.early_stopping — early-stopping policies for tuning.

Detour and patching — pg.detouring, pg.patching

• pg.detour([(A, B)]) — redirect A.__new__ to B.__new__ inside a context manager. Works on non-symbolic classes; useful for intercepting object creation inside code you don't own.
• pg.patch, pg.patcher, pg.patch_on_key / _path / _value / _type / _member — declarative, composable rewrites over a symbolic tree.
• pg.from_uri — build (or patch) symbolic objects from URI-like strings.

Context-aware components — pg.contextual

• Thread-local stack of variable bindings installed by pg.contextual.override and read via pg.contextual.get.
• pg.ContextualObject — symbolic fields that resolve from the surrounding context when unset, propagating into child threads via pg.contextual.with_override.

Visualization — pg.views

• pg.view, pg.view_options — pluggable rendering for symbolic values.
• pg.to_html / pg.to_html_str — self-contained, interactive HTML tree-view of any symbolic object (rendered inline in Jupyter or saved to disk).
• pg.controls — small interactive widgets composable into HTML views.

Concurrent execution — pg.concurrent

• pg.concurrent.execute / execute_async — parallel map preserving input order, with per-item and overall timeouts.
• pg.concurrent.map — lazy (input, output, error) stream with a thread-safe progress bar.
• pg.concurrent.with_retry / with_retry_async — bounded retries with optional exponential backoff.
• ExecutorPool keyed by resource id so global rate limits are honored across nested calls.
• Sync/async bridging that propagates pg.contextual overrides across the boundary.

Code generation — pg.coding

• pg.coding.parse / evaluate / run — compile and run Python source under a declared permission set (assignment, function definition, imports, ...).
• pg.coding.make_function — synthesize a callable from a source string.
• pg.coding.sandbox_call / maybe_sandbox_call — run a callable in a child process with a timeout.

Pluggable IO — pg.io

• Abstract FileSystem / File / Sequence interfaces.
• In-memory backend for tests, plus an fsspec bridge that picks up any registered filesystem (local, GCS, S3, ...).
• Used by pg.save, pg.load, and pg.open_jsonl.

Instrumentation — pg.instrument

• pg.logging — project-scoped logger.
• pg.monitoring — metric collection.
• pg.timeit — lightweight scoped timing.

Errors and topology — pg.error_handling, pg.topology, pg.formatting

• pg.catch_errors, pg.match_error, pg.ErrorInfo — pattern-matched exception suppression and structured, JSON-serializable error capture.
• pg.KeyPath, pg.MISSING_VALUE, pg.traverse, pg.JSONConvertible — the tree/value vocabulary the rest of the library is built on.
• pg.format, pg.print, pg.colored — readable, optionally-colored rendering of symbolic values.

Importing — pg.importing

• pg.reload — module reload that follows dependency edges, refreshing transitively-affected modules instead of leaving stale references behind.

Background

PyGX was originally built at Google Brain / DeepMind by Daiyi Peng to power automated machine learning (AutoML), under the name PyGlove. The abstraction underneath — symbolic object-oriented programming (SOOP) — turned out to be much more general than AutoML, and PyGlove grew into a toolkit for advanced Python programming used well beyond ML. The original PyGlove paper was published at NeurIPS 2020.

Documentation & examples

• Bird's-eye view of PyGX — 5–10 minute tour of the core ideas.
• Authoring pg.Object subclasses — style guide for positional __init__, field defaults, and ClassVar config.
• User Guide — topic-specific tracks for Python, ML, AutoML, and Evolution.
• Learning PyGX — conceptual material on Symbolic Object-Oriented Programming and Symbolic Detour.
• API Reference — generated from source.

Runnable notebooks live in docs/notebooks/ and source examples in examples/.

Citing PyGX

@inproceedings{peng2020pyglove,
  title={PyGlove: Symbolic programming for automated machine learning},
  author={Peng, Daiyi and Dong, Xuanyi and Real, Esteban and Tan, Mingxing and Lu, Yifeng and Bender, Gabriel and Liu, Hanxiao and Kraft, Adam and Liang, Chen and Le, Quoc},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  volume={33},
  pages={96--108},
  year={2020}
}

License

Apache License 2.0. PyGX is derived from PyGlove (also Apache 2.0); see LICENSE and the per-file copyright headers for attribution.

PyGX is developed by Daiyi Peng.