Skip to main content

Hydra-Python

This package contains the Python coder DSL sources: Python modules that describe how to translate Hydra modules into Python source code. The runnable Python head (hand-written primitives, DSL runtime, pyproject.toml, test runner) lives in heads/python/. The generated Python kernel lives in dist/python/hydra-kernel/.

Hydra is a type-aware data transformation toolkit which aims to be highly flexible and portable. It has its roots in graph databases and type theory, and provides APIs in Haskell, Java, Python, Scala, TypeScript, and Lisp. See the main Hydra README for more details.

📖 API documentation: Sphinx-generated API docs for the Python packages are published at the Hydra Python API index (individual packages: hydra-kernel, hydra-rdf, hydra-pg).

Getting started

Hydra-Python requires Python 3.12 or later.

Install uv:

curl -LsSf https://astral.sh/uv/install.sh | sh

Create the Python virtual environment in the Python head directory:

cd heads/python
uv venv --python 3.12
source .venv/bin/activate

Install the dependencies:

uv sync

Documentation

For comprehensive documentation about Hydra's architecture and usage, see:

Testing

Hydra-Python has two types of tests: the hydra-kernel test suite (shared across all Hydra implementations) and Python-specific tests. See the Testing wiki page for comprehensive documentation.

Hydra-kernel test suite

The hydra-kernel test suite (hydra.test.testSuite) ensures parity across all Hydra implementations. Passing all hydra-kernel test suite cases is the criterion for a true Hydra implementation.

To run all tests (from heads/python/):

cd heads/python && pytest

To run only the hydra-kernel test suite:

cd heads/python && pytest src/test/python/test_suite_runner.py

The test suite is generated from Hydra DSL sources and includes:

  • Primitive function tests (lists, strings, math, etc.)
  • Case conversion tests (camelCase, snake_case, etc.)
  • Type inference tests
  • Type checking tests
  • Evaluation tests
  • JSON coder tests
  • Rewriting and hoisting tests

Python-specific tests

Python-specific tests validate implementation details and Python-specific functionality. These are located in heads/python/src/test/python/ alongside the hydra-kernel test suite runner.

To run a specific test file:

cd heads/python && pytest src/test/python/test_grammar.py

To match a specific test by name:

cd heads/python && pytest -k test_grammar

To see printed outputs, use the -s flag:

cd heads/python && pytest -s

Code organization

Hydra's Python code is split across three locations (see Code organization wiki page for the full picture):

  • This package (packages/hydra-python/src/main/python/hydra/sources/python/) — the Python coder DSL sources (written in Python). These are the source of truth for the hydra.python.* modules (syntax.py, language.py, coder.py, serde.py, names.py, utils.py, environment.py, testing.py, plus the _python_helpers.py / _kernel_refs.py support modules).

  • Python kernel overlay (overlay/python/hydra-kernel/src/main/python/) — hand-written Python kernel runtime, overlaid onto dist/python/hydra-kernel/ by heads/python/bin/copy-kernel-runtime.sh so the published hydra-kernel wheel is self-contained

    • hydra/overlay/python/lib/ — primitive function implementations
    • hydra/overlay/python/dsl/ — DSL utilities (FrozenDict, Maybe, ...)
    • hydra/overlay/python/util/ — ConsList, Lazy, PersistentMap, PersistentSet
    • hydra/overlay/python/sources/libraries.py — primitive registration
  • Python head (heads/python/src/main/python/) — bootstrap layer above the kernel (bootstrap.py, generation.py, the hydra.python coder package). pyproject.toml lives in heads/python/.

  • Generated Python kernel (dist/python/hydra-kernel/src/main/python/)

    • hydra/core.py — core types (Term, Type, Literal, ...)
    • hydra/graph.py, hydra/packaging.py — graph and packaging structures
    • hydra/coders.py — type adapters and coder framework
    • hydra/reduction.py, hydra/rewriting.py, hydra/hoisting.py — term transformations
    • hydra/inference.py, hydra/checking.py — type inference and checking
    • Generated from the kernel DSL sources using the Python coder
  • Generated Python test suite (dist/python/hydra-kernel/src/test/python/)

    • Common tests ensuring parity with Haskell, Java, Scala, and Lisp

Generate Python code

Python code generation has two stages: first the Python coder modules' DSL sources are exported to JSON (Phase 1), then the JSON is loaded by the Python host and used to generate dist/python/hydra-kernel/ (Phase 2). The two stages live in different scripts and can be invoked independently.

Phase 1: regenerate dist/json/hydra-python/ from the Python DSL sources

bin/generate-hydra-python-from-python.sh is the self-hosting entry point: it runs the Python DSL sources in this package through the Python host and writes dist/json/hydra-python/.

# Regenerate hydra-python JSON from packages/hydra-python/src/main/python/hydra/sources/python/
bin/generate-hydra-python-from-python.sh

# Same, with byte-compare against the existing canonical
bin/generate-hydra-python-from-python.sh --compare

# Use PyPy for ~4x faster generation (CPython is the default)
bin/generate-hydra-python-from-python.sh --pypy

# Force a rebuild of the Python host (kernel JSON + dist/python/hydra-kernel) first
bin/generate-hydra-python-from-python.sh --force-rebuild

The script:

  1. Runs bin/sync-python.sh to ensure dist/python/hydra-kernel/ and dist/python/hydra-python/ are current (these are the only trees the Python DSL → JSON driver reads). Gated by HYDRA_IN_SYNC=1 so that sync.sh Phase 5 invoking us doesn't recurse.
  2. Runs bin/update-python-json.py, which loads the kernel universe from dist/json/hydra-kernel/, imports the Python DSL source modules, infers types, and writes the resulting JSON.

End-to-end is ~110 seconds under PyPy (faster than the Haskell incremental pipeline) and ~500 seconds under CPython, once dist/ is current. See bin/update-python-json.md for background.

Note: bin/sync.sh Phase 5 invokes generate-hydra-python-from-python.sh automatically — the native Python DSL path is the sole source of truth (the legacy Haskell DSL copies under packages/hydra-python/src/main/haskell/ were deleted in #346). See claude/pitfalls.md for the HYDRA_IN_SYNC convention around wrapper-script self-syncing.

Phase 2: regenerate dist/python/ from the JSON

The recommended end-to-end script is:

bin/sync-python.sh

(equivalent to bin/sync.sh --hosts python --targets python)

This will:

  1. Generate / refresh dist/json/ from the native Python DSL sources
  2. Generate the Python kernel into dist/python/hydra-kernel/src/main/python
  3. Generate the kernel tests into dist/python/hydra-kernel/src/test/python
  4. Run the pytest suite

Validate generated code

find dist/python/hydra-kernel/src -name "*.py" -exec python3 -m py_compile {} +

Formatting, linting, and type checking

Install Ruff, pyright, and pytest, e.g. on macOS:

brew install ruff
brew install pyright
brew install pytest

All of these commands run from the heads/python/ directory (files/directories can also be specified as arguments).

Formatting

Format the hand-written Python code:

ruff format

Linting

Run the linter:

ruff check

Fix fixable linting errors (e.g. removing unused imports):

ruff check --fix

Static type checking

Run the type checker:

pyright

Numeric types

Hydra's decimal type is implemented as Python decimal.Decimal with the default 28-digit context precision. Operations exceeding this precision round per the active context; users requiring higher precision should adjust decimal.getcontext().prec before performing arithmetic. This differs from Haskell Scientific and Java BigDecimal (which are effectively unbounded for exact operations) but matches Python's standard decimal behavior.

Collections

Hydra-Python uses an API/implementation split for list, map, and set values, mirroring Hydra-Java:

  • At the type level, generated Python uses the standard collections.abc abstract base classes — Sequence[E] for lists, Mapping[K, V] for maps, and Set[E] for sets. Public function signatures stay generic and dependency-free; callers can pass any compatible collection.
  • At the implementation level, generated term-level literals construct immutable collection classes from hydra.python.util: ConsList (a frozen sequence), PersistentMap (a frozen map), and PersistentSet (a frozen set). Each implements the corresponding collections.abc ABC, so ConsList IS a Sequence, PersistentMap IS a Mapping, and PersistentSet IS a Set.

These classes are thin facades over native tuple, dict, and frozenset. All mutations build a fresh native container via {**self, **other}, tuple(...), or frozenset(...) and freeze it under the immutable wrapper. The cost is full O(n) copy on every update (no structural sharing); the benefit is C-speed inner loops. Hydra-Python has no third-party runtime dependencies beyond the standard library.

Where ordered iteration matters (hydra.lib.maps.{keys, elems, to_list}, the various *_list() extraction helpers, PersistentSet.__iter__), elements are sorted at extraction time via a fall-through comparator: natural < where it works, structural comparison for Hydra Term/Type and other complex values that don't define ordering in Python.

The classes live under hydra.python.util rather than hydra.util because the latter is already a kernel-generated module (containing Comparison, CaseConvention, etc.) shared across all Hydra implementations. Putting the Python-runtime helpers under hydra.python.util keeps the kernel namespace intact while making the host/kernel separation explicit. hydra.python itself is a pkgutil-style namespace package so heads-side helpers and the kernel-generated hydra.python.{coder,environment,...} modules coexist cleanly.

CPython vs PyPy

The bootstrap demo and bin/run-bootstrapping-demo.sh prefer pypy3 when available and fall back to CPython 3.12+. Both interpreters pass the full test suite. For most real workloads, PyPy is the better choice — its JIT makes term-level transformation (the dominant Python-host cost) several times faster than CPython.

Rough guide:

Workload Faster on
bin/run-bootstrapping-demo.sh codegen CPython by ~5%
Type inference on large modules (hydra.codegen.infer_modules_given) PyPy by ~4×
hydra.lib.* primitive microbenchmarks CPython by ~2.5×

The microbench gap reflects CPython's C-level dict/frozenset/tuple operations beating PyPy's pure-Python equivalents. The inference gap reflects PyPy's JIT amortizing per-call dispatch overhead in long-running term walks. For day-to-day development you can pick whichever is convenient (CPython is usually already installed); PyPy becomes worthwhile when you hit term-level workloads measured in seconds or minutes.

Set HYDRA_PYTHON_INTERPRETER=pypy3 (or any path/name) in the environment to force a specific interpreter for the bootstrap demo.

Future enhancements

Recommendations from #233 that haven't been adopted yet. Recorded here so the design intent survives any future re-evaluation. These are deliberate non-goals today, not bugs.

__match_args__ for structural pattern matching

Python 3.10's match/case can destructure objects that declare __match_args__. The Python coder could emit this attribute on every generated dataclass-like type, letting consumers write:

match lit:
    case LiteralString(value=v):
        return v
    case LiteralInteger(value=iv):
        match iv:
            case IntegerValueInt32(value=n):
                return n

instead of today's isinstance() chains. Cheap codegen tweak; purely additive (existing isinstance code continues to work). Should ship after other in-flight Python work is stable so the coder edit lands on a quiet baseline.

Kwargs syntax for record construction

# Today
record(Name("Person"), [field(Name("name"), string("Alice")),
                        field(Name("age"), int32(30))])

# Proposed (combined with the existing str→Name auto-coercion)
record("Person", name=string("Alice"), age=int32(30))

Python's **kwargs preserves insertion order (3.7+), so field order is stable. List-of-field() form would remain as an overload for the programmatic case. Edge case: field names that are Python keywords (class, type) need either trailing-underscore convention or a mix of kwargs + explicit field().

Decorator-based element definitions

@hydra_element(my_module, type_=T.function(T.int32(), T.string()))
def show_number(x):
    return Strings.print_int32(x)

Idiomatic Python metadata mechanism. Risk: users may expect the decorator to "compile" arbitrary Python into Hydra terms, which it can't — the decorated body still has to be a TTerm expression. Clear docs required.

Context managers for module scoping

with hydra_module("myModule", namespace="com.example") as m:
    m.define("Person", T.record(name=T.string(), age=T.int32()))
    m.define("greet", lam("p", Strings.cat2(string("Hello, "),
                                            project("Person", "name") @ var("p"))))

Tension with Hydra's functional philosophy (modules are declarative data, not imperative side-effects). Mitigation: the context manager collects definitions and produces an immutable Module on __exit__.

Dataclass-style decorator for record type definitions

@hydra_record("com.example.Person")
class Person:
    name: str             # → T.string()
    age: int              # → T.int32()
    email: Optional[str]  # → T.optional(T.string())

Familiar to anyone who's used @dataclass. Significant scope limits: mapping Python annotations to Hydra types only works for records with simple fields; unions, wrapped types, polymorphic types have no natural Python-annotation equivalent. Would cover a subset only.

Metadata

Release files for hydra-python 0.17.7

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for hydra-python 0.17.7
File Size Uploaded Provenance Actions
hydra_python-0.17.7.tar.gz 116.1 kB Without provenance Details for hydra_python-0.17.7.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for hydra-python 0.17.7
File Interpreter ABI Platform Actions
hydra_python-0.17.7-py3-none-any.whl Python 3 none any Details for hydra_python-0.17.7-py3-none-any.whl

Total release size: 233.8 kB

Release files / hydra_python-0.17.7.tar.gz

Download URL hydra_python-0.17.7.tar.gz
Size 116.1 kB
Tags Source
SHA-256 checksum
How to use checksums
2ea26624982659b2c12014add994c6a2cf594632d39f28d11516065db75fd381
BLAKE2b-256 checksum
How to use checksums
f756e7ee91ffabf60cdb2ee907ab957de15687b39bfc72d21333754bcd09bb3b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.8

Release files / hydra_python-0.17.7-py3-none-any.whl

Download URL hydra_python-0.17.7-py3-none-any.whl
Size 117.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
72be82c7960cc8bfbae02d7c3f2bffd12927786dd8d37cbb6a301a139c431809
BLAKE2b-256 checksum
How to use checksums
f16ee581abb6d5101f3e36497fc77520cdd755549e00016a63b7621a412f7fb6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.8

Security for hydra-python 0.17.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page