hypernodes 0.4.4

A hierarchical, modular pipeline system for ML/AI workflows

[Installation] | [Documentation] | [License]

HyperNodes

Hierarchical, Modular Data Pipelines for AI/ML

HyperNodes is designed to make building complex data pipelines intuitive and scalable. It starts with a simple premise: define your logic on single items, then compose and scale.

✨ Key Features

Simplicity First: Think Singular

Define your functions and pipelines as if they are processing a single item (e.g., one PDF, one audio file, one text string). This makes your code:

• Easy to grasp: No complex loops or batch logic cluttering your business logic.
• Testable: Write standard unit tests for individual functions.
• Debuggable: Step through your code easily.

🪆 Hierarchical Composition

Once you have your building blocks, compose them into progressively more complex pipelines.

• Nesting: Pipelines are nodes. You can use a pipeline as a node inside another pipeline.
• Mapping: Seamlessly apply a pipeline over a list of inputs using .map().
• Visual: Always keep one level of understanding. Look at the high-level flow, then dive deeper into specific sub-pipelines as needed.

Structured Data & Protocols

HyperNodes integrates deeply with Pydantic and dataclasses. Define reusable protocols and data structures to ensure type safety and clarity throughout your pipeline. This leads to a codebase that is easy to maintain and extend.

🚀 Powered by Daft: Performance & Caching

When you're ready to scale, HyperNodes leverages Daft as its computation engine.

• Distributed Execution: Run on your laptop or a cluster without changing code.
• Intelligent Caching: Caching is a first-class citizen. Daft handles distributed caching, ensuring you only recompute what's necessary.

---

💡 Inspiration

HyperNodes stands on the shoulders of giants. It was inspired by and grew from working with:

• Pipefunc: For its elegant approach to function composition.
• Apache Hamilton: For its paradigm of defining dataflows using standard Python functions.

HyperNodes aims to bring native support for hierarchical pipelines and advanced caching to this ecosystem.

---

📚 Documentation

The full documentation is available in the docs/ directory:

• Introduction & Quick Start
• Core Concepts
  • Nodes
  • Pipelines
  • Execution
• Data Processing
  • Mapping (Parallel Processing)
  • Nesting Pipelines
  • Caching
• Scaling & Optimization
  • Engines Overview
  • Distributed Daft Engine
  • Stateful Parameters
• Observability
  • Visualization
  • Telemetry & Progress

---

🚀 Quick Start

Installation

pip install hypernodes
# or with uv
uv add hypernodes

Basic Example

from hypernodes import Pipeline, node

# 1. Define functions on a single item
@node(output_name="cleaned_text")
def clean_text(passage: str) -> str:
    return passage.strip().lower()

@node(output_name="word_count")
def count_words(cleaned_text: str) -> int:
    return len(cleaned_text.split())

# 2. Build pipeline
pipeline = Pipeline(nodes=[clean_text, count_words])

# 3. Test with single input
result = pipeline.run(inputs={"passage": "Hello World"})
print(result)  # {'cleaned_text': 'hello world', 'word_count': 2}

# 4. Scale with .map()
results = pipeline.map(
    inputs={"passage": ["Hello", "World", "Foo", "Bar"]},
    map_over="passage",
)

High-Performance Execution with Daft

from hypernodes import Pipeline
from hypernodes.engines import DaftEngine

# Distributed execution using Daft
# Requires: pip install getdaft
engine = DaftEngine(use_batch_udf=True)
pipeline = Pipeline(nodes=[clean_text, count_words], engine=engine)

# Auto-batches and executes in parallel
# Each item is cached independently
results = pipeline.map(
    inputs={"passage": ["Hello", "World"] * 1000},
    map_over="passage"
)

---

📄 License

MIT License - see LICENSE file for details.