pytemplify 0.2.9

A generic generator framework using Jinja2 and user-defined parsers.

pytemplify

Data-driven code generation that safely regenerates files while preserving manual edits.

This README has been rewritten to be shorter, easier to scan, and to include two diagrams:

• Architecture: how components interact
• Data flow: from data -> templates -> generated files (with manual sections preserved)

Goals for this README

• concise install & quick-start
• minimal examples (CLI + Python API)
• two diagrams (Mermaid) to illustrate structure and flow
• a short "contract" and common edge-cases

Quick facts (contract)

• Inputs: JSON/YAML data + Jinja2 templates + optional helpers/config
• Outputs: Generated files in an output directory; existing manual sections are preserved
• Error modes: invalid template syntax, bad iterate expressions, missing data keys
• Success: files written with preserved manual sections and deterministic filtering

Install

pip install pytemplify

Optional tooling to install separately:

• Formatters: pip install black, npm install -g prettier, system clang-format
• Validation extras: install whatever your validators require (e.g., GTest)

Quick Start (CLI)

1. data (services.json)

{ "services": [ { "name": "user", "port": 8080 }, { "name": "order", "port": 8081 } ] }

2. template (templates/_foreach_service_{{ service.name }}_config.py.j2)

# {{ service.name | upper() }} config
SERVICE_NAME = "{{ service.name }}"
SERVICE_PORT = {{ service.port }}

# MANUAL SECTION START: extras
# add custom code here (preserved on regen)
# MANUAL SECTION END

3. config (config.yaml)

templates:
  - name: "Services"
    folder: "templates"
    output: "generated"
    iterate: "service in services"

4. run

yagen -c config.yaml -d services.json -o ./generated

Result: one file per service under generated/ with manual sections preserved.

Minimal Python example

from pytemplify.renderer import TemplateRenderer

data = {"project": "example", "services": [{"name": "a"}]}
renderer = TemplateRenderer(data)
renderer.generate("templates", "generated")

Diagrams

Architecture (components)

flowchart LR
  A[Input Data: JSON or YAML] --> B[Parser & DataHelpers]
  B --> C[TemplateRenderer - Jinja2, Helpers, Filters]
  C --> D[FileGenerator]
  D --> E["Output Files<br/>(manual sections preserved)"]
  subgraph CLI
    Y[Yagen]
  end
  Y --> C

Data flow for one template iteration

sequenceDiagram
  participant User
  participant Data
  participant Renderer
  participant FS as FileSystem

  User->>Data: provide services.json
  User->>Renderer: provide templates/ config
  Renderer->>Data: parse JSON/YAML
  Data->>Renderer: return parsed data
  Renderer->>FS: read template files
  FS->>Renderer: return template content
  Renderer->>Renderer: render templates with data
  Renderer->>FS: write generated files
  FS->>User: files written successfully

Features

• Template-based Code Generation: Use Jinja2 templates to generate files from data
• Data Helpers: Extend data with custom helper classes for complex transformations
• Manual Section Preservation: Keep user-edited sections intact between regenerations
• Manual Section Backup/Restore: Dedicated utility for backing up and restoring manual sections
• Flexible Configuration: YAML-based configuration with schema validation
• Configuration File Validation: Load and validate YAML/TOML/JSON config files with JSON Schema
• Code Formatting: Optional automatic formatting of generated files
• Validation Framework: Built-in validators (GTest, JSON Schema, File Structure) for generated code

Code Formatting

pytemplify supports optional code formatting for generated files. Install the formatter binaries you plan to use (Black/Prettier/clang-format are not bundled).

Configuration

Add a format section to your templates.yaml configuration:

format:
  enabled: true  # Enable/disable formatting globally

  defaults:
    preserve_manual_sections: true  # Keep manual sections intact during formatting
    ignore_patterns: ["*.min.*", "*.generated.*"]  # Files to skip formatting

  formatters:
    # Python files
    ".py":
      type: "black"
      enabled: true
      options:
        line_length: 88
        string_normalization: true

    # JavaScript/TypeScript files
    ".js|.ts|.jsx|.tsx":
      type: "prettier"
      options:
        printWidth: 80
        semi: true

    # C/C++ files (builtin formatter - recommended)
    ".c|.cpp|.h|.hpp":
      type: "cpp_format"
      options:
        style: "Google"
        indent_width: 2

    # Alternative: command-line clang-format
    # ".c|.cpp|.h|.hpp":
    #   type: "command"
    #   command: "clang-format ${input} > ${output}"
    #   options:
    #     timeout: 30

Supported Formatters

• Black: Python code formatter (pip install black)
• Prettier: JavaScript, TypeScript, JSON, CSS, HTML, Markdown formatter (npm install -g prettier)
• cpp_format: C/C++ formatter using clang-format (requires clang-format on PATH)
• clang-format: C/C++ formatter (requires system installation)
• Custom Command: Any command-line formatter using ${input} and ${output} placeholders

Manual Section Preservation

When preserve_manual_sections is enabled (default), manual sections in templates are preserved during formatting. The formatter temporarily removes manual sections, formats the code, and then restores the original manual sections.

Installation

Install the formatters you need:

pip install black            # Python
npm install -g prettier      # JS/TS/JSON/CSS/HTML/Markdown
# Install clang-format from your package manager for C/C++

Configuration File Validation

pytemplify supports loading and validating external configuration files (YAML, TOML, JSON) with JSON Schema validation. This is useful for complex projects that separate deployment settings, build configuration, and other data from the main template configuration.

Configuration

Add extra_data to your config.yaml:

globals:
  version: "1.0.0"
  project: "MicroservicesApp"

extra_data:
  # YAML file with schema validation
  - key: "deployment"
    path: "configs/deployment.yaml"
    schema: "schemas/deployment-schema.json"
    format: "yaml"  # optional: auto-detected from extension
    description: "Deployment environment configuration"

  # TOML file (auto-detected)
  - key: "build"
    path: "configs/build.toml"
    description: "Build and packaging settings"

  # JSON file with schema validation
  - key: "database"
    path: "configs/database.json"
    schema: "schemas/database-schema.json"
    description: "Database connection configuration"

Features

• Multiple Format Support: JSON, YAML (.yaml/.yml), TOML
• JSON Schema Validation: Validate config files against JSON Schema (Draft-07)
• Cross-Platform Paths: Works on Windows and Linux with automatic path resolution
• Clickable Error URIs: Error messages include file:// URIs that are clickable in VSCode, PyCharm, and modern terminals
• Environment Variables: Supports ${VAR} (Unix) and %VAR% (Windows) in paths
• Optional Files: Use required: false for optional configuration files

Error Messages

When schema validation fails, you get detailed, actionable errors:

TemplateConfigError: Schema validation failed for extra data key 'deployment':
  - Extra data key 'deployment': at 'environments.0.replicas': -1 is less than the minimum of 1
    ↳ file:///home/user/project/configs/deployment.yaml:5

Click the file:// link to jump directly to the error location in your IDE!

Example

See examples/with_schema_validation/ for a complete working example.

Python API

Basic Usage

from pytemplify.renderer import TemplateRenderer

# Your data (dict, JSON, or custom object)
data = {
    "project_name": "MyApp",
    "services": [
        {"name": "auth", "port": 8080},
        {"name": "api", "port": 8081}
    ]
}

# Create renderer with data
renderer = TemplateRenderer(data)

# Render a string template
template = "Project: {{ project_name }}"
result = renderer.render_string(template)
print(result)  # "Project: MyApp"

# Render a file template
renderer.render_file("template.txt.j2", "output.txt")

# Generate entire directory structure
renderer.generate("templates/", "output/")

Custom Filters

def reverse_string(text):
    return text[::-1]

custom_filters = {"reverse": reverse_string}
renderer = TemplateRenderer(data, filters=custom_filters)

template = "{{ 'hello' | reverse }}"  # "olleh"

# Disable auto-registration of built-in filters if needed
renderer = TemplateRenderer(data, auto_register_filters=False)

Environment Configuration

Configure Jinja2 environment options for template rendering:

data = {"items": ["a", "b", "c"]}
renderer = TemplateRenderer(data)

# Configure multiple environment options at once
renderer.set_env_options(
    trim_blocks=True,      # Remove first newline after template tag
    lstrip_blocks=True,    # Strip leading spaces/tabs before block tags
    autoescape=False       # Disable automatic HTML escaping
)

# Template with better whitespace control
template = """
{% for item in items %}
  - {{ item }}
{% endfor %}
"""

result = renderer.render_string(template)
# Output with clean formatting (no extra blank lines)

Common environment options:

• trim_blocks: Remove first newline after template tag
• lstrip_blocks: Strip leading spaces/tabs from start of line to block
• autoescape: Enable automatic escaping of variables
• keep_trailing_newline: Keep trailing newline at end of template

Manual Sections

Manual sections allow you to preserve user-modified content between template regenerations:

template = """
# Configuration
project_name = {{ project_name }}

# Custom settings
MANUAL SECTION START: custom_settings
# Add your custom settings here
debug = True
MANUAL SECTION END
"""

# Previously rendered content with user modifications
previous = """
# Configuration
project_name = OldProject

# Custom settings
MANUAL SECTION START: custom_settings
# User added these settings
debug = True
verbose = True
log_level = DEBUG
MANUAL SECTION END
"""

# The manual section will be preserved in the new render
result = renderer.render_string(template, previous)

Manual Section Backup/Restore Utility

For template development workflows, pytemplify includes a dedicated utility for backing up and restoring manual sections:

# Backup manual sections from generated files
manual-sections backup ./generated/ --recursive --output backup.json

# Preview what would be restored
manual-sections restore backup.json ./target/ --preview

# Restore with section mapping (useful when template structure changes)
manual-sections restore backup.json ./target/ --section-map "old_section:new_section"

# Generate human-readable reports
manual-sections report backup.json --output report.md

# View specific sections
manual-sections view backup.json --file "utils.py" --section "helpers"

This utility is essential when:

• Refactoring templates and need to migrate manual sections
• Working with multiple template versions
• Sharing manual sections between team members
• Creating backups before major template changes

Content Injection

You can inject content into specific parts of existing files:

template = """
<!-- injection-pattern: imports -->
pattern: (?P<injection>import .*)
<!-- injection-string-start -->
import os
import sys
import json
<!-- injection-string-end -->
"""

existing_file = """
import os
import sys
# Other code here
"""

# Will inject the new import statements
result = renderer.inject_string(template, existing_file)

Template Directory Generation

Generate an entire directory structure from templates:

# Will process all .j2 files in templates/ and generate the corresponding
# structure in output/ with rendered content
renderer.generate(Path("templates"), Path("output"))

For more details, see the API documentation in the code.

YAML-based Generation (yagen)

The YAML-based generator (yagen) is the recommended approach for most use cases. It provides a declarative, powerful way to generate code from data without writing custom Python parsers.

Why Choose yagen?

✨ No Code Required: Define everything in YAML config + Jinja2 templates 🔄 Data Helpers: Extend your data with computed properties 🎯 Advanced Iteration: Filter, nest, and iterate with expressive syntax 🎨 Flexible Output: Multiple template sets, conditional generation 📦 Production Ready: Schema validation, error handling, CLI interface

Core Features

• Multiple Iteration Patterns: Simple, conditional, nested, and array iterations
• Template Filtering: Include/exclude patterns with glob and regex support (CLI and config-level)
• Data Helpers Integration: Add computed properties to your data
• Manual Section Preservation: Regenerate safely without losing custom code
• Advanced Configuration: YAML-based template sets with flexible output control
• Schema Validation: Built-in validation for template configurations
• Template Organization: Automatic file filtering based on iteration context

Basic Usage

1. Create a YAML configuration file (templates.yaml):

globals:
  version: "1.0.0"
  project: "MyProject"

templates:
  - name: "Service Files"
    folder: "service_templates"
    output: "services"
    iterate: "service in dd.services"
    enabled: true

2. Create your JSON data file (data.json):

{
  "services": [
    {"name": "user-service", "port": 8080},
    {"name": "auth-service", "port": 8081}
  ]
}

3. Run the generator:

yagen --config templates.yaml --data data.json

Advanced Iteration Patterns

Simple Iteration

iterate: "service in dd.services"

Conditional Iteration

iterate: "endpoint in dd.endpoints if endpoint.enabled"

Nested Iteration

iterate: "module in dd.modules >> component in module.components"

Array Iteration (Multiple patterns)

iterate:
  - "service in dd.services if service.public"
  - "module in dd.modules if module.documented"

Template Filtering

Control which templates are processed using command-line filters:

# Include only service-related templates
yagen --config templates.yaml --data data.json --output ./output \
      --include "Service*"

# Exclude test templates
yagen --config templates.yaml --data data.json --output ./output \
      --exclude "*Test*"

# Use regex patterns
yagen --config templates.yaml --data data.json --output ./output \
      --include "regex:^(Service|API).*"

Template Organization

Use _foreach_ prefixes to organize templates by iteration context:

templates/
├── _foreach_service_main.py.j2     # Only processed during service iteration
├── _foreach_module_init.py.j2      # Only processed during module iteration
└── static_config.py.j2             # Always processed

Complete Examples

See the examples/ directory for comprehensive examples including:

• Static template generation
• Service and API file generation
• Nested module/component structures
• Documentation generation
• Test file generation with filtering

Data Helpers Integration

Yagen supports data_helpers for enhanced template capabilities. Data helpers add computed properties and methods to your JSON data without modifying the original structure.

Quick Start with Data Helpers

# Using CLI arguments
yagen --config templates.yaml --data data.json \
      --helpers "my_helpers.CompanyHelpers" \
      --helper-path "./helpers/"

# Using YAML configuration
yagen --config templates_with_helpers.yaml --data data.json

YAML Configuration with Data Helpers

# templates.yaml
globals:
  version: "1.0.0"
  project: "MyProject"

# Load additional data from external JSON files or inline dictionaries
extra_data:
  - path: "infrastructure.json"
    key: "infra"
  - path: "secrets.json"
    key: "secrets"
    required: false  # Optional file

  # Inline dictionary data
  - value:
      database:
        host: "localhost"
        port: 5432
        name: "myapp"
      redis:
        host: "localhost"
        port: 6379
    key: "config"
    description: "Application configuration settings"

  - value:
      api_version: "v2"
      features:
        auth: true
        caching: false
    key: "constants"
    description: "API version and feature flags"

# Enhanced with data_helpers
data_helpers:
  helpers:
    - "my_helpers.CompanyHelpers"
    - "my_helpers.EmployeeHelpers"
  discovery_paths:
    - "./helpers/"

templates:
  - name: "Company Report"
    folder: "company_templates"
    output: "reports"
    # Templates can now use helper properties and extra data

Creating Data Helpers

# helpers/company_helpers.py
from pytemplify.data_helpers import DataHelper

class CompanyHelpers(DataHelper):
    @staticmethod
    def matches(data: dict) -> bool:
        return "company_name" in data and "departments" in data

    @property
    def total_employees(self) -> int:
        """Calculate total employees across all departments."""
        return sum(len(dept._data.get("employees", []))
                  for dept in self._data.departments)

    @property
    def company_summary(self) -> str:
        return f"{self._data.company_name}: {self.total_employees} employees"

Enhanced Templates

<!-- company_templates/overview.md.j2 -->
# {{ dd.company_name }} Overview

**Total Employees:** {{ dd.total_employees }}
**Summary:** {{ dd.company_summary }}

## Infrastructure
**Database Host:** {{ infra.database.host }}
**Redis Port:** {{ infra.redis.port }}

## Configuration
**DB Host:** {{ config.database.host }}
**DB Port:** {{ config.database.port }}
**API Version:** {{ constants.api_version }}

## Departments
{% for dept in dd.departments %}
- **{{ dept.department_name }}**: {{ dept.employee_count }} employees
  - Manager: {{ dept.manager_name }}
  {% if dept.is_large_department %}
  - 🏢 Large Department
  {% endif %}
{% endfor %}

Helper Loading Methods

1. CLI Arguments (highest precedence)
2. YAML Configuration
3. Auto-discovery from specified paths

# Multiple specification methods
yagen --config templates.yaml --data data.json \
      --helpers "company.helpers.CompanyHelpers" \
      --helpers "shared.utilities" \
      --helper-path "./custom_helpers/"

For complete documentation and examples:

• examples/DATA_HELPERS_GUIDE.md - Comprehensive guide
• examples/helpers/company_helpers.py - Example helpers
• examples/templates_with_helpers.yaml - Configuration example
• examples/data_with_helpers.json - Sample data

Command Reference

yagen --help                         # Show all options
yagen -c templates.yaml -d data.json -o output    # Basic generation
yagen --config templates.yaml --data data.json --output output --verbose  # Verbose output

# With data helpers
yagen -c templates.yaml -d data.json --helpers "my.helpers" --helper-path "./helpers/"

Documentation

📚 Complete documentation available in docs/ directory (10+ comprehensive guides)

Getting Started

• Getting Started - Installation, quick start, and core concepts
• Examples - 5+ working examples (basic, advanced, with helpers, etc.)

Core Features

• YAGEN Guide - YAML-based generator (recommended approach)
• Template Guide - Complete Jinja2 template writing guide
• Filters Reference - All 70+ built-in filters documented
• Data Helpers Guide - Add computed properties to data
• Formatting Guide - Automatic code formatting (Black, Prettier, etc.)

API Reference

• API Reference - Complete Python API documentation

For Contributors

• Documentation Hub - Navigate all documentation

TIPs

Activate uv virtual environment:

source .venv/bin/activate

Deactivate uv virtual environment:

deactivate