pysmartyaml 1.0.0b2

Extended YAML format with custom directives for imports, environment variables, and more

SmartYAML

A powerful YAML processing library that extends standard YAML parsing with advanced features for templating, variable substitution, environment variable integration, conditional logic, and schema validation. SmartYAML processes input YAML files through a comprehensive pipeline to resolve all special constructs and output plain YAML data structures.

Features

🔧 Core Processing Pipeline

• 6-Stage Processing: parsing → metadata → templates → directives → variables → validation
• Recursive Processing: Depth-first resolution of nested structures and includes
• Version Compatibility: Built-in __version checking for future-proofing
• Error Context: Comprehensive error reporting with file/field location details

📝 Metadata System

• __vars: Variable definitions with inheritance and precedence
• __template: Powerful template inheritance and overlay system
• __schema: JSON Schema validation of final resolved YAML
• __version: Version compatibility checking
• Auto-removal: All __* metadata fields automatically removed from final output

🎯 Directive System

• Environment Variables: !env, !env_int, !env_float, !env_bool, !secret
• File Operations: !include, !include_if, !include_yaml, !include_yaml_if
• Template Loading: !template, !template_if
• Data Manipulation: !merge, !concat
• Variable Operations: !var, !expand
• Conditionals: !if, !switch

🔒 Security Features

• Path Sanitization: Directory traversal protection
• File Size Limits: Configurable limits (default 10MB)
• Recursion Protection: Prevents infinite recursion and circular imports
• Environment Controls: Whitelist/blacklist for environment variables
• Sandbox Mode: Restricts file and environment access
• No Code Execution: Safe YAML parsing only

Installation

From PyPI (Recommended)

pip install pysmartyaml

From GitHub Repository

# Install latest from main branch
pip install git+https://github.com/apuigsech/smartyaml.git

# Install specific version/tag
pip install git+https://github.com/apuigsech/smartyaml.git@v1.0.0-alpha-1

# Clone and install for development
git clone https://github.com/apuigsech/smartyaml.git
cd smartyaml
pip install -e ".[dev]"

Quick Start

import smartyaml

# Basic loading
data = smartyaml.load("config.yaml")

# Load with template support
data = smartyaml.load("config.yaml", template_path="templates")

# Load with variables and full configuration
variables = {"environment": "production", "version": "2.0.0"}
data = smartyaml.load('config.yaml',
                     base_path='/custom/path',
                     template_path='/templates',
                     variables=variables,
                     max_file_size=5*1024*1024)

# Load from string content
yaml_content = """
__vars:
  app: "MyApp"
  env: !env ['ENVIRONMENT', 'development']

__template:
  path: 'templates/base.yaml'
  overlay: true

database:
  name: !expand "{{app}}_{{env}}_db"
  host: !env ['DB_HOST', 'localhost']

features: !include_yaml_if ['ENABLE_FEATURES', 'features.yaml']
"""
data = smartyaml.loads(yaml_content, template_path="templates")

Complete Example

# config.yaml - Comprehensive SmartYAML demonstration
__version: "1.0.0"

# Variables with environment integration
__vars:
  app_name: "MyApplication"
  environment: !env ['ENVIRONMENT', 'development']
  version: !env ['APP_VERSION', '1.0.0']
  debug_mode: !env ['DEBUG', false]

# Template inheritance
__template:
  use: 'apps.base_config'  # Loads from templates/apps/base_config.yaml
  overlay: true           # Merge template with current content

# JSON Schema validation
__schema:
  type: object
  properties:
    app:
      type: object
      required: [name, version]
    database:
      type: object
      required: [host, name]
  required: [app, database]

# Application configuration with variable operations
app:
  name: !var "app_name"                                      # Direct string variable
  full_title: !expand "{{app_name}} v{{version}}"            # String templating
  environment: !var "environment"                            # Direct string variable
  debug: !var "debug_mode"                                   # Direct boolean variable
  api_url: !expand "https://api-{{environment}}.example.com" # String templating

# Database with conditional configuration
database: !merge
  - !include_yaml 'config/database_base.yaml'
  - host: !env ['DB_HOST', 'localhost']
    name: !expand "{{app_name}}_{{environment}}"         # String templating for name
    password: !env ['DB_PASSWORD']

# Environment-specific features
logging: !include_yaml_if ['DEBUG', 'config/debug_logging.yaml']
monitoring: !include_yaml_if ['PRODUCTION', 'config/monitoring.yaml']

# External content inclusion
documentation: !include 'docs/api_guide.md'
queries:
  users: !include 'sql/select_users.sql'

# Conditional service configuration
services: !switch ['DEPLOYMENT_TYPE']
  - case: 'kubernetes'
    deployment: !include_yaml 'k8s/deployment.yaml'
  - case: 'docker'
    deployment: !include_yaml 'docker/compose.yaml'
  - default: 'standalone'
    deployment: !include_yaml 'config/standalone.yaml'

Metadata Fields Reference

SmartYAML recognizes only specific metadata fields that have special processing behavior. All other fields starting with __ are custom metadata and are ignored.

Recognized Metadata Fields

• __version: Version compatibility checking
• __vars: Variable definitions (available for substitution)
• __template: Template inheritance and overlay configuration
• __schema: JSON Schema validation rules

Custom Metadata Fields

Any other field starting with __ (like __author, __created, __description) is custom metadata and will be removed from the final output.

Example:

# Recognized metadata - these work as expected
__vars:
  app_name: "MyApp"
__version: "1.0.0"

# Custom metadata - these are ignored
__author: "John Doe"
__created: "2024-01-01"

app:
  name: "{{app_name}}"

__version

Version compatibility checking to ensure YAML files work with library version.

__version: "1.0.0"  # Requires SmartYAML v1.0.0+

__vars

Variable definitions with inheritance support. Variables can contain directives.

__vars:
  app_name: "MyApp"
  environment: !env ['ENVIRONMENT', 'development']
  db_host: !env ['DB_HOST', 'localhost']
  connection_string: !expand "postgresql://{{db_host}}/{{app_name}}"

# Usage with expansion
database_url: !expand "{{connection_string}}_{{environment}}"

Variable Precedence (highest to lowest):

1. Function parameters: smartyaml.load(file, variables={...})
2. Document __vars: In the main YAML file
3. Template __vars: From inherited templates

__template

Template inheritance and overlay system.

__template:
  path: 'templates/base.yaml'     # Direct file path
  use: 'apps.microservice'        # Template name (loads from template_path)
  overlay: true                   # true=merge, false=replace (default: true)

Template Inheritance Chain:

# templates/base.yaml
__vars:
  timeout: 30
  log_level: "INFO"

config:
  timeout: !expand "{{timeout}}"
  logging:
    level: !expand "{{log_level}}"

# service.yaml
__vars:
  service_name: "UserAPI"
  timeout: 60  # Overrides template value

__template:
  use: 'base'

# Service-specific additions
service:
  name: !expand "{{service_name}}"
  # Inherits config.timeout=60, config.logging.level="INFO"

__schema

JSON Schema validation of the final resolved YAML.

__schema:
  type: object
  properties:
    app:
      type: object
      properties:
        name:
          type: string
        version:
          type: string
          pattern: "^\\d+\\.\\d+\\.\\d+$"
      required: [name, version]
    database:
      type: object
      properties:
        host: {type: string}
        port: {type: integer, minimum: 1, maximum: 65535}
      required: [host, port]
  required: [app, database]

# Schema composition with directives
__schema: !merge
  - !include_yaml 'schemas/common.yaml'
  - !include_yaml 'schemas/app_specific.yaml'

Directives Reference

Environment Variables

!env ['VAR_NAME', 'default']

Access environment variables as strings with optional defaults. Environment variables are always returned as strings.

database_host: !env ['DB_HOST', 'localhost']
port: !env ['PORT', '8080']          # Returns string '8080'
debug_mode: !env ['DEBUG', 'false']  # Returns string 'false'

!env_int ['VAR_NAME', default_int]

Access environment variables and convert to integers.

port: !env_int ['PORT', 8080]              # Returns integer 8080
max_connections: !env_int ['MAX_CONN', 100]  # Returns integer 100
workers: !env_int ['WORKERS', 4]            # Returns integer 4

!env_float ['VAR_NAME', default_float]

Access environment variables and convert to floats.

timeout: !env_float ['TIMEOUT', 30.5]       # Returns float 30.5
cpu_limit: !env_float ['CPU_LIMIT', 1.0]    # Returns float 1.0
memory_ratio: !env_float ['MEM_RATIO', 0.8] # Returns float 0.8

!env_bool ['VAR_NAME', default_bool]

Access environment variables and convert to booleans. Accepts: true/false, yes/no, 1/0 (case-insensitive).

debug_mode: !env_bool ['DEBUG', false]      # Returns boolean false
ssl_enabled: !env_bool ['SSL_ENABLE', true] # Returns boolean true
auto_deploy: !env_bool ['AUTO_DEPLOY', false] # Returns boolean false

!secret ['SECRET_NAME', 'default']

Same as !env - returns strings (future versions will support secure secret stores).

api_key: !secret ['API_KEY', 'dev_key_123']
jwt_secret: !secret ['JWT_SECRET', 'fallback_secret']

File Operations

!include 'filename'

Include file content as string (supports both text and YAML files).

# Text files loaded as strings
sql_query: !include 'queries/users.sql'
html_template: !include 'templates/email.html'

# YAML files processed through full SmartYAML pipeline
config: !include 'config/database.yaml'

!include_if ['CONDITION', 'filename']

Conditional file inclusion based on environment variable.

debug_config: !include_if ['DEBUG_MODE', 'debug.yaml']
prod_settings: !include_if ['PRODUCTION', 'prod_overrides.yaml']

Truthy values: true, 1, yes, on, enabled (case-insensitive)

!include_yaml 'filename'

Include raw YAML content without processing SmartYAML directives.

# Loads YAML but doesn't process any !directives within it
external_config: !include_yaml 'third_party/config.yaml'

!include_yaml_if ['CONDITION', 'filename']

Conditional raw YAML inclusion.

legacy_config: !include_yaml_if ['USE_LEGACY', 'legacy/config.yaml']

Template System

!template 'template_name'

Load templates from template directory using dot notation.

# Loads templates/database/postgres.yaml
database: !template 'database.postgres'

# Loads templates/services/api.yaml
api_service: !template 'services.api'

!template_if ['CONDITION', 'template_name']

Conditional template loading.

cache: !template_if ['ENABLE_CACHE', 'components.redis']
monitoring: !template_if ['PRODUCTION', 'observability.prometheus']

Data Operations

!merge [item1, item2, ...]

Deep merge multiple data structures (objects/arrays).

# Merge multiple configuration sources
database_config: !merge
  - !include_yaml 'config/db_base.yaml'
  - !include_yaml 'config/db_env_overrides.yaml'
  - host: !env ['DB_HOST', 'localhost']
    pool_size: !env ['DB_POOL_SIZE', 10]

# Merge template with local overrides
service_config: !merge
  - !template 'services.base'
  - name: 'user-service'
    replicas: 3

!concat [item1, item2, ...]

Concatenate arrays/lists.

# Combine multiple lists
all_endpoints: !concat
  - !include_yaml 'api/public_endpoints.yaml'
  - !include_yaml 'api/admin_endpoints.yaml'
  - ['/health', '/metrics']  # Additional endpoints

# Dynamic list building
middleware: !concat
  - ['cors', 'logging']
  - !include_yaml_if ['DEVELOPMENT', 'middleware/dev.yaml']
  - !include_yaml_if ['PRODUCTION', 'middleware/prod.yaml']

Variable Operations

!var 'variable_name'

Direct variable resolution with original type preservation. Unlike !expand, this returns the variable value in its original data type.

__vars:
  service_name: "user-api"        # string
  port: 8080                      # integer
  ssl_enabled: true               # boolean
  timeout: 30.5                   # float
  nullable_setting: null          # null

# Direct variable access (preserves original types)
app:
  name: !var "service_name"       # Returns string: "user-api"
  port: !var "port"               # Returns integer: 8080
  ssl: !var "ssl_enabled"         # Returns boolean: true
  timeout: !var "timeout"         # Returns float: 30.5
  setting: !var "nullable_setting" # Returns null

# Supports default values with pipe syntax
database:
  host: !var "db_host|localhost"         # String default
  port: !var "db_port|5432"              # Integer default
  ssl: !var "db_ssl|true"                # Boolean default

# Supports nested variable access with dot notation
config:
  db_host: !var "database.host"
  first_service: !var "services.0.name"

!expand 'text with {{variables}}'

String expansion with variable substitution. Always returns strings with template processing.

__vars:
  service_name: "user-api"
  environment: !env ['ENVIRONMENT', 'dev']
  version: !env ['VERSION', '1.0.0']
  port: 8080                      # integer

# String template expansion (always returns strings)
app_title: !expand "{{service_name}} v{{version}}"     # Returns: "user-api v1.0.0"
api_endpoint: !expand "https://{{service_name}}-{{environment}}.example.com"
connection_string: !expand "postgresql://{{db_host}}/{{service_name}}_{{environment}}"
port_string: !expand "{{port}}"                        # Returns: "8080" (string)

# Complex template processing
container_name: !expand "{{service_name}}-{{environment}}-{{version}}"
log_path: !expand "/var/log/{{service_name}}/{{environment}}.log"

Key Differences:

• !var "port" → 8080 (integer preserved)
• !expand "{{port}}" → "8080" (string result)
• !var is for direct variable access with type preservation
• !expand is for string templating with variable substitution

Conditionals

!if ['ENV_VAR', value]

Conditional field inclusion.

# Include field only if DEBUG is truthy
debug_panel: !if ['DEBUG', {'enabled': true, 'level': 'verbose'}]

# Conditional with complex value
development_tools: !if ['DEVELOPMENT']
  webpack_dev_server: true
  hot_reload: true
  source_maps: true

!switch ['ENV_VAR', cases]

Multi-way conditional based on environment variable value.

database: !switch ['DATABASE_TYPE']
  - case: 'postgres'
    driver: 'postgresql+psycopg2'
    port: 5432
    pool_size: 20
  - case: 'mysql'
    driver: 'mysql+pymysql'
    port: 3306
    pool_size: 15
  - case: 'sqlite'
    driver: 'sqlite'
    file: 'app.db'
  - default: 'postgres'
    driver: 'postgresql+psycopg2'
    port: 5432

# Service configuration based on deployment type
deployment: !switch ['DEPLOYMENT']
  - case: 'kubernetes'
    type: 'k8s'
    config: !include_yaml 'k8s/deployment.yaml'
  - case: 'docker'
    type: 'container'
    config: !include_yaml 'docker/compose.yaml'
  - default: 'standalone'
    type: 'process'
    config: !include_yaml 'config/standalone.yaml'

Advanced Examples

Multi-Service Configuration

# services.yaml - Configure multiple services with shared templates
__vars:
  company: "ACME Corp"
  environment: !env ['ENVIRONMENT', 'development']
  version: !env ['VERSION', '1.0.0']

__template:
  use: 'infrastructure.base'

# API Gateway
gateway: !merge
  - !template 'services.api_gateway'
  - name: !expand "{{company}}-gateway"
    version: !expand "{{version}}"
    config:
      upstream_services: !concat
        - !include_yaml 'services/core_services.yaml'
        - !include_yaml_if ['ENABLE_ANALYTICS', 'services/analytics.yaml']

# User Service
user_service: !merge
  - !template 'services.microservice'
  - name: 'user-service'
    database: !switch ['USER_DB_TYPE']
      - case: 'postgres'
        config: !template 'databases.postgres'
      - case: 'mongodb'
        config: !template 'databases.mongodb'
      - default: 'sqlite'
        config: !template 'databases.sqlite'

# Environment-specific overrides
overrides: !include_yaml_if ['PRODUCTION', 'config/production_overrides.yaml']

Configuration with Schema Validation

# app_config.yaml - Production application configuration
__version: "1.0.0"

__vars:
  app_name: "ProductionApp"
  environment: "production"
  replica_count: !env ['REPLICAS', 3]

__schema:
  type: object
  properties:
    application:
      type: object
      properties:
        name: {type: string, minLength: 1}
        replicas: {type: integer, minimum: 1, maximum: 100}
        database:
          type: object
          properties:
            host: {type: string}
            port: {type: integer, minimum: 1, maximum: 65535}
            ssl: {type: boolean}
          required: [host, port]
      required: [name, replicas, database]
  required: [application]

application:
  name: !var "app_name"                                    # String variable
  replicas: !var "replica_count"                          # Integer variable (preserves type for schema validation)
  database:
    host: !env ['DB_HOST']  # Required - will fail validation if not set
    port: !env_int ['DB_PORT', 5432]                       # Integer from environment
    ssl: !env_bool ['DB_SSL', true]                        # Boolean from environment
    connection_string: !expand "postgresql://{{database.host}}:{{database.port}}/{{app_name}}"

Configuration API

SmartYAML provides a fluent configuration API:

import smartyaml
from smartyaml.config import SmartYAMLConfigBuilder

# Basic configuration
config = SmartYAMLConfigBuilder().build()

# Development preset
config = SmartYAMLConfigBuilder().development_mode().build()

# Production preset
config = SmartYAMLConfigBuilder().production_mode().build()

# Custom configuration
config = (SmartYAMLConfigBuilder()
          .with_template_path("./templates")
          .with_max_file_size(50 * 1024 * 1024)  # 50MB
          .with_security_options(sandbox_mode=True)
          .with_allowed_env_vars(['APP_*', 'DB_*'])
          .build())

# Use with load functions
data = smartyaml.load("config.yaml", config=config)

Error Handling

SmartYAML provides comprehensive error reporting:

import smartyaml
from smartyaml import (
    SmartYAMLError, VersionMismatchError, FileNotFoundError,
    DirectiveSyntaxError, VariableNotFoundError, SchemaValidationError
)

try:
    data = smartyaml.load("config.yaml")
except VersionMismatchError as e:
    print(f"Version incompatible: {e}")
except FileNotFoundError as e:
    print(f"Missing file: {e}")
except DirectiveSyntaxError as e:
    print(f"Invalid directive syntax: {e}")
except SchemaValidationError as e:
    print(f"Schema validation failed: {e}")
except SmartYAMLError as e:
    print(f"SmartYAML error: {e}")

All errors include:

• File context: Which file caused the error
• Field context: Which field/path in the YAML
• Operation context: What operation was being performed
• Detailed messages: Clear explanation of the issue

Security Considerations

SmartYAML implements multiple security layers:

Safe Defaults

• File size limits: 10MB default (configurable)
• Recursion limits: 20 levels max (configurable)
• Path validation: Prevents directory traversal
• No code execution: Only safe YAML parsing

Environment Variable Controls

# Whitelist specific variables
config = SmartYAMLConfigBuilder().with_allowed_env_vars([
    'APP_NAME', 'DB_HOST', 'API_*'
]).build()

# Blacklist sensitive variables
config = SmartYAMLConfigBuilder().with_forbidden_env_vars([
    'SECRET_KEY', 'PRIVATE_KEY', '*_PASSWORD'
]).build()

Sandbox Mode

# Restricts file access and environment variables
config = SmartYAMLConfigBuilder().with_security_options(
    sandbox_mode=True,
    strict_security=True
).build()

Development

Testing

python -m pytest                    # Run all tests (230+)
python -m pytest --cov=smartyaml    # Run with coverage
python -m pytest -v                 # Verbose output

Code Quality

black smartyaml/          # Format code
isort smartyaml/          # Sort imports
flake8 smartyaml/         # Lint code
mypy smartyaml/           # Type check

Building

python -m build          # Build packages
pip install -e ".[dev]"  # Development install

Requirements

• Python: 3.9+ (3.7+ for basic usage)
• Dependencies: PyYAML 5.1+, minimal external dependencies
• Optional: jsonschema for schema validation

Compatibility

SmartYAML files are valid YAML files. Standard YAML parsers will treat custom directives as regular tagged values, making the format backward-compatible.

License

MIT License - see LICENSE file for details.

Contributing

Contributions welcome! Please see CONTRIBUTING.md for guidelines.

---

SmartYAML: Making YAML configurations smarter, more powerful, and easier to maintain.