j-perm 1.2.0

json permutation library

J-Perm

A composable JSON transformation DSL with a powerful, extensible architecture.

J-Perm lets you describe data transformations as executable specifications — a list of steps that can be applied to input documents. It supports JSON Pointer addressing, template interpolation with ${...} syntax, special constructs ($ref, $eval, $and, $or, $not), and a rich set of built-in operations.

---

Quick Example

from j_perm import build_default_engine

engine = build_default_engine()

# Source data
source = {
    "users": [
        {"name": "Alice", "age": "17"},
        {"name": "Bob", "age": "22"}
    ]
}

# Transformation spec (using shorthands)
spec = {
    "op": "foreach",
    "in": "/users",
    "do": {
        "/adults[]": {
            "$eval": [
                {"/name": "/item/name", "/age": "${int:/item/age}"},
                {"op": "if", "path": "/age", "cond": "${?dest.age >= `18`}", "else": {"~delete": "/age"}}
            ]
        }
    }
}

result = engine.apply(spec, source=source, dest={})
# → {"adults": [{"name": "Bob", "age": 22}]}

---

Installation

pip install j-perm

(or copy the package into your project)

---

Architecture Overview

J-Perm is built on a pipeline architecture with two main levels:

┌─────────────────────────────────────────────────────────┐
│  spec (user input)                                      │
│    │                                                    │
│    ▼                                                    │
│  ┌──────────────────────────────────────────────────┐   │
│  │ STAGES (batch preprocessing, priority order)     │   │
│  │  • ShorthandExpansion → expand ~delete, etc      │   │
│  │  • YourCustomStage                               │   │
│  └──────────────────────────────────────────────────┘   │
│    │                                                    │
│    ▼                                                    │
│  List[step]                                             │
│    │                                                    │
│    ▼  for each step:                                    │
│  ┌──────────────────────────────────────────────────┐   │
│  │ MIDDLEWARES (per-step, priority order)           │   │
│  │  • Validation, logging, etc.                     │   │
│  └──────────────────────────────────────────────────┘   │
│    │                                                    │
│    ▼                                                    │
│  ┌──────────────────────────────────────────────────┐   │
│  │ REGISTRY (hierarchical dispatch tree)            │   │
│  │  • SetHandler, CopyHandler, ForeachHandler, ...  │   │
│  └──────────────────────────────────────────────────┘   │
│    │                                                    │
│    │  handlers call ctx.engine.process_value(...)       │
│    └─────────────────────────────────────┐              │
│                                          ▼              │
│  ┌──────────────────────────────────────────────────┐   │
│  │ VALUE PIPELINE (stabilization loop)              │   │
│  │  • SpecialResolveHandler ($ref, $eval)           │   │
│  │  • TemplSubstHandler (${...})                    │   │
│  │  • RecursiveDescentHandler (containers)          │   │
│  │  • IdentityHandler (scalars)                     │   │
│  └──────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────┘

Core Components

Component	Purpose
Engine	Orchestrates pipelines, manages context, runs stabilization loop
Pipeline	Runs stages → middlewares → registry dispatch for each step
StageRegistry	Tree of batch preprocessors (run-all, priority order)
ActionTypeRegistry	Tree of action handlers (first-match or run-all)
ValueResolver	Abstraction for addressing (JSON Pointer implementation)

---

Core API

Building an Engine

from j_perm import build_default_engine

# Default engine with all built-ins
engine = build_default_engine()

# Custom specials (None = use defaults: $ref, $eval, $and, $or, $not)
engine = build_default_engine(
    specials={"$ref": my_ref_handler, "$custom": my_handler},
    casters={"int": lambda x: int(x), "json": lambda x: json.loads(x)},
    jmes_options=jmespath.Options(custom_functions=CustomFunctions())
)

Applying Transformations

result = engine.apply(
    spec,  # DSL script (dict or list)
    source=source,  # Source context (for pointers, templates)
    dest=dest,  # Initial destination (default: {})
)

Returns: Deep copy of the final dest after all transformations.

---

Features

1. JSON Pointer Addressing

J-Perm uses RFC 6901 JSON Pointer with extensions:

from j_perm import PointerResolver

resolver = PointerResolver()

# Basic pointers
resolver.get("/users/0/name", data)  # → "Alice"

# Root references (work on scalars too!)
resolver.get(".", 42)  # → 42
resolver.get("/", "text")  # → "text"

# Parent navigation
resolver.get("/a/b/../c", data)  # → data["a"]["c"]

# List slices
resolver.get("/items[1:3]", data)  # → [item1, item2]

# Append notation
resolver.set("/items/-", data, "new")  # Append to list

Key feature: Unlike standard JSON Pointer, PointerResolver works on any type (scalars, lists, dicts) for root references.

Data Source Prefixes

J-Perm supports prefixes to specify which context to read from:

Prefix	Source	Description
/path	source	Default - read from source context
@:/path	dest	Read from destination being built
_:/path	metadata	Read from execution metadata

Example: Accessing destination in templates

# Build incrementally, referencing previous values
spec = [
    {"/name": "Alice"},
    {"/greeting": "Hello, ${@:/name}!"}  # Reference dest value
]

result = engine.apply(spec, source={}, dest={})
# → {"name": "Alice", "greeting": "Hello, Alice!"}

Example: Comparing source vs dest

spec = [
    {"/dest_value": "from_dest"},
    {"/comparison": "Source: ${/source_value}, Dest: ${@:/dest_value}"}
]

result = engine.apply(
    spec,
    source={"source_value": "from_source"},
    dest={}
)
# → {"dest_value": "from_dest", "comparison": "Source: from_source, Dest: from_dest"}

Example: Using in conditionals

# Check destination state in conditions
spec = [
    {"/status": "ready"},
    {
        "op": "if",
        "path": "@:/status",
        "equals": "ready",
        "then": [{"/message": "System is ready"}]
    }
]

---

2. Template Interpolation (${...})

Templates are resolved by TemplSubstHandler in the value pipeline.

JSON Pointer lookup

"${/user/name}"  # → Resolve pointer from source

Type casters (built-in)

"${int:/age}"  # → int(value)
"${float:/price}"
"${bool:/flag}"  # → bool(int(value)) if int/str, else bool(value)
"${str:/id}"

JMESPath queries

"${?source.items[?price > `10`].name}"  # → Query source with JMESPath
"${?dest.total}"                         # → Query destination
"${?add(dest.x, source.y)}"              # → Mix source and dest

Built-in JMESPath functions: add(a, b), subtract(a, b)

Data structure: JMESPath expressions use explicit namespaces:

• source.* – access source document
• dest.* – access destination document

Nested templates

"${${/path_to_field}}"  # → Resolve inner template first

Escaping

$${ → ${ (literal)
$$  → $  (literal)

---

3. Special Constructs

Special values are resolved by SpecialResolveHandler.

$ref — Reference resolution

{
    "$ref": "/path/to/value",
    "$default": "fallback"
}

• Resolves pointer from source context
• Returns deep copy (no aliasing)
• Supports $default fallback

$eval — Nested evaluation

{
    "$eval": [
        {
            "op": "set",
            "path": "/x",
            "value": 1
        }
    ],
    "$select": "/x"
}

• Executes nested DSL with dest={}
• Optionally selects sub-path from result

$and — Logical AND with short-circuit

{
    "$and": [
        [{"op": "assert", "path": "/x", "return": true}],
        [{"op": "copy", "from": "/y", "path": "/"}]
    ]
}

• Executes actions in order with empty dest for each
• Returns last result if all are truthy
• Short-circuits and returns first falsy result

$or — Logical OR with short-circuit

{
    "$or": [
        [{"op": "assert", "path": "/x", "return": true}],
        [{"op": "copy", "from": "/y", "path": "/"}]
    ]
}

• Executes actions in order with empty dest for each
• Returns first truthy result
• Returns last result if all are falsy

$not — Logical negation

{
    "$not": [{"op": "assert", "path": "/missing", "return": true}]
}

• Executes action with empty dest
• Returns logical negation of the result

---

4. Functions and Error Handling

J-Perm supports defining reusable functions and controlled error handling.

$def — Define a function

{
    "$def": "myFunction",
    "params": ["arg1", "arg2"],
    "body": [
        {"/result": "${arg1}"},
        {"/total": "${int:${arg2}}"}
    ],
    "return": "/total",
    "on_failure": [
        {"/error": "Function failed"}
    ]
}

• params — list of parameter names (optional, default: [])
• body — actions to execute when function is called
• return — path in local context to return (optional, default: entire dest)
• on_failure — error handler actions (optional)

Special variable _: Inside function body, the original source is accessible via /_/path:

{
    "$def": "getFromSource",
    "body": [{"/value": {"$ref": "/_/data"}}]
}

When called, functions can access:

• Parameters via /${param_name}
• Original source via /_/path (source from engine.apply)
• Current dest via @:/path

Example:

spec = [
    {
        "$def": "combineData",
        "params": ["userInput"],
        "body": [
            {"/input": "${userInput}"},
            {"/global": {"$ref": "/_/config"}},  # From original source
            {"/result": "Input: ${@:/input}, Config: ${@:/global/setting}"}
        ],
        "return": "/result"
    },
    {"/output": {"$func": "combineData", "args": ["test"]}}
]

result = engine.apply(
    spec,
    source={"config": {"setting": "production"}},
    dest={}
)
# → {"output": "Input: test, Config: production"}

$func — Call a function

{
    "$func": "myFunction",
    "args": [10, 20]
}

• args — list of arguments to pass (optional, default: [])

Functions are stored in the execution context metadata and can be called multiple times within the same transformation.

$raise — Raise an error

{
    "$raise": "Invalid data: ${/error_details}"
}

Raises a JPermError with the specified message. The error can be:

• Caught by on_failure handlers in function definitions
• Used for validation and control flow
• Combined with templates for dynamic error messages

Example with error handling:

[
    {
        "$def": "validateAge",
        "params": ["age"],
        "body": [
            {
                "op": "if",
                "cond": {"$not": [{"op": "assert", "value": "${int:${age}}", "return": true}]},
                "then": [{"$raise": "Age must be a number"}]
            },
            {
                "op": "if",
                "cond": "${?source.age < `0`}",
                "then": [{"$raise": "Age cannot be negative"}]
            }
        ],
        "on_failure": [
            {"/validation_error": "@:/"}
        ]
    },
    {"/result": {"$func": "validateAge", "args": [25]}}
]

---

5. Shorthand Syntax

Shorthands are expanded by priority-ordered StageProcessors before execution.

~assert

{
    "~assert": {
        "/x": 10,
        "/y": 20
    }
}

Expands to:

[
    {
        "op": "assert",
        "path": "/x",
        "equals": 10
    },
    {
        "op": "assert",
        "path": "/y",
        "equals": 20
    }
]

~delete

{
    "~delete": [
        "/tmp",
        "/cache"
    ]
}

Expands to:

[
    {
        "op": "delete",
        "path": "/tmp"
    },
    {
        "op": "delete",
        "path": "/cache"
    }
]

Append notation (field[])

{
    "/items[]": 123
}

Expands to:

{
    "op": "set",
    "path": "/items/-",
    "value": 123
}

Pointer assignment

{
    "/name": "/user/fullName"
}

Expands to:

{
    "op": "copy",
    "from": "/user/fullName",
    "path": "/name",
    "ignore_missing": true
}

Literal assignment

{
    "/status": "active"
}

Expands to:

{
    "op": "set",
    "path": "/status",
    "value": "active"
}

Priority order: ~assert (100) → ~delete (50) → pointer/literal assignment (0)

---

Built-in Operations

All operations are registered as ActionHandler instances in the main registry.

set

Write value to destination path.

{
    "op": "set",
    "path": "/target",
    "value": "...",
    "create": true,
    // Auto-create parents (default: true)
    "extend": true
    // Extend lists on append (default: true)
}

Special: path ending with /- appends to list.

---

copy

Copy value from source to destination.

{
    "op": "copy",
    "from": "/source/path",
    "path": "/dest/path",
    "ignore_missing": false,
    // Skip if missing (default: false)
    "default": "..."
    // Fallback value
}

---

delete

Remove value at path.

{
    "op": "delete",
    "path": "/remove",
    "ignore_missing": true
    // Don't error if missing (default: true)
}

---

foreach

Iterate over array/mapping.

{
    "op": "foreach",
    "in": "/items",
    "as": "item",
    // Variable name (default: "item")
    "do": [
        ...
    ],
    // Nested actions
    "skip_empty": true,
    // Skip if empty (default: true)
    "default": []
    // Fallback if missing
}

Note: If source is a dict, iterates over (key, value) tuples.

Special variable _: Inside foreach body, you can access:

• Current item via /${var} (e.g., /item)
• Original source via /_/path (source from engine.apply)
• Current dest via @:/path

Example:

# Enrich items with global config from source
spec = {
    "op": "foreach",
    "in": "/products",
    "as": "product",
    "do": {
        "/results[]": {
            "$eval": [
                {"/name": "${product/name}"},
                {"/price": "${product/price}"},
                {"/tax_rate": {"$ref": "/_/config/tax"}},  # From original source
                {"/total": "${?source.price * source.tax_rate}"}
            ]
        }
    }
}

result = engine.apply(
    spec,
    source={
        "products": [{"name": "A", "price": 100}, {"name": "B", "price": 200}],
        "config": {"tax": 1.2}
    },
    dest={}
)
# → {"results": [{"name": "A", "price": 100, "tax_rate": 1.2, "total": 120}, ...]}

---

while

Loop while condition holds.

Path mode:

{
    "op": "while",
    "path": "/counter",
    "equals": 0,
    // Or "exists": true
    "do": [
        ...
    ],
    "do_while": false
    // Execute at least once (default: false)
}

Expression mode:

{
    "op": "while",
    "cond": "${?dest.counter < `10`}",
    "do": [
        ...
    ]
}

Note: Condition is checked against destination state. Use do_while: true to execute body at least once before checking condition.

---

if

Conditional execution.

Path mode:

{
    "op": "if",
    "path": "/check",
    "equals": "value",
    // Optional
    "exists": true,
    // Optional
    "then": [
        ...
    ],
    // Success branch
    "else": [
        ...
    ]
    // Failure branch
}

Expression mode:

{
    "op": "if",
    "cond": "${?source.age >= `18`}",
    "then": [
        ...
    ]
}

---

exec

Execute nested script.

From source:

{
    "op": "exec",
    "from": "/script",
    "merge": false
    // Replace dest (default) or merge into it
}

Inline:

{
    "op": "exec",
    "actions": [
        ...
    ]
}

---

update

Merge mapping into target.

{
    "op": "update",
    "path": "/obj",
    "value": {
        "b": 2
    },
    // Or "from": "/source/obj"
    "deep": false
    // Recursive merge (default: false)
}

---

distinct

Remove duplicates from list.

{
    "op": "distinct",
    "path": "/items",
    "key": "/id"
    // Optional: compare by nested field
}

---

assert

Assert value existence/equality.

Basic usage:

{
    "op": "assert",
    // Check source
    "path": "/required",
    "equals": "value"
    // Optional
}

With direct value:

{
    "op": "assert",
    "value": "${?source.computed}",
    // Check computed value instead of path
    "equals": "expected"
}

With return mode:

{
    "op": "assert",
    "path": "/optional",
    "return": true,
    // Return value instead of raising error
    "to_path": "/result"
    // Optional: write result to destination
}

• return: true — returns value on success, false on failure (instead of raising error)
• to_path — destination path for return value
• value — alternative to path, checks direct value

---

Extending J-Perm

Custom Operations

Create a new ActionHandler and register it:

from j_perm import ActionHandler, ActionNode, OpMatcher, ExecutionContext


class MyOpHandler(ActionHandler):
    def execute(self, step, ctx: ExecutionContext):
        # Your logic here
        return ctx.dest


# Register in main registry (in build_default_engine or custom factory)
registry.register(ActionNode(
    name="my_op",
    priority=10,
    matcher=OpMatcher("my_op"),
    handler=MyOpHandler(),
))

---

Custom Special Constructs

Add a SpecialFn to the specials dict:

def my_special(node, ctx):
    value = ctx.engine.process_value(node["$mySpecial"], ctx)
    return value.upper()


engine = build_default_engine(specials={
    "$ref": ref_handler,
    "$eval": eval_handler,
    "$mySpecial": my_special,
})

---

Custom Stages

Create a StageProcessor for batch preprocessing:

from j_perm import StageProcessor, StageNode, StageRegistry


class ValidateStage(StageProcessor):
    def apply(self, steps, ctx):
        # Validate/transform steps
        return steps


# Register in main pipeline stages
stages = build_default_shorthand_stages()
stages.register(StageNode(
    name="validate",
    priority=200,  # Higher = runs earlier
    processor=ValidateStage(),
))

# Use in custom engine
main_pipeline = Pipeline(stages=stages, registry=main_registry)

---

Custom Casters

Provide custom casters to TemplSubstHandler:

from j_perm import TemplSubstHandler

custom_casters = {
    "int": lambda x: int(x),
    "json": lambda x: json.loads(x),
}

handler = TemplSubstHandler(casters=custom_casters)

Or use the default (built-in int, float, bool, str).

---

Custom Matchers

Implement ActionMatcher or StageMatcher:

from j_perm import ActionMatcher


class PrefixMatcher(ActionMatcher):
    def __init__(self, prefix):
        self.prefix = prefix

    def matches(self, step):
        return isinstance(step, dict) and
            step.get("op", "").startswith(self.prefix)

---

Advanced Topics

Value Stabilization Loop

When handlers call ctx.engine.process_value(value, ctx), the value pipeline runs repeatedly until:

1. Output equals input (stable)
2. value_max_depth iterations reached (default: 50)

This resolves nested templates and special constructs:

# Input: {"$ref": "/path_to_template"}
# Pass 1: {"$ref": ...} → "${/nested}"
# Pass 2: "${/nested}" → "final"
# Pass 3: "final" → "final" (stable ✓)

---

Hierarchical Registries

Both StageRegistry and ActionTypeRegistry support tree structures:

# Group related operations
math_registry = ActionTypeRegistry()
math_registry.register(ActionNode("add", 10, AddMatcher(), AddHandler()))
math_registry.register(ActionNode("sub", 10, SubMatcher(), SubHandler()))

# Mount as sub-tree
main_registry.register_group(
    "math",
    math_registry,
    matcher=OpMatcher("math"),
    priority=50,
)

---

Priority and Execution Order

Stages: All matching stages run in priority order (high → low).

Actions: First matching handler executes (unless exclusive=False).

Shorthands:

1. AssertShorthandProcessor (100) — extracts ~assert
2. DeleteShorthandProcessor (50) — extracts ~delete
3. AssignShorthandProcessor (0) — fallback for all remaining keys

---

Unescape Rules

After value stabilization, registered UnescapeRule callables strip escape sequences:

from j_perm import UnescapeRule

# Built-in: template_unescape (strips $${ → ${, $$ → $)
# Registered at priority 0

# Add custom unescape
engine.unescape_rules.append(
    UnescapeRule(name="custom", priority=10, unescape=my_unescape_fn)
)

---

API Reference

Core Classes

from j_perm import (
    # Core infrastructure
    ExecutionContext,
    ValueResolver,
    Engine,
    Pipeline,

    # Stage system
    StageProcessor,
    StageMatcher,
    StageNode,
    StageRegistry,

    # Action system
    ActionHandler,
    ActionMatcher,
    ActionNode,
    ActionTypeRegistry,

    # Middleware
    Middleware,

    # Unescape
    UnescapeRule,
)

Handlers

from j_perm import (
    # Value handlers
    TemplMatcher,
    TemplSubstHandler,
    SpecialMatcher,
    SpecialResolveHandler,
    ContainerMatcher,
    RecursiveDescentHandler,
    IdentityHandler,

    # Special construct functions
    ref_handler,
    eval_handler,
    and_handler,
    or_handler,
    not_handler,

    # Function handlers
    DefMatcher,
    CallMatcher,
    DefHandler,
    CallHandler,
    RaiseMatcher,
    RaiseHandler,
    JPermError,

    # Operation handlers
    SetHandler,
    CopyHandler,
    DeleteHandler,
    ForeachHandler,
    WhileHandler,
    IfHandler,
    ExecHandler,
    UpdateHandler,
    DistinctHandler,
    AssertHandler,
)

Utilities

from j_perm import (
    # Matchers
    OpMatcher,
    AlwaysMatcher,

    # Resolver
    PointerResolver,

    # Shorthand stages
    AssertShorthandProcessor,
    DeleteShorthandProcessor,
    AssignShorthandProcessor,

    # Factory
    build_default_engine,
    build_default_shorthand_stages,
)

---

Examples

Example 1: Data Filtering

spec = {
    "op": "foreach",
    "in": "/products",
    "do": {
        "op": "if",
        "cond": "${?source.item.price < `100`}",
        "then": {"/affordable[]": "/item"}
    }
}

Example 2: Conditional Copy with Default

spec = {
    "/result": {
        "$ref": "/maybe_missing",
        "$default": "not found"
    }
}

Example 3: Nested Evaluation

spec = {
    "/computed": {
        "$eval": [
            {"op": "set", "path": "/x", "value": "${int:/a}"},
            {"op": "set", "path": "/y", "value": "${int:/b}"}
        ],
        "$select": "${?add(dest.x, dest.y)}"
    }
}

Example 4: Mixed Shorthands

spec = [
    {"~assert": {"/user/id": 123}},
    {"~delete": "/temp"},
    {"/output": "/user/name"}
]

Example 5: Functions with Error Handling

spec = [
    # Define a validation function with error recovery
    {
        "$def": "validateAge",
        "params": ["age"],
        "body": [
            # Validate age is a number
            {
                "op": "if",
                "cond": {"$not": [{"op": "assert", "value": "${int:${age}}", "return": True}]},
                "then": [{"$raise": "Age must be a number"}]
            },
            # Validate age is positive
            {
                "op": "if",
                "cond": "${?source.age < `0`}",
                "then": [{"$raise": "Age cannot be negative: ${age}"}]
            },
            {"/valid": True}
        ],
        "return": "/valid",
        "on_failure": [
            # Capture the error in dest
            {"/validation_failed": True},
            {"/last_error": "Validation error occurred"}
        ]
    },
    # Use function to validate user input
    {"/user_age_valid": {"$func": "validateAge", "args": [25]}},
    # Access dest values with @: prefix
    {
        "op": "if",
        "path": "@:/validation_failed",
        "equals": True,
        "then": [{"/message": "Please check your input: ${@:/last_error}"}]
    }
]

---

License

MIT (or adapt to your project as needed)

---

Contributing

Issues and pull requests welcome!