json-castle 0.1.1

Built on top of native json module for deserialization from JSON to data classes with additional support for nested objects with variables, environment variables, and post-load overrides.

JsonCastle

JsonCastle is a Python module built on top of the native json module for deserialization from JSON to data classes with additional support for:

• nested objects
• JSON nariables
• environment variables
• post-load overrides, including overriding, adding, and removing collection elements via special CLI argument syntax like node.next_node.number=2, ~node.tags[0], +node.next_node.tags=foo, ~node.next_node.tags=buzz, etc.

With these features, you might find it especially useful in CI/CD pipelines.

Getting Started

Imagine you have a JSON configuration file for an automated process, but you want to run this process with slightly different variants of this file. Typically, you'd end up with multiple very similar configuration files, or you'd create an ad-hoc solution that would allow you to override certain parameters via CLI arguments. JsonCastle gives you such a solution out of the box for all properties of your data scheme, even for those in nested objects.

Since native json doesn't support variables, such configuration files might have a lot of repeated strings. Editing such files can be tedious and error-prone. You write a JSON file with variables and environment variables as the following example:

{
    "$home_path": "%HOME%",
    "$package_path": "${home_path}debug",
    "exe_path": "${package_path}/mytool.exe",
    "node": {
        "number": 0,
        "tags": ["foo", "bar"],
        "next_node": {
            "number": 1,
            "tags": ["fizz", "buzz"]
        }
    }
}

Assuming the HOME environment variable is set, for example, to C:/users/cicd/, then the value of the $package_path variable will be expanded from ${home_path}debug to C:/users/cicd/debug, and the value of exe_path will be C:/users/cicd/debug/mytool.exe.

In Python, you'd define your data classes that match the scheme:

from dataclasses import dataclass

@dataclass
class Cfg:
    exe_path: str = None
    node: Node = None

@dataclass
class Node:
    number: int
    tags: list[str]
    processed: False
    next_node: Node = None

Now you can use the JsonCastle.load_from_file method to read and parse example.json and get an instance of the Cfg class:

from pprint import pprint
import sys

from json_castle import JsonCastle

cfg = JsonCastle.load_from_file(Cfg, "example.json", **JsonCastle.parse_args(sys.argv))
pprint(cfg)

If you run the Python script without any arguments, you’d see cfg printed as expected:

Cfg(exe_path='C:/users/cicd/debug/mytool.exe',
    node={'next_node': {'number': 1, 'tags': ['fizz', 'buzz']},
          'number': 0,
          'tags': ['foo', 'bar']})

However, by passing **JsonCastle.parse_args(sys.argv) as the third argument, you can override any values post-load via CLI. Running the script again with the arguments node.number=1 ~node.tags[0] node.next_node.number=2 +node.next_node.tags=foo ~node.next_node.tags=buzz will give you a cfg instance like this:

Cfg(exe_path='C:/users/cicd/debug/mytool.exe',
    node={'next_node': {'number': 2, 'tags': ['fizz', 'foo']},
          'number': 1,
          'tags': ['bar']})

If you want to deserialize your JSON from a string stream, you can use the static method load(cls, stream: IO[str], **kwargs), the usage is the same as of the load_from_file(cls, path: str, **kwargs) in the example above, but you provide your stream as the second argument instead of filepath.

CLI Overrides Syntax

Although some of the syntax for post-load overrides via the CLI was introduced already in the Getting Started section, here's an overview of all options with examples.

Basic Syntax

JsonCastle.parse_args method accepts a list of strings, where each element is a key=value pair except the one for removing an item by index, that only has a key, in which case, in the dictionary the method returns, the value is None.

Of course, you don't have to use this method and construct a dictionary by yourself instead; the JsonCastle.parse_args just provides you a convenient way for post-load overriding via CLI, which is particularly useful in CI/CD pipelines. Here are some examples of CLI args for overriding top-level pairs:

age=27 name=John full_name="John Smith" active=true balance=123.45

Nested Objects

If you need to override a nested value, you specify a path, separating objects with ., as you can see in the Getting Started. The following example shows how to override several pairs nested in the person object.

person.age=27 person.name=John person.full_name="John Smith" person.active=true person.balance=123.45

Working with Collections

When it comes to collections, you can change, add, or remove an item as well. If the collection is a nested object, the rules are the same as described in the Nested Objects section.

Overriding an Item

To override an item, you provide a path to the collection, in square brackets an index of the item you wish to override, and a new value on the right side of the = symbol. The following example shows how to change the value of the first element of the tags collection nested in the page object to programming.

page.tags[0]=programming

Adding a new Item

When you wish to add a new item to a collection, you prefix the key=value pair with a + symbol. The following example shows how to add a new value of python to the tags collection nested in the page object.

+page.tags=python

Removing an Item by Index

If you wish to remove an item at specific index, you prefix a path to the collection with index in square brackets of the element you wish to remove with the ~ symbol. The following example shows how to remove the second item from the tags collection nested in the page object.

~page.tags[1]

If any items follow the removed one, they will be pushed down, and the new length of the collection will be n-1.

Removing an Item by Value

If you wish to remove an item of a specific value, instead of the index as described in Removing an Item by Index, you provide a key=value pair prefixed with ~. The following example shows how to remove an item with a value of “programming” from the tags collection nested in the page object.

~page.tags=programming

If there is more than one occurence if programming string in the collection, only the first one will be removed.

Unit Tests

Unit Tests are not just a great way to ensure nothing is broken when a new feature is added, but also for implementing new features using TDD, which is the recommended way for extending this library. These tests can also serve as an overview of what the library is capable of. You can find all tests in the unit_test.py file.

Future Ideas

• Add support for conditionals, loops, and mathematical expressions in JSON.
• Extend the parse_args method to allow adding a custom object to a collection.
• Add regex support for removing items by value.
• When removing an item from a collection by value, let the user decide whether they want to remove just one or all items that match the value (~page.tags=programming removes the first; ~!page.tags=programming removes all).
• Add support for removing items from a numerical collection by conditions >, <, <= , or =>.
• Add support for removing items by range (i.e. ~items[1:4] would remove items at indices 1, 2, 3 and 4).
• Add support for removing custom items by condition (i.e., ~people={age < 16} would remove from the people collection all objects with the age key-value pair with age less than 16).