Index

• Infer Schema
• Schema evolvement
• JSON Transform
  • json_remove_field
  • json_traverse_transform
  • json_convert_to_array

Infer Schema

With function infer_schema, you can get schema from json object. The output complys to JSON Schema Specification. Here is an example:

#!/usr/bin/env python
# -*- coding: UTF-8 -*-

from json_tools2 import infer_schema
import json

def main():
    schema = infer_schema({"foo": 1})
    print(json.dumps(schema, indent=4))

if __name__ == '__main__':
    main()

Output

{
    "type": "object",
    "properties": {
        "foo": {
            "type": "integer"
        }
    }
}

When you have an array of element from different types, or object with the same property but from different type, we need to merge schema.

case 1: 2.1 and 2 are of different types and we need to merge schema

[2.1, 2]

case 2: 2.1 and 2 are of different types and we need to merge schema

[{"foo": 2.1}, {"foo": 2}]

Following rules applies when we do schema merging:

• For null, any type can merged to it and return a nullable type, for example, sting merge to null and return a nullable string
• For string, only null can merged to string and return nullable string.
• For number, followings are the only valid type that can merge to it
  • integer can merged to it without changing it
  • null can merge to it and become nullable number
• For integer, followings are the only valid type that can merge to it
  • number can merged to it and upgrade it to number
  • null can merge to it and become nullable number
• For boolean, followings are the only valid type that can merge to it
  • null can merge to it and become nullable boolean
• For object, followings are the only valid type that can merge to it
  • null can merge to it and become nullable object
• For array, followings are the only valid type that can merge to it
  • null can merge to it and become nullable array

We use anyOf to represent nullable type, for example: This represent a nullable integer type.

{
    "anyOf": [{"type": "integer"},{"type": "null"}]
}

For any schema inferred, you should be able to validate the payload with the schema inferred, here is an example:

#!/usr/bin/env python
# -*- coding: UTF-8 -*-

# You need to do "pip install jsonschema"
from json_tools2 import infer_schema
from jsonschema import validate
import json


def main():
    payload = {"foo": 1}
    schema = infer_schema(payload)
    validate(instance=payload, schema=schema) # it should not throw exception

if __name__ == '__main__':
    main()

Schema evolvement

Think about such case:

• You receive json object every day.
• Every day, you want to load schema inferred from the json object from the prior day, and merge it with the schema inferred from today's object, and then save the inferred schema.

Here is an example:

#!/usr/bin/env python
# -*- coding: UTF-8 -*-

from json_tools2 import get_schema, ValueSchema
import json

def main():
    schema = get_schema(1)
    print(json.dumps(schema.to_json(), indent=4))
    saved_schema = schema.dump_json()
    # You got a json object saved_schema which you can save the inferred schema

    # load schema
    schema = ValueSchema.load_json(saved_schema)
    schema.merge_from(get_schema(None))
    print(json.dumps(schema.to_json(), indent=4))


if __name__ == '__main__':
    main()

Output

{
    "type": "integer"
}
{
    "anyOf": [
        {
            "type": "integer"
        },
        {
            "type": "null"
        }
    ]
}

JSON Transform

json_remove_field

json_remove_field(obj:Any, field_name:str) -> Any:

It removes field specified in field_name recursively.

Example:

json_remove_field({
    "foo": 1, 
    "bar": {"foo": 1, "t": 2}
}, "foo")

it will remove field "foo" recursively, so it will return

{
    "bar": {
        "t": 2
    }
}

json_traverse_transform

json_traverse_transform(obj: Any, path:str, handler:Callable[[Any], None]) -> Any:

It traverse the input json object obj, for each object that match the path pattern specified in path, it calls the function handler. Eventually obj is retuened however, it might be mutated by handler.

Path pattern:

Given object, a path represent a list of object extracted from object.

For example, given object X, here are example pathes:

"foo"        represent X.foo
"foo.bar"    represent X.foo.bar
"foo[]"      represent [X.foo[0], X.foo[1], ...]
"foo[].x"    represent [X.foo[0].x, X.foo[1].x, ...]
"foo[].x[]"  represent [X.foo[0].x[0], X.foo[0].x[1], ..., X.foo[1].x[0], X.foo[1].x[1], ...]

Example:

def handler(o):
    if type(o) != dict:
        return o
    x = o.get("x", 0)
    y = o.get("y", 0)
    if type(x) in (int, float) and type(y) in (int, float):
        o["sum"] = x+y

json_traverse_transform(
    {
        "cords": [
            {"x": 1, "y": 2},
            {"x": 2, "y": 3},
        ]
    }, "cords[]", handler
)

returns

{
    "cords": [
        {"x": 1, "y": 2, "sum": 3},
        {"x": 2, "y": 3, "sum": 5},
    ]
}

json_convert_to_array

json_convert_to_array(obj:Any, field_name:str, new_field_name:str, path="")->Any:

For each matching child object based on pattern path, it convert it's field field_name from dictionary into list, result is stored in field new_field_name, the original field is either deleted or overritten.

Example:

json_convert_to_array([
    {
        "q1": {
            "score": {"math": 1, "physics": 2}
        }
    },
    {
        "q1": {
            "score": {"math": 2, "physics": 3}
        }
    }
], "score", "score2", "[].q1")

returns

[
    {
        "q1": {
            "score2": [
                {"key": "math", "value": 1},
                {"key": "physics", "value": 2}
            ]
        }
    },
    {
        "q1": {
            "score2": [
                {"key": "math", "value": 2},
                {"key": "physics", "value": 3}
            ]
        }
    },
]