This package aims to automate the manual labour of lazy loading with pysimdjson.

Simply feed the JSON-schema in and the input data will be traversed and loaded with pysimdjson accordingly.

Supports

• pydantic>=1,<3
• python>=3.8,<3.12
• simdjson>=2,<6 (with caveats)

Does not support complex schemas (it may be not very reasonable from the practical standpoint anyway), e.g.,

• anyOf (Union[Model1, Model2])
• ...

In such cases it will fully (not lazily) load the underlying objects.

• Input JSON data is large relatively to what is needed in there, i.e., selectivity is small.
• Other deserialization methods appear to be slower and/or more memory consuming.

If you can check all the boxes, then, this package may prove useful to you. Never use it as a default deserialization method: run some benchmarks for your particular case first, otherwise, it may and will disappoint you.

pip install pysimdjson-schemaful

If you need pydantic support

pip install "pysimdjson-schemaful[pydantic]"

import json
from simdjson_schemaful import loads

schema = {
  "type": "array",
  "items": {
    "$ref": "#/definitions/Model"
  },
  "definitions": {
    "Model": {
      "type": "object",
      "properties": {
        "key": {"type": "integer"},
      }
    }
  }
}

data = json.dumps([
    {"key": 0, "other": 1},
    {"missing": 2},
])

parsed = loads(data, schema=schema)

assert parsed == [
    {"key": 0},
    {},
]

Example with additionalProperties:

schema = {
  "type": "object",
  "additionalProperties": {
    "$ref": "#/definitions/Model",
  },
  "definitions": {
    "Model": {
      "type": "object",
      "properties": {
        "key": {"type": "integer"},
      }
    }
  }
}

data = json.dumps({
    "some": {"key": 0, "other": 1},
    "other": {"missing": 2},
})

parsed = loads(data, schema=schema)

assert parsed == {
    "some": {"key": 0},
    "other": {},
}

With re-used simdjson parser (recommended when used in a single thread, otherwise better consult pysimdjson project on thread-safety):

from simdjson import Parser

parser = Parser()
parsed = loads(data, schema=schema, parser=parser)

assert parsed == {
    "some": {"key": 0},
    "other": {},
}

With model (call BaseModel.parse_raw_simdjson):

import json
from simdjson_schemaful.pydantic.v1 import BaseModel

class Model(BaseModel):
  key: int

data = json.dumps({"key": 0, "other": 1})

obj = Model.parse_raw_simdjson(data)

With type (call parse_raw_as_simdjson):

import json
from typing import List
from simdjson_schemaful.pydantic.v1 import BaseModel, parse_raw_simdjson_as

class Model(BaseModel):
  key: int

Type = List[Model]

data = json.dumps([
  {"key": 0, "other": 1},
  {"key": 1, "another": 2},
])

obj1, obj2 = parse_raw_simdjson_as(Type, data)

With model (call BaseModel.model_validate_simdjson):

import json
from simdjson_schemaful.pydantic.v2 import BaseModel

class Model(BaseModel):
  key: int

data = json.dumps({"key": 0, "other": 1})

obj = Model.model_validate_simdjson(data)

With type adapter (call TypeAdapter.validate_simdjson)

import json
from typing import List
from simdjson_schemaful.pydantic.v2 import BaseModel, TypeAdapter

class Model(BaseModel):
  key: int

adapter = TypeAdapter(List[Model])

data = json.dumps([
  {"key": 0, "other": 1},
  {"key": 1, "another": 2},
])

obj1, obj2 = adapter.validate_simdjson(data)

TBD