vitalibo.norma 0.2.0

Pandas & PySpark data validation via JSON Schema

Norma

Norma is a data validation framework designed for Pandas and PySpark DataFrames, leveraging JSON Schema or a Python API to enforce data integrity. Unlike traditional validation libraries that halt execution on errors, Norma introduces a resilient, non-intrusive approach that captures validation details in a dedicated errors column while resetting erroneous values to null.

Install

Install with pip:

pip install vitalibo.norma

Quick Start

First, create a validation schema from JSON Schema:

{
  "$id": "https://example.com/address.schema.json",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "properties": {
    "name": {
      "type": "string",
      "minLength": 2
    },
    "age": {
      "type": "integer",
      "minimum": 0,
      "maximum": 120
    },
    "sex": {
      "type": "string",
      "enum": [
        "M",
        "F"
      ]
    },
    "email": {
      "type": "string",
      "pattern": "^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\\.[a-zA-Z0-9-.]+$"
    }
  },
  "required": [
    "name",
    "age",
    "email"
  ]
}

from norma.schema import Schema

schema = Schema.from_json_schema({... see above ...})

alternatively, build the schema with the Python API:

from norma.schema import Column, Schema
from norma import rules

schema = Schema({
    'name': Column(str, min_length=2),
    'age': Column(int, ge=0, le=120),
    'sex': Column(str, rules=rules.isin(['M', 'F'])),
    'email': Column(str, pattern=r'^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$')
})

Now, we are ready to validate our data. Let's create a Pandas DataFrame with some invalid data.

import pandas as pd

df = pd.DataFrame({
    'name': ['John', 'Rian', 'Alice'],
    'age': [42, 130, 25],
    'sex': ['M', 'F', 'X'],
    'email': ['[e-mail]', '[e-mail]', 'alice.spring@ep']
})

or, you can create the same example as a PySpark DataFrame:

from pyspark.sql import SparkSession

spark_session = SparkSession.builder.getOrCreate()

df = spark_session.createDataFrame([
    ('John', 42, 'M', '[e-mail]'),
    ('Rian', 130, 'F', '[e-mail]'),
    ('Alice', 25, 'X', 'alice.spring@ep')
], ['name', 'age', 'sex', 'email'])

Finally, validate the dataframe:

actual = schema.validate(df)

Output:

For Pandas DataFrame we have:

    name   age   sex           email  errors
0   John    42     M  [e-mail]  {}
1   Rian  <NA>     F  [e-mail]  {'age': {'details': [{'type': 'less_than_equal', 'msg': 'Input should be less than or equal to 120'}], 'original': '130'}}
2  Alice    25  <NA>            <NA>  {'sex': {'details': [{'type': 'enum', 'msg': 'Input should be "M" or "F"'}], 'original': '"X"'}, 'email': {'details': [{'type': 'string_pattern_mismatch', 'msg': 'String should match pattern "^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\\.[a-zA-Z0-9-.]+$"'}], 'original': '"alice.spring@ep"'}}

and dtypes:

name      string[python]
age                Int64
sex       string[python]
email     string[python]
errors            object
dtype: object

Similarly, for PySpark DataFrame:

+-----+----+----+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|name |age |sex |email         |errors                                                                                                                                                                                         |
+-----+----+----+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|John |42  |M   |[e-mail]|{}                                                                                                                                                                                             |
|Rian |NULL|F   |[e-mail]|{age -> {[{less_than_equal, Input should be less than or equal to 120}], 130}}                                                                                                                 |
|Alice|25  |NULL|NULL          |{email -> {[{string_pattern_mismatch, String should match pattern "^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$"}], "alice.spring@ep"}, sex -> {[{enum, Input should be "M" or "F"}], "X"}}|
+-----+----+----+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

and schema:

root
 |-- name: string (nullable = true)
 |-- age: integer (nullable = true)
 |-- sex: string (nullable = true)
 |-- email: string (nullable = true)
 |-- errors: map (nullable = false)
 |    |-- key: string
 |    |-- value: struct (valueContainsNull = true)
 |    |    |-- details: array (nullable = false)
 |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |-- type: string (nullable = true)
 |    |    |    |    |-- msg: string (nullable = true)
 |    |    |-- original: string (nullable = true)

Supported validation rules

Norma supports a variety of validation rules, including:

rule	pandas	pandas[object]	pandas[array]	pyspark	pyspark[object]	pyspark[array]
required	✅	✅	✅	✅	✅	✅
equal_to	✅	✅	✅	✅	✅	✅
not_equal_to	✅	✅	✅	✅	✅	✅
greater_than	✅	✅	✅	✅	✅	✅
greater_than_equal	✅	✅	✅	✅	✅	✅
less_than	✅	✅	✅	✅	✅	✅
less_than_equal	✅	✅	✅	✅	✅	✅
multiple_of	✅	✅	✅	✅	✅	✅
min_length	✅	✅	✅	✅	✅	✅
max_length	✅	✅	✅	✅	✅	✅
pattern	✅	✅	✅	✅	✅	✅
isin	✅	✅	✅	✅	✅	✅
notin	✅	✅	✅	✅	✅	✅
extra_forbidden	✅	✅	➖	✅	✅	➖
unique_items	✅	➖	✅	✅	➖	✅
max_items	✅	➖	✅	✅	➖	✅
min_items	✅	➖	✅	✅	➖	✅
int_parsing	✅	✅	✅	✅	✅	✅
float_parsing	✅	✅	✅	✅	✅	✅
str_parsing	✅	✅	✅	✅	✅	✅
bool_parsing	✅	✅	✅	✅	✅	✅
date_parsing	✅	✅	✅	✅	✅	✅
time_parsing	✅	✅	✅	✅	✅	✅
datetime_parsing	✅	✅	✅	✅	✅	✅
duration_parsing	✅	✅	✅	✅	✅	✅
uuid_parsing	✅	✅	✅	✅	✅	✅
ipv4_address	✅	✅	✅	✅	✅	✅
ipv6_address	✅	✅	✅	✅	✅	✅
uri_parsing	✅	✅	✅	✅	✅	✅
object_parsing	✅	✅	✅	✅	✅	✅
array_parsing	✅	✅	❌	✅	✅	❌

✅ - supported ❌ - not supported ➖ - not applicable

Errors

The error format is a dictionary where the key is the column name, and value is a structure with two fields: details and original. The details field is a list of details about the error, and the original field is the original value that caused the error.

{
  "name": "Alice",
  "age": 25,
  "sex": null,
  "email": null,
  "errors": {
    "sex": {
      "details": [
        {
          "type": "enum",
          "msg": "Input should be \"M\" or \"F\""
        }
      ],
      "original": "\"X\""
    },
    "email": {
      "details": [
        {
          "type": "string_pattern_mismatch",
          "msg": "String should match pattern \"^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\\.[a-zA-Z0-9-.]+$\""
        }
      ],
      "original": "\"alice.spring@ep\""
    }
  }
}

Special case, when DataFrame has array validation rules, the error format is slightly different. In this case, the details array has an additional loc field that indicates the indexes of the array elements that failed validation and incorrect values are replaced with null in an array.

{
  "tags": [
    null,
    "tag1",
    null
  ],
  "errors": {
    "tags[]": {
      "details": [
        {
          "loc": [
            0,
            2
          ],
          "type": "enum",
          "msg": "Input should be \"tag1\", \"tag2\" or \"tag3\""
        }
      ],
      "original": "[\"tag0\",\"tag1\",\"tag4\"]"
    }
  }
}