DataRules

Goal and motivation

The idea of this project is to define rules to validate and correct datasets. Whenever possible, it does this in a vectorized way, which makes this library fast.

Reasons to make this:

• Implement an alternative to https://github.com/data-cleaning/ based on python and pandas.
• Implement both validation and correction. Most existing packages provide validation only.
• Support a rule based way of data processing. The rules can be maintained in a separate file (python or yaml) if required.
• Apply vectorization to make processing fast.

Usage

This package provides two operations on data:

1. Checks (if data is correct). Also knows as validations.
2. Corrections (how to fix incorrect data).

Example

Create some data

import pandas as pd

df = pd.DataFrame([
    {"width": 3, "height": 7},
    {"width": 3, "height": 8},
])

1. Check the data

from datarules import CheckList, Check
from uneval import var

checks = CheckList([
    Check(name="almost_square",
          tags=["low-priority"],
          test=(var.width - var.height).abs() <= 4),
])
check_report = checks.run(df)
print(check_report)

Output:

CheckReport
-----------
          name                         test  items  passes  fails  NAs error  warnings
 almost_square  (width - height).abs() <= 4      2       1      1    0  None         0

2. Correct the data

from datarules import CorrectionList, Correction

corrections = CorrectionList([
    Correction(name="correct_square",
               trigger=checks[0].fails,
               action={"height": var.height / 2 + var.width / 2}),
])
correction_report = corrections.run(df)
print(correction_report)
print(f"Modified data:\n{df}")

Output:

CorrectionReport
----------------
           name                             trigger                           action  applied error  warnings
 correct_square  almost_square.fails(height, width)  height = height / 2 + width / 2        1  None         0

Modified data:
   width  height
0      3     7.0
1      3     5.5

See more examples on DataRules examples.

Similar work (python)

These work on pandas, but only do validation:

• Pandera - Like us, their checks are also vectorized.
• Pandantic - Combination of validation and parsing based on pydantic.

The following offer validation only, but none of them seem to be vectorized or support pandas directly.

• Great Expectations - An overengineered library for validation that has confusing documentation.
• contessa - Meant to be used against databases.
• validator
• python-valid8
• pyruler - Dead project that is rule-based.
• pyrules - Dead project that supports rule based corrections (but no validation).

Similar work (R)

This project is inspired by https://github.com/data-cleaning/. Similar functionality can be found in the following R packages:

• validate - Checking data (implemented)
• dcmodify - Correcting data (implemented)
• errorlocate - Identifying and removing errors (A start has been made here)
• deductive - Deductive correction based on checks (not yet implemented)