datarules 0.2.2

Rules for validating and correcting datasets

DataRules

Goal and motivation

The idea of this project is to define rules to validate and correct datasets. Whenever possible, it does this in a vectorized way, which makes this library fast.

Reasons to make this:

• Implement an alternative to https://github.com/data-cleaning/ based on python and pandas.
• Implement both validation and correction. Most existing packages provide validation only.
• Support a rule based way of data processing. The rules can be maintained in a separate file (python or yaml) if required.
• Apply vectorization to make processing fast.

Usage

This package provides two operations on data:

1. Checks (if data is correct). Also knows as validations.
2. Corrections (how to fix incorrect data).

Example

Create some data

import pandas as pd

df = pd.DataFrame([
    {"width": 3, "height": 7},
    {"width": 3, "height": 8},
])

1. Check the data

from datarules import CheckList, Check
from uneval import var

checks = CheckList([
    Check(name="almost_square",
          tags=["low-priority"],
          test=(var.width - var.height).abs() <= 4),
])
check_report = checks.run(df)
print(check_report)

Output:

CheckReport
-----------
          name                         test  items  passes  fails  NAs error  warnings
 almost_square  (width - height).abs() <= 4      2       1      1    0  None         0

2. Correct the data

from datarules import CorrectionList, Correction

corrections = CorrectionList([
    Correction(name="correct_square",
               trigger=checks[0].fails,
               action={"height": var.height / 2 + var.width / 2}),
])
correction_report = corrections.run(df)
print(correction_report)
print(f"Modified data:\n{df}")

Output:

CorrectionReport
----------------
           name                             trigger                           action  applied error  warnings
 correct_square  almost_square.fails(height, width)  height = height / 2 + width / 2        1  None         0

Modified data:
   width  height
0      3     7.0
1      3     5.5

See more examples on DataRules examples.

Similar work (python)

These work on pandas, but only do validation:

• Pandera - Like us, their checks are also vectorized.
• Pandantic - Combination of validation and parsing based on pydantic.

The following offer validation only, but none of them seem to be vectorized or support pandas directly.

• Great Expectations - An overengineered library for validation that has confusing documentation.
• contessa - Meant to be used against databases.
• validator
• python-valid8
• pyruler - Dead project that is rule-based.
• pyrules - Dead project that supports rule based corrections (but no validation).

Similar work (R)

This project is inspired by https://github.com/data-cleaning/. Similar functionality can be found in the following R packages:

• validate - Checking data (implemented)
• dcmodify - Correcting data (implemented)
• errorlocate - Identifying and removing errors (A start has been made here)
• deductive - Deductive correction based on checks (not yet implemented)