tabularcompare 0.0.4

Tabular data comparison wrapper for DataComPy

TabularCompare

Tabular data comparison wrapper for DataComPy.

Quick Install

pip install tabularcompare

Basic Usage

Comparison object

import pandas as pd
from tabularcompare import Comparison

df1 = pd.DataFrame(
    {
        "idx1": ["A", "B", "B", "C"],
        "idx2": ["01", "01", "02", "03"],
        "colA": ["AA", "BA", "BB", "CA"],
        "colB": [100, 200, 200, 300]
     }
)
df2 = pd.DataFrame(
    {
        "idx1": ["A", "B", "C"],
        "idx2": ["01", "01", "03"],
        "colA": ["AA", "XA", "CA"],
        "colB": [101, 200, 300],
        "colC": ["foo", "bar", "baz"]
     }
)

comparison = Comparison(
    df1, df2, join_columns=["idx1", "idx2"]
)

Added Functionalities

• Diverging Subset

This method introduces an enhanced look at the changes identified at the intersection of compared DataFrames, following the notation {df1} --> {df2}.

comparison.diverging_subset()

	idx1	idx2	colA	colB
0	A	01	NaN	{100} --> {101}
1	B	01	{BA} --> {XA}	NaN

Rows that are unique to either DataFrame can be called via .df1_unq_rows() and .df2_unq_rows() methods.

comparison.df1_unq_rows()

	idx1	idx2	colA	colB
2	B	02	BB	200

Columns that are unique to either DataFrame can be called via .df1_unq_columns() and .df2_unq_columns() methods.

comparison.df2_unq_columns()

	idx1	idx2	colC
0	A	01	foo
1	B	01	bar
2	C	03	baz

• Enhanced Reporting

The report functionality is now callable from the comparison object. It includes a .txt, .html, and .xlsx version.

comparison.report_to_txt("./results/Report.txt")
comparison.report_to_html("./results/Report.html")
comparison.report_to_xlsx("./results/Report.xlsx", write_originals=True)

The Excel report will output complete comparison results, with tabs dedicated to:

• Original dataframes (when write_originals=True).
• Columns present only on df1 and/or df2.
• Rows present only on df1 and/or df2.
• Diverging subset showing all the changes identified from df1 to df2.

The HTML report will also output a rendered table of the diverging subset on top of the file, alongside the native DataComPy summary report.

• Fully-fledged Command Line Interface

---

DataComPy Methods

Most methods native to datacompy.Compare functionality are still present, including;

• .report()
• .df1_unq_rows() / .df2_unq_rows()
• .df1_unq_columns() / .df2_unq_columns()
• .intersect_columns()
• .intersect_rows()

The native datacompy.Compare method is also callable from the tabularcompare core module:

from tabularcompare import Compare

# datacompy.Compare method
comparison = Compare(
    df1, df2, join_columns=["idx1", "idx2"]
)
print(comparison.report())

For a complete documentation on DataComPy you can head to DataComPy.

---

CLI

Command Line Interface to output an Excel .xlsx report and, optionally, html and txt summaries.

tabularcompare --help

Usage: tabularcompare [OPTIONS] DF1 DF2

Options:
  -c, --columns, --on TEXT    Comma-separated list of key column(s) to compare
                              df1 against df2. When not provided, df1 and df2
                              will be matched on index.
  -ic, --ignore_columns TEXT  Comma-separated list of column(s) to ignore from
                              df1 and df2.
  -n1, --df1_name TEXT        Alias for Data frame 1. Default = df1
  -n2, --df2_name TEXT        Alias for Data frame 2. Default = df2
  -is, --ignore_spaces        Flag to strip and ignore whitespaces from string
                              columns.
  -ci, --case_insensitive     Flag to compare string columns on a case-
                              insensitive manner.
  -cl, --cast_lowercase       Flag to cast column names to lower case before
                              comparison.
  -at, --abs_tol FLOAT        Absolute tolerance between two numeric values.
  -rt, --rel_tol FLOAT        Relative tolerance between two numeric values.
  -txt, --txt                 Flag to output a .txt report with a comparison
                              summary.
  -html, --html               Flag to output an HTML report with a comparison
                              summary.
  -od, --only_deltas          Flag to suppress original dataframes from the
                              output .xlsx report.
  -o, --output, --out PATH    Output location for report files. Defaults to
                              current location.
  -e, --encoding TEXT         Character encoding to read df1 and df2.
  -v, --verbose               Verbosity.
  --help                      Show this message and exit.

Sample usage

The application reads from two file paths input for csv, json, or excel files.

cd ./data/
tabularcompare ./df1.csv ./df2.csv -c 'idx1,idx2' -n1 myTable1 -n2 myTable2 -o ../results/

---

Caveat

The comparison results will take into account data types across columns. I.E. If we update our sample dataframe df2 to include a missing value on colB, it will now be of dtype object, as oposed to Int64 in df1. This might lead to miss-leading interpretations of the results to users without information on data types.

import pandas as pd
from tabularcompare import Comparison

df1 = pd.DataFrame(
    {
        "idx1": ["A", "B", "B", "C"],
        "idx2": ["01", "01", "02", "03"],
        "colA": ["AA", "BA", "BB", "CA"],
        "colB": [100, 200, 200, 300]
     }
)
df2 = pd.DataFrame(
    {
        "idx1": ["A", "B", "C"],
        "idx2": ["01", "01", "03"],
        "colA": ["AA", "XA", "CA"],
        "colB": [pd.NA, 200, 300],
        "colC": ["foo", "bar", "baz"]
     }
)

comparison = Comparison(
    df1, df2, join_columns=["idx1", "idx2"]
)
comparison.diverging_subset()

Which will return:

	idx1	idx2	colA	colB
0	A	01	NaN	{100} --> {}
1	B	01	{BA} --> {XA}	{200} --> {200}
3	C	03	NaN	{300} --> {300}