stat-merge 1.0.1

A library that runs pre-join analysis, detecting Cartesian explosions and suggesting composite keys.

Advanced Pre-merge Analysis Library

A library that runs pre-join analysis, giving basic stats on the tables to be joined, prediction on the final table, composite join keys suggestion and also detects potential risks like NaNs or Cartesian Product explosions. Stats are available both in console and in json formats to be used in your apps or dashboards

CLI Output Example

======================================== Merge on: CUST_NAME ========================================
                    Table 1 Table 2
Rows                  38168   20000
Dtypes      [str, NoneType]   [str]
NaN                    2757       0
Unique                32929   18005
Duplicates             5238    1995
Matches: 23

Left                     Right                    Inner                    Outer
Rows: 38170              Rows: 20001              Rows: 26                 Rows: 58145
Type: multiple-multiple  Type: multiple-multiple  Type: multiple-multiple  Type: multiple-multiple
RAM~: 5.5 MB             RAM~: 2.9 MB             RAM~: 3.8 KB             RAM~: 8.3 MB



------------------------------
💡 COMPOSITE KEY SUGGESTION
------------------------------
Joining on INN: 38168 rows
Joining on composition of {'INN', 'CUST_NAME'}: 37316 rows, reduces duplicates by 4387 rows.

Installation

You can install the library using pip:

pip install stat-merge 

Importing the Library

import stat_merge as sm

Usage

sm.show(df1, df2)
sm.get_json(df1, df2) 

JSON Output Example

{'merge_on': 'INN',
  'stats': {'Table 1': {'Rows': 38168,
    'Dtypes': ['str'],
    'NaN': 0,
    'Unique': 36611,
    'Duplicates': 1557},
   'Table 2': {'Rows': 20000,
    'Dtypes': ['str'],
    'NaN': 0,
    'Unique': 20000,
    'Duplicates': 0}},
  'prediction': {'Left': {0: 'Rows: 38168',
    1: 'Type: multiple-single',
    2: 'RAM~: 2.7 MB'},
   'Right': {0: 'Rows: 20051', 1: 'Type: single-multiple', 2: 'RAM~: 1.4 MB'},
   'Inner': {0: 'Rows: 1168', 1: 'Type: multiple-single', 2: 'RAM~: 85.5 KB'},
   'Outer': {0: 'Rows: 57051', 1: 'Type: multiple-single', 2: 'RAM~: 4.1 MB'}},
  'composite_key_suggestion': "Joining on INN: 38168 rows. Joining on composition of {'INN', 'CUST_NAME'}: 37316 rows, reduces duplicates by 705 rows."}

Release 1.0.1 Notes

• JSON format is available now