Perform QA between 2 dataframes
Project description
The New DST QA Library (DST2)
DST2 is the new QA library that addresses issues like ease of use, dynamic reporting and error management. The new library has only one function called "perform_qa" that does most of the check for the deliverables (similar pandas dataframes). Functionalities:
- Dynamic Error Management
- Flexible Reporting
- Search Operations
- Shorter Notebook Codes
- More..
Installation
Install the package via pip with code below:
> pip install DST2
To Upgrade
> pip install --upgrade DST2
#For importing the QA library
import DST2.QA as q
import pandas as pd
The package is built on top of pandas thus making it easier to compare dataframes
dfOld = pd.read_excel('OLD_FILE.xyz') #The extension .xyz could be xlsx, csv, json or any that can be read by pandas
dfNew = pd.read_excel('NEW_FILE.xyz') #The extension .xyz could be xlsx, csv, json or any that can be read by pandas
Starting a QA process
When initiating a new QA process, you will have to provide the following:
- Name of the Excel report
- The previous and new deliverable via pandas
- The index column (a column name or a list of columns)
#Initiate a QA process
qa = q.QA_Report("Report 1",dfOld,dfNew,'Entity ID')
#Create Reports
qa.create_report()
#Let's create another report specifying parameters
#Start a Report
qa2 = q.QA_Report("Report 2",dfOld,dfNew,'Entity ID')
Perform QA
This is the core of the QA process where you decide to:
- Perform column or score comparisons
- Set deltas
- Search columns for QA
- Perform QA on all columns
#Perform QA on Columns comparison
spec_cols = ['Highest Controversy Level-Answer Category','Does the company meet your screening criteria?'] #fields in both files
qa2.perform_qa(columns=spec_cols)
#Perform QA on Score changes with default delta = 5
cols = ['Total ESG Score','Percentile']
qa2.perform_qa(columns=cols,type='score', delta=5) #default is 5 anyways
#Create Reports
qa2.create_report()
Recap!
We have used 3 parameters with the perform_qa function which are:
- choosing an index that identifies each row uniquely - 'Entity ID'
- columns -- To specify the columns to perform QA on
- type -- To specify if it is a column or score comparison and by default it performs a column comparison
- delta -- By default it is set to 5 and it is used when we perform a score comparison to define a threshold.
#Start a Report
qa3 = q.QA_Report("Report 3",dfOld,dfNew,'Entity ID')
More on parameters
We have used 3 more parameters with the perform_qa function which are
- all_cols -- To perform QA on all columns and it is set to False by default
- keywords -- To search for some keywords in field names eligible for QA
- takeout_keywords -- To search for some keywords in field names and remove those fieldnames NOT eligible for QA
- In this last example we have added the type score because we are performing score changes
qa3.perform_qa(all_cols=True,takeout_keywords=['score','percentile'])
qa3.perform_qa(keywords=['score','percentile'],takeout_keywords='overall',type='score', delta=10)
qa3.create_report()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.