calista 0.3.2

Comprehensive Python package designed to simplify data quality checks across multiple platforms

Table of contents

• Calista
  • Installing from PyPI
  • Getting Started
    • Example
  • Documentation
  • License

Calista

Calista is a comprehensive Python package designed to simplify data quality checks across multiple platforms using a consistent syntax. Inspired by modular libraries, Calista aims to streamline data quality tasks by providing a unified interface.

Built on popular Python libraries like Pyspark and SQLAlchemy, Calista leverages their capabilities for efficient large-scale data processing. By abstracting engine-specific complexities, Calista allows users to focus on data quality without dealing with implementation details.

At its core, Calista offers a cohesive set of classes and methods that consolidate functionalities from various engine-specific modules. Users can seamlessly execute operations typically associated with Spark or SQL engines through intuitive Calista interfaces.

Currently developed in Python 3.10, Calista supports data quality checks using engines such as Spark, Pandas, Polars, Snowflake and BigQuery.

Whether orchestrating data pipelines or conducting assessments, Calista provides the tools needed to navigate complex data quality checks with ease and efficiency.

Installing from PyPI

To use our framework, simply install it via pip. This command will install the framework along with the default engines pandas and polars:

pip install calista

If you require support for another engines such as Snowflake, Spark, or BigQuery, use the following command and replace EngineName with the name of your desired engine:

pip install calista[EngineName]

Getting Started

To start using Calista, import the appropriate class:

from calista import CalistaEngine

With Calista, you can easily analyze your data quality, regardless of the underlying engine. The unified API streamlines your workflow and enables seamless integration across different environments.

Example

Here's an example using the Pandas Engine. Suppose you have a dataset represented as a table:

ID	status	last increase	salary
0	Célibataire	2022-12-31	36000
1		2023-12-31	53000
2	Marié	2018-12-31	28000

You can load this table using CalistaEngine with the Pandas engine:

from calista import CalistaEngine

table_pandas = CalistaEngine(engine="pandas").load(path="examples/demo_new_model.csv", file_format="parquet")

You can define custom rules using Calista functions to analyze specific conditions within your data:

from calista import functions as F

my_rule = F.is_not_null(col_name="status") & F.is_integer("salary")

print(table.analyze(rule_name="demo_new_model", rule=my_rule))

The output of the analysis provides insights into data quality based on the defined rule:

rule_name : demo_new_model
total_row_count : 3
valid_row_count : 2
valid_row_count_pct : 66.66
timestamp : 2024-04-23 10:00:59.449193

You can also just enhance your data by applying the rule:

from calista import functions as F

my_rule = F.is_not_null(col_name="status") & F.is_integer("salary")

print(table.apply_rule(rule_name="demo_new_model", rule=my_rule))

When printing, you'll get the following result:

ID	status	last increase	salary	demo_new_model
0	Célibataire	2022-12-31	36000	True
1		2023-12-31	53000	False
2	Marié	2018-12-31	28000	True

You also have the possibility to only retrieve the data that validate or invalidate the rule. For example, to get data invalidating the rule:

from calista import functions as F

my_rule = F.is_not_null(col_name="status") & F.is_integer("salary")

print(table.get_invalid_rows(rule=my_rule))

When printing, you'll get the following result:

ID	status	last increase	salary	demo_new_model
1		2023-12-31	53000	False

Documentation

Calista docs

License

Licensed under the Apache License