Skip to main content

An Apache Airflow provider for whylogs

Project description

whylogs Airflow Operator

This is a package for the whylogs provider, the open source standard for data and ML logging. With whylogs, users are able to generate summaries of their datasets (called whylogs profiles) which they can use to:

  • Track changes in their dataset
  • Create data constraints to know whether their data looks the way it should
  • Quickly visualize key summary statistics about their datasets

This Airflow operator focuses on simplifying whylogs' usage along with Airflow. Users are encouraged to benefit from their existing Data Profiles, which are created with whylogs and can bring a lot of value and visibility to track their data changes over time.

Installation

You can install this package on top of an existing Airflow 2.0+ installation (Requirements) by simply running:

$ pip install airflow-provider-whylogs

To install this provider from source, run these instead:

$ git clone git@github.com:whylabs/airflow-provider-whylogs.git
$ cd airflow-provider-whylogs
$ python3 -m venv .env && source .env/bin/activate
$ pip3 install -e .

Usage example

In order to benefir from the existing operators, users will have to profile their data first, with their processing environment of choice. To create and store a profile locally, run the following command on a pandas DataFrame:

import whylogs as why

df = pd.read_csv("some_file.csv")
results = why.log(df)
results.writer("local").write()

And after that, you can use our operators to either:

  • Create a Summary Drift Report, to visually help you identify if there was drift in your data
from whylogs_provider.operators.whylogs import WhylogsSummaryDriftOperator

summary_drift = WhylogsSummaryDriftOperator(
        task_id="drift_report",
        target_profile_path="data/profile.bin",
        reference_profile_path="data/profile.bin",
        reader="local",
        write_report_path="data/Profile.html",
    )
  • Run a Constraints check, to check if your profiled data met some criteria
from whylogs_provider.operators.whylogs import WhylogsConstraintsOperator
from whylogs.core.constraints.factories import greater_than_number

constraints = WhylogsConstraintsOperator(
        task_id="constraints_check",
        profile_path="data/profile.bin",
        reader="local",
        constraint=greater_than_number(column_name="my_column", number=0.0),
    )

NOTE: It is important to note that even though it is possible to create a Dataset Profile with the Python Operator, Airflow tries to separate the concern of orchestration from processing, so that is one of the reasons why we didn't want to have a strong opinion on how to read data and profile it, enabling users to best adjust this step to their existing scenario.

A full DAG example can be found on the whylogs_provider package directory.

Requirements

The current requirements to use this Airflow Provider are described on the table below.

PIP package Version required
apache-airflow >=2.0
whylogs[viz, s3] >=1.0.10

Contributing

Users are always welcome to ask questions and contribute to this repository, by submitting issues and communicating with us through our community Slack. Feel free to reach out and make whylogs even more awesome to use with Airflow.

Happy coding! 😄

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

airflow-provider-whylogs-0.0.3.tar.gz (9.5 kB view hashes)

Uploaded Source

Built Distribution

airflow_provider_whylogs-0.0.3-py3-none-any.whl (10.6 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page