Enable the use of whylogs profiles to be used in flyte tasks to get aggregate statistics about data.
Project description
Flytekit whylogs Plugin
whylogs is an open source library for logging any kind of data. With whylogs, you are able to generate summaries of datasets (called whylogs profiles) which can be used to:
- Create data constraints to know whether your data looks the way it should
- Quickly visualize key summary statistics about a dataset
- Track changes in a dataset over time
pip install flytekitplugins-whylogs
To generate profiles, you can add a task like the following:
from whylogs.core import DatasetProfileView
import whylogs as ylog
import pandas as pd
@task
def profile(df: pd.DataFrame) -> DatasetProfileView:
result = ylog.log(df) # Various overloads for different common data types exist
profile = result.view()
return profile
NOTE: You'll be passing around
DatasetProfileView
from tasks, notDatasetProfile
.
Validating Data
A common step in data pipelines is data validation. This can be done in
whylogs
through the constraint feature. You'll be able to create failure tasks
if the data in the workflow doesn't conform to some configured constraints, like
min/max values on features, data types on features, etc.
@task
def validate_data(profile: DatasetProfileView):
column = profile.get_column("my_column")
print(column.to_summary_dict()) # To see available things you can validate against
builder = ConstraintsBuilder(profile)
numConstraint = MetricConstraint(
name='numbers between 0 and 4 only',
condition=lambda x: x.min > 0 and x.max < 4,
metric_selector=MetricsSelector(metric_name='distribution', column_name='my_column'))
builder.add_constraint(numConstraint)
constraint = builder.build()
valid = constraint.validate()
if(not valid):
raise Exception("Invalid data found")
Check out our constraints notebook for more examples.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for flytekitplugins-whylogs-1.1.3.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | a208d346ad2a7ae51a8613ffc0a42e9226977492b06fcf6a334d15e4699ef601 |
|
MD5 | 2fc229a61b5ce90f9eec34c5d46e44c5 |
|
BLAKE2b-256 | ea14d7c40a4637a9f1361b6c5918ca15c938ce0b69b9da18e65102a3d1d572b4 |
Hashes for flytekitplugins_whylogs-1.1.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9535c7366ab3e38b171b524946115609b9ffacb32166da57c10c062e9fcbb445 |
|
MD5 | fa3037ed8e91a0ccbb109e8e768b6d3b |
|
BLAKE2b-256 | e176a86a1c5b1c5a5a8da108d56df29315bffc9fdefe7fe801c8a836df024f0e |