Flytekit whylogs Plugin
whylogs is an open source library for logging any kind of data. With whylogs, you are able to generate summaries of datasets (called whylogs profiles) which can be used to:
- Create data constraints to know whether your data looks the way it should
- Quickly visualize key summary statistics about a dataset
- Track changes in a dataset over time
pip install flytekitplugins-whylogs
To generate profiles, you can add a task like the following:
import whylogs as why
from whylogs.core import DatasetProfileView
import pandas as pd
from flytekit import task
@task
def profile(df: pd.DataFrame) -> DatasetProfileView:
result = why.log(df) # Various overloads for different common data types exist
profile_view = result.view()
return profile
NOTE: You'll be passing around
DatasetProfileViewfrom tasks, notDatasetProfile.
Validating Data
A common step in data pipelines is data validation. This can be done in
whylogs through the constraint feature. You'll be able to create failure tasks
if the data in the workflow doesn't conform to some configured constraints, like
min/max values on features, data types on features, etc.
from whylogs.core.constraints.factories import greater_than_number, mean_between_range
@task
def validate_data(profile_view: DatasetProfileView):
builder = ConstraintsBuilder(dataset_profile_view=profile_view)
builder.add_constraint(greater_than_number(column_name="my_column", number=0.14))
builder.add_constraint(mean_between_range(column_name="my_other_column", lower=2, upper=3))
constraint = builder.build()
valid = constraint.validate()
if valid is False:
print(constraint.report())
raise Exception("Invalid data found")
If you want to learn more about whylogs, check out our example notebooks.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file flytekitplugins_whylogs-1.16.28.tar.gz.
File metadata
- Download URL: flytekitplugins_whylogs-1.16.28.tar.gz
- Upload date:
- Size: 5.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.12.13
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
61fbc610f8d77c3b1391bd0e6a3cc0b3dd5f310197d27895d4126830383a3b3b
|
|
| MD5 |
8b6880accae8cce5c22c1b4732f7cd12
|
|
| BLAKE2b-256 |
62db00d964daac40ea9162e4e263403f518bdee1c7dc0c652f727586ecc7eb1a
|
File details
Details for the file flytekitplugins_whylogs-1.16.28-py3-none-any.whl.
File metadata
- Download URL: flytekitplugins_whylogs-1.16.28-py3-none-any.whl
- Upload date:
- Size: 5.9 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.12.13
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
e4b24dbbcd8f58b675d459c52126f6f3742f8696e28523262db99303f7de8fe3
|
|
| MD5 |
821af9c792fe48b5046960ae64caa6f5
|
|
| BLAKE2b-256 |
8377fd83182ceeadf04743676538543b8557e92080dd89d55d18d60e21db0468
|
