lightweight pandas.DataFrame schema
Project description
DFS (aka Dataframe_Schema)
DFS is a lightweight validator for pandas.DataFrame
. You can think of it as a jsonschema
for dataframe.
Key features:
- Lightweight: only dependent on
pandas
andpydantic
(which depends only ontyping_extensions
) - Explicit: inspired by
JsonSchema
, all schemas are stored as json (or yaml) files and can be generated or changed on the fly. - Simple: Easy to use, no need to change your workflow and dive into the implementation details.
- Comprehensive: Summarizes all errors in a single summary exception, checks for distributions, works on subsets of the dataframe
- Rapid: base schemas can be generated from given dataframe or sql query (using
pd.read_sql
). - Handy: Supports command line interface (with
[cli]
extra). - Extendable: Core idea is to validate dataframes of any type. While now supports only pandas, we'll add abstractions to run same checks on different types of dataframes (CuDF, Dask, SparkDF, etc )
QuickStart
1. Validate DataFrame
Via wrapper
import pandas as pd
import dfschema as dfs
df = pd.DataFrame({
"a": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
"b": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
})
schema_pass = {
"shape": {"min_rows": 10}
}
schema_raise = {
"shape": {"min_rows": 20}
}
dfs.validate(df, schema_pass) # won't raise any issues
dfs.validate(df, schema_raise) # will Raise DataFrameSchemaError
Alternatively (v2 optional), you can use the root class, DfSchema
:
dfs.DfSchema.from_dict(schema_pass).validate(df) # won't raise any issues
dfs.DfSchema.from_dict(schema_raise).validate(df) # will Raise DataFrameSchemaError
2. Generate Schema
dfs.DfSchema.from_df(df)
3. Read and Write Schemas
schema = dfs.DfSchema.from_file('schema.json')
schema.to_file("schema.yml")
4. Using CLI
Note: requires [cli] extra as relies on
Typer
andclick
Validate via CLI
dfschema validate --read_kwargs_json '{delimiter="|"}' FILEPATH SCHEMA_FILEPATH
Supports
- csv
- xlsx
- parquet
- feather
Generate via CLI
dfs generate --format 'yaml' DATA_PATH > schema.yaml
Installation
WIP
Alternatives
- TableScheme
- GreatExpectations. Large and complex package with Html reports, Airflow Operator, connectors, etc. an work on out-of-memory data, SQL databases, parquet, etc
- Pandera - awesome package, great and suitable for type hinting, compatible with
hypothesis
- Tensorflow validate
- DTF expectations
Changes
- [[changelog]]
Roadmap
- Add tutorial Notebook
- Support tableschema
- Support Modin models
- Support SQLAlchemy ORM models
- Built-in Airflow Operator?
- Interactive CLI/jupyter for schema generation
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
dfschema-0.0.11.tar.gz
(18.2 kB
view hashes)
Built Distribution
dfschema-0.0.11-py3-none-any.whl
(21.9 kB
view hashes)
Close
Hashes for dfschema-0.0.11-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 87f8b291d86298942c6358e5f8df35f682ecb5680599c331ef2b4dc9fc8ca0bc |
|
MD5 | b0dd8081c36a424081e2ec27121e1830 |
|
BLAKE2b-256 | a6cf0d985bee3872b7f059d7aac45d2d783e82324bbab58e7fc1ad3a503fccc5 |