Skip to main content

If your data is messy - Use Shmessy!

Project description

Shmessy

PyPI version PyPI - Downloads Coverage report CI License PyPI - Python Version OS OS OS Code style: black

If your data is messy - Use Shmessy!

Shmessy designed to deal with messy pandas dataframes. We all knows the frustrating times when we as analysts or data-engineers should handle messy dataframe and analyze them by ourselves.

The goal of this tiny tool is to identify the physical / logical data type for each Dataframe column. It based on fast validators that will validate the data (Based on a sample) against regex / pydantic types or any additional validation function that you want to implement.

As you understand, this tool was designed to deal with dirty data, ideally developed for Dataframes generated from CSV / Flat files or any source that doesn't contain strict schema.

Installation

pip install shmessy

Usage

You have two ways to use this tool

Identify the Dataframe schema

import pandas as pd
from shmessy import Shmessy

df = pd.read_csv('/tmp/file.csv')
inferred_schema = Shmessy().infer_schema(df)

Output (inferred_schema dump):

{
    "infer_duration_ms": 12,
    "columns": [
        {
            "field_name": "id",
            "source_type": "Integer",
            "inferred_type": "Integer"
        },
        {
            "field_name": "email_value",
            "source_type": "String",
            "inferred_type": "Email"
        },
        {
            "field_name": "date_value",
            "source_type": "String",
            "inferred_type": "Date",
            "inferred_pattern": "%d-%m-%Y"
        },
        {
            "field_name": "datetime_value",
            "source_type": "String",
            "inferred_type": "Datetime",
            "inferred_pattern": "%Y/%m/%d %H:%M:%S"
        },
        {
            "field_name": "yes_no_data",
            "source_type": "String",
            "inferred_type": "Boolean",
            "inferred_pattern": [
                "YES",
                "NO"
            ]
        },
        {
            "field_name": "unix_value",
            "source_type": "Integer",
            "inferred_type": "UnixTimestamp",
            "inferred_pattern": "ms"
        },
        {
            "field_name": "ip_value",
            "source_type": "String",
            "inferred_type": "IPv4"
        }
    ]
}

Identify and fix Pandas Dataframe

This piece of code will change the column types of the input Dataframe according to Messy infer.

import pandas as pd
from shmessy import Shmessy

df = pd.read_csv('/tmp/file.csv')
fixed_df = Shmessy().fix_schema(df)

Original Dataframe

Original Dataframe

Fixed Dataframe

After fix

Read Messy CSV file

from shmessy import Shmessy
df = Shmessy().read_csv('/tmp/file.csv')

Original file

Original Dataframe

Fixed Dataframe

After fix

API

Constructor

shmessy = Shmessy(
    sample_size: Optional[int] = 1000,
    reader_encoding: Optional[str] = "UTF-8",
    locale_formatter: Optional[str] = "en_US",
    use_random_sample: Optional[bool] = True,
    types_to_ignore: Optional[List[str]] = None,
    max_columns_num: Optional[int] = 500,
    fallback_to_string: Optional[bool] = False,  # Fallback to string in case of casting exception
    fallback_to_null: Optional[bool] = False,  # Fallback to null in case of casting exception
    use_csv_sniffer: Optional[bool] = True,  # Use python sniffer to identify the dialect (seperator / quote-char / etc...)
    fix_column_names: Optional[bool] = False,  # Replace non-alphabetic/numeric chars with underscore
)

read_csv

shmessy.read_csv(filepath_or_buffer: Union[str, TextIO, BinaryIO]) -> DataFrame

infer_schema

shmessy.infer_schema(df: Dataframe) -> ShmessySchema

fix_schema

shmessy.fix_schema(df: Dataframe) -> DataFrame

get_inferred_schema

shmessy.get_inferred_schema() -> ShmessySchema

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

shmessy-2.0.2.tar.gz (12.3 kB view details)

Uploaded Source

Built Distribution

shmessy-2.0.2-py3-none-any.whl (18.1 kB view details)

Uploaded Python 3

File details

Details for the file shmessy-2.0.2.tar.gz.

File metadata

  • Download URL: shmessy-2.0.2.tar.gz
  • Upload date:
  • Size: 12.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.8.3 CPython/3.9.16 Linux/6.5.0-1022-azure

File hashes

Hashes for shmessy-2.0.2.tar.gz
Algorithm Hash digest
SHA256 2f68657e73b353bd3631b732b274ce4ac4d3cc6bef7ed49bcdf9137d64e34b07
MD5 c5f1d3581c4eb526946d72b5a19a2b88
BLAKE2b-256 9054a624af12b4f921f00126da83e84c58aca7836198abda0d3eac75353bc34c

See more details on using hashes here.

File details

Details for the file shmessy-2.0.2-py3-none-any.whl.

File metadata

  • Download URL: shmessy-2.0.2-py3-none-any.whl
  • Upload date:
  • Size: 18.1 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.8.3 CPython/3.9.16 Linux/6.5.0-1022-azure

File hashes

Hashes for shmessy-2.0.2-py3-none-any.whl
Algorithm Hash digest
SHA256 76484c070d320a8bcf8d2bf25df6fdcc1bec23b2e00ab573c90e23fc92e6e266
MD5 8c12133d9c1a352331c2e7c4955dd612
BLAKE2b-256 653e29218cd7e745dccf33f7145f16664cc6be011d1311752e85ae443d3ec000

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page