Skip to main content
Help the Python Software Foundation raise $60,000 USD by December 31st!  Building the PSF Q4 Fundraiser

automated data cleaning tool

Project description

License: MIT

datacleanbot

Automated Data Cleaning Tool. The main goal is to develop a Python tool datacleanbot such that: Given a random parsed raw dataset representing a supervised learning problem, the Python tool is capable of automatically identifying the potential issues and reporting the results and recommendations to the end-user in an effective way.

Install

$ pip install datacleanbot

QuickStart

Install OpenML (version 0.9.0):

OpenML is used to easily import datasets and share models and experiments.

$ pip install openml

For Windows, you need to have C++ Compiler installed.

Acquire data from OpenML:

>>> import openml as oml
>>> data = oml.datasets.get_dataset(id) # id: openml dataset id
>>> X, y, categorical_indicator, features = data.get_data(target=data.default_target_attribute, dataset_format='array')
>>> Xy = np.concatenate((X,y.reshape((y.shape[0],1))), axis=1)

Autoclean data with datacleanbot:

>>> import datacleanbot.dataclean as dc
>>> Xy = dc.autoclean(Xy, data.name, features)

Description

datacleanbot is equipped with the following capabilities:

  • Present an overview report of the given dataset
    • The most important features
    • Statistical information (e.g., mean, max, min)
    • Data types of features
  • Clean common data problems in the raw dataset
    • Duplicated records
    • Inconsistent column names
    • Missing values
    • Outliers

The two aspects datacleanbot meaningfully automates are marked in bold.

User's Guide

The user's guide can be found at datacleanbot.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for datacleanbot, version 0.91
Filename, size File type Python version Upload date Hashes
Filename, size datacleanbot-0.91-py3-none-any.whl (199.1 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size datacleanbot-0.91.tar.gz (14.9 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page