bamboochute 1.2.1.1

Data cleaning package built on top of Pandas

BambooChute: Data Cleaning for Pandas

BambooChute is a comprehensive data cleaning toolkit built on top of Pandas, offering an array of functions to streamline your data preparation process.

Table of Contents

• Installation
• Getting Started
• Key Features Overview
  • 1. Data Loading
  • 2. Imputation (Handling Missing Data)
  • 3. Outlier Detection & Removal
  • 4. Categorical Data Processing
  • 5. Date Handling & Transformation
  • 6. Data Type Validation & Conversion
  • 7. Duplicates & Near-Duplicates
  • 8. Data Formatting & String Cleaning
  • 9. Data Profiling
  • 10. Pipelines
  • 11. Undo/Redo & Logging
  • 12. Data Validation Rules
• Example Usage
• Testing
• Contributing
• License

---

Installation

Install BambooChute using pip:

pip install BambooChute

(If you’re installing from source or a local repo, ensure any dependencies in requirements.txt are satisfied.)

---

Getting Started

BambooChute can load data from multiple file formats or directly from a Pandas DataFrame. Below is a minimal quickstart example demonstrating data loading, missing value imputation, outlier detection, and exporting.

import pandas as pd
from bamboochute import Bamboo

# 1. Load data
data = pd.read_csv("data.csv")
bamboo = Bamboo(data)

# 2. Preview data
print(bamboo.preview_data())

# 3. Impute missing values (mean for numeric, mode for categorical)
bamboo.impute_missing(strategy='mean')

# 4. Detect outliers with Z-score (threshold=3) and remove them
bamboo.remove_outliers(method='zscore', threshold=3)

# 5. Export cleaned data
bamboo.export_data("cleaned_data.csv", format="csv")

---

Key Features Overview

1. Data Loading

BambooChute supports:

• CSV: Bamboo("path/to/data.csv")
• Excel: Bamboo("path/to/data.xlsx")
• JSON: Bamboo("path/to/data.json")
• Pandas DataFrame: Bamboo(df)

It automatically infers the file type and loads it into a Pandas DataFrame, or uses the DataFrame you provide.

2. Imputation (Handling Missing Data)

BambooChute offers multiple imputation strategies out of the box:

• Basic Strategies
  • Mean, Median, Mode

  bamboo.impute_missing(strategy='mean')
  

• KNN Imputation

  bamboo.impute_knn(n_neighbors=5)
  

• Regression Imputation

  bamboo.impute_regression(target_column='Y', predictor_columns=['X1','X2'])
  

• MICE (Multiple Imputation by Chained Equations)

  bamboo.impute_mice(max_iter=10, tol=1e-3)
  

• EM (Expectation-Maximization)

  bamboo.impute_em(max_iter=100, tol=1e-3)
  

• Custom Function

  bamboo.fill_with_custom(lambda x: 'Unknown' if pd.isna(x) else x)
  

Or simply drop missing data:

bamboo.drop_missing(axis=0, how='any')

3. Outlier Detection & Removal

Various outlier detection methods and corresponding removal or clipping:

• Z-score, IQR, Isolation Forest, DBSCAN, LOF, Robust Covariance, Modified Z-score:

# Detect outliers with Z-score
outliers = bamboo.detect_outliers_zscore(threshold=3)

# Remove outliers with Isolation Forest
bamboo.remove_outliers_isolation_forest(contamination=0.1)

You can also clip outliers to a specific value or range:

bamboo.cap_outliers(method='iqr', lower_cap=0, upper_cap=100)

4. Categorical Data Processing

• Convert to Categorical

  bamboo.convert_to_categorical(columns=['CategoryColumn'])
  

• Categorical Encoding (One-Hot, Label, Frequency)

  bamboo.encode_categorical(method='onehot')
  bamboo.encode_frequency(['CategoryColumn'])
  

• Rare Category Detection & Replacement

  rare = bamboo.detect_rare_categories('CategoryColumn', threshold=0.01)
  bamboo.replace_rare_categories('CategoryColumn', replacement='Other')
  

5. Date Handling & Transformation

• Convert to Datetime

  bamboo.convert_to_datetime(['DateColumn'])
  

• Extract Date Parts (year, month, day, weekday…)

  bamboo.extract_date_parts('DateColumn', parts=['year','month','weekday'])
  

• Create/Shift/Round Dates & Detect Time Gaps

  bamboo.shift_dates(['DateColumn'], periods=7, freq='D')  # Shift a week forward
  missing_date_gaps = bamboo.detect_time_gaps('DateColumn', freq='D')
  

6. Data Type Validation & Conversion

• Check Data Type Consistency

  bamboo.check_dtype_consistency()
  

• Convert & Enforce Column Types

  bamboo.enforce_column_types({'Age': 'int64', 'Price': 'float64'})
  

• Detect Numeric & Categorical Columns

  numeric_cols = bamboo.detect_numeric_columns()
  cat_cols = bamboo.detect_categorical_columns()
  

7. Duplicates & Near-Duplicates

• Identify, Drop, or Mark Duplicates

  duplicates = bamboo.identify_duplicates(subset=['Name'])
  bamboo.drop_duplicates(keep='first')
  

• Merge Duplicates with different strategies (most_frequent, most_recent).
• Near-Duplicate Detection via fuzzy matching

  bamboo.handle_near_duplicates(column='Name', threshold=0.8)
  

8. Data Formatting & String Cleaning

• Trim Whitespace & Standardize Case

  bamboo.trim_whitespace().standardize_case(case='lower')
  

• Remove Special Characters

  bamboo.remove_special_characters(columns=['TextColumn'], chars_to_remove='@#$')
  

• Format Dates

  bamboo.format_dates(format='%Y-%m-%d', columns=['DateColumn'])
  

• Currency Formatting

  bamboo.standardize_currency_format(columns=['Price'])
  

9. Data Profiling

BambooChute provides an array of profiling methods:

• Basic Summary

  summary = bamboo.basic_summary()
  

• Missing Data Report

  missing_report = bamboo.missing_data_report()
  

• Outliers Report

  outliers_report = bamboo.outliers_report(method='zscore', threshold=3)
  

• Distribution & Correlation Reports

  bamboo.distribution_report(columns=['Price','Quantity'])
  corr_matrix = bamboo.correlation_report()
  

• Duplicate Report

  dup_report = bamboo.duplicate_report()
  

10. Pipelines

Create reproducible data cleaning pipelines:

from bamboochute import Bamboo, BambooPipeline

pipeline = BambooPipeline()
pipeline.add_step('impute_missing', strategy='mean')
pipeline.add_step('drop_missing', axis=0, how='any')
pipeline.add_step('remove_outliers', method='zscore', threshold=3)

# Save to JSON file
pipeline.save_pipeline("my_pipeline.json")

# Load & execute pipeline
loaded_pipeline = BambooPipeline.load_pipeline("my_pipeline.json")
bamboo = Bamboo("data.csv")
cleaned_bamboo = loaded_pipeline.execute_pipeline(bamboo)

11. Undo/Redo & Logging

• Undo/Redo: BambooChute automatically tracks changes:

  bamboo.save_state()  # Save a snapshot
  # ...some cleaning...
  bamboo.undo()        # Revert the last change
  bamboo.reset_data()  # Revert to original data
  

• Logging: By default, logging is on. You can enable/disable:

  from bamboochute.settings.log import set_logging
  set_logging(False)   # Turn off logging globally
  set_logging(True)    # Re-enable logging
  

12. Data Validation Rules

Built-in validation methods to ensure data integrity:

• Validate Missing Data

  no_missing = bamboo.validate_missing_data(columns=['ColumnA','ColumnB'])
  

• Validate Data Types

  dtype_ok = bamboo.validate_data_types({'ColumnA': 'int64', 'ColumnB': 'object'})
  

• Validate Value Ranges

  in_range = bamboo.validate_value_ranges(column='Age', min_value=0, max_value=120)
  

• Validate Unique Values, Valid Categories, Date Ranges, or write a Custom Validation function.

---

Example Usage

Below is a more extended snippet demonstrating how you might chain multiple cleaning operations:

import pandas as pd
from bamboochute import Bamboo

df = pd.read_csv("raw_dataset.csv")
bamboo = Bamboo(df)

# 1. Convert certain columns to date & categorical
bamboo.convert_to_datetime(columns=['date'])
bamboo.convert_to_categorical(columns=['category_col'])

# 2. Impute missing data with advanced methods
bamboo.impute_knn(n_neighbors=5, columns=['numeric_col1','numeric_col2'])
bamboo.impute_mice(columns=['numeric_col3'], max_iter=5)

# 3. Handle outliers
bamboo.remove_outliers_isolation_forest(contamination=0.05)

# 4. Clean strings & format currency
bamboo.trim_whitespace().standardize_case()
bamboo.standardize_currency_format(columns=['price_col'])

# 5. Validate and produce a summary report
assert bamboo.validate_missing_data() is True, "Error: Missing data!"
summary = bamboo.basic_summary()
print(summary)

# 6. Export cleaned dataset
bamboo.export_data("final_dataset.csv", format='csv')

---

Testing

BambooChute uses pytest for testing.
To run all tests in the tests folder:

pytest tests/

Optional flags:

• -v for verbose mode
• --maxfail=3 to stop after three test failures

---

Contributing

1. Fork the repository.
2. Create a new branch.
3. Make changes and submit a pull request.
   • You will receive an email asking about your changes—please reply.
4. Thank you for helping improve BambooChute!

---

License

BambooChute is released under the MIT License. You’re free to use, modify, and distribute this library for personal or commercial projects, subject to the license terms.

---

Happy Cleaning! For additional examples or advanced usage, refer to the documentation or explore the source code in this repo.