pandas-cleanroom 1.3.4

Tiny, extensible pandas utilities for aliasing and basic data cleaning

Cleanroom 🧹

Tiny, extensible pandas utilities for aliasing and basic data cleaning

Cleanroom provides a simple set of pure functions you can compose to clean messy real-world data. It's designed to be lightweight, extensible, and work seamlessly with pandas DataFrames.

🚀 Features

• Flexible Column Matching: Automatically detects column variations (first_name, First Name, f name, fname, etc.)
• Email Cleaning: Normalize email addresses to lowercase
• Name Formatting: Smart title case with proper handling of particles (van, de, etc.)
• Phone Standardization: Convert phone numbers to international format
• Address Normalization: Clean ZIP codes, standardize states and countries
• Schema-Based Cleaning: Apply custom cleaning rules with flexible schemas
• Auto-Clean: Automatically clean common columns with zero configuration

📦 Installation

pip install cleanroom

🔧 Quick Start

import pandas as pd
import cleanroom

# Sample messy data
df = pd.DataFrame({
    'f name': ['  john ', 'JANE'],
    'e-mail': ['[e-mail]', '[e-mail]'],
    'pnumber': ['[phone]', '555.987.6543'],
    'zip code': ['12345-6789', '98765'],
    'st': ['California', 'TX']
})

# One-line cleaning with flexible column matching
cleaned_df = cleanroom.auto_clean(df)
print(cleaned_df)

Output:

  f name e-mail              pnumber    zip code st
0   John  [e-mail]   [phone]  123456789  CA
1   Jane  [e-mail]      [phone]     98765  TX

📚 API Reference

Individual Cleaning Functions

clean_email(series)

Normalize email addresses to lowercase and strip whitespace.

emails = pd.Series(['  [e-mail]  ', '[e-mail]'])
cleaned = cleanroom.clean_email(emails)
# Result: ['[e-mail]', '[e-mail]']

clean_name(series, alias_map=None, case="title")

Clean and format names with smart title casing.

names = pd.Series(['  john smith  ', 'JANE DOE', 'bob o\'connor'])
cleaned = cleanroom.clean_name(names)
# Result: ['John Smith', 'Jane Doe', 'Bob O\'Connor']

# With custom aliases
aliases = {'johnny': 'John', 'bobby': 'Robert'}
cleaned = cleanroom.clean_name(names, alias_map=aliases)

clean_phone(series, default_country="US")

Standardize phone numbers to international format.

phones = pd.Series(['[phone]', '555.987.6543', '[phone]'])
cleaned = cleanroom.clean_phone(phones)
# Result: ['[phone]', '[phone]', '[phone]']

clean_number(series)

Extract only digits (useful for ZIP codes, IDs).

zips = pd.Series(['12345-6789', 'ABC 98765', '  54321  '])
cleaned = cleanroom.clean_number(zips)
# Result: ['123456789', '98765', '54321']

clean_state(series, state_map=None)

Standardize US state names to abbreviations.

states = pd.Series(['California', 'TX', 'new york'])
cleaned = cleanroom.clean_state(states)
# Result: ['CA', 'TX', 'NY']

clean_country(series, country_map=None)

Standardize country names to ISO codes.

countries = pd.Series(['United States', 'USA', 'United Kingdom'])
cleaned = cleanroom.clean_country(countries)
# Result: ['US', 'US', 'GB']

DataFrame Operations

auto_clean(df)

Automatically clean common columns with flexible name matching.

Recognizes these column patterns:

• Email: email, e-mail, mail, email_address, e_mail
• Names: first_name, First Name, fname, f_name, given_name
• Phone: phone, telephone, pnumber, phone_number, mobile
• Address: zip, zip_code, postal_code, state, country

# Works with any column naming convention
df_messy = pd.DataFrame({
    'First Name': ['john', 'JANE'],
    'family name': ['smith', 'DOE'],  
    'E-Mail Address': ['[e-mail]', '[e-mail]'],
    'telephone': ['[phone]', '555.987.6543']
})

cleaned = cleanroom.auto_clean(df_messy)

apply_schema(df, schema)

Apply custom cleaning rules with a flexible schema.

schema = {
    'clean_email': {
        'func': cleanroom.clean_email,
        'source': ['email', 'email_address', 'e_mail']  # Try multiple columns
    },
    'full_name': {
        'func': cleanroom.clean_name,
        'source': ['first_name', 'last_name'],  # Combine columns
        'kwargs': {'case': 'title'}
    }
}

cleaned_df = cleanroom.apply_schema(df, schema)

🌍 Flexible Column Matching

Cleanroom automatically handles various column naming conventions:

Data Type	Recognized Patterns
First Name	first_name, First Name, fname, f_name, f name, firstname, given_name
Last Name	last_name, Last Name, lname, surname, family_name, lastname
Email	email, e-mail, mail, email_address, e_mail
Phone	phone, telephone, pnumber, phone_number, mobile, cell
ZIP Code	zip, zip_code, zipcode, postal_code, postcode
State	state, st, province, region
Country	country, nation, nationality

🏗️ Design Philosophy

• Simple: Small set of pure functions you can compose
• Extensible: Pass custom alias maps and cleaning rules
• Flexible I/O: Works on pd.Series, DataFrame columns, or entire DataFrames
• Non-destructive: Always returns new objects, never modifies input data
• Pandas-native: Leverages pandas' powerful string operations

🧪 Examples

Real-world messy data

import pandas as pd
import cleanroom

# Typical messy customer data
customers = pd.DataFrame({
    'f name': ['  alice  ', 'BOB', 'Charlie Brown'],
    'surname': ['SMITH', 'jones', 'o\'connor'],
    'e-mail': ['[e-mail]', '[e-mail]', '  [e-mail]  '],
    'pnumber': ['555-1234', '[phone]', '+1.212.555.9876'],
    'zip code': ['12345-6789', 'ABC 90210', '10001'],
    'st': ['California', 'TX', 'new york'],
    'nation': ['USA', 'United States', 'US']
})

# Clean everything with one function call
clean_customers = cleanroom.auto_clean(customers)

Custom cleaning pipeline

# Build your own cleaning pipeline
def clean_customer_data(df):
    result = df.copy()
    
    # Clean emails
    if 'email' in result.columns:
        result['email'] = cleanroom.clean_email(result['email'])
    
    # Standardize names with custom aliases
    name_aliases = {'bobby': 'Robert', 'mike': 'Michael'}
    if 'first_name' in result.columns:
        result['first_name'] = cleanroom.clean_name(
            result['first_name'], 
            alias_map=name_aliases
        )
    
    return result

📋 Requirements

• Python 3.8+
• pandas >= 1.3.0
• numpy >= 1.21.0

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

📄 License

MIT License - see LICENSE file for details.

🔗 Links

• Documentation: GitHub README
• Issues: GitHub Issues
• Source Code: GitHub Repository