mistaker 0.4.2

A package for emulating common data entry errors

Mistaker

Mistaker is a Python package designed to emulate common data entry errors that occur in real-world datasets. It's particularly useful for testing data quality tools, generating synthetic training data, and simulating typical mistakes found in OCR output, manual transcription, and legacy data migration projects.

Features

• Simulate common transcription and data entry errors for:
  • Text strings and words
  • Personal names and business names
  • Dates in various formats
  • Numeric data
  • Addresses and locations
• Configurable error types and rates
• Support for multiple input formats
• Preserves data structure while introducing realistic errors
• Deterministic error generation available for testing

Installation

pip install mistaker

Quick Start

Command Line

Generate variations with mistakes from your CSV data:

# Basic usage
mistaker data.csv > output.csv

# Using standard input
cat data.csv | mistaker > output.csv

# The tool will automatically use config.json from current directory if it exists
# Or specify a custom config file:
mistaker data.csv -c custom_config.json

# Adjust mistake generation via command line
mistaker data.csv --min-duplicates 3 --max-duplicates 6 --min-chaos 1 --max-chaos 3

Configuration

Control the mistake generation process via a JSON configuration file:

{
    "min_duplicates": 2,
    "max_duplicates": 5,
    "min_chaos": 1,
    "max_chaos": 4,
    "missing_weights": {
        "full_name": 0.05,
        "dob": 0.1,
        "phone": 0.8,
        "email": 0.95,
        "ssn": 0.9,
        "dl_num": 0.5,
        "full_address": 0.1
    },
    "mistake_weights": {
        "full_name": 1.0,
        "dob": 1.0,
        "phone": 1.0,
        "email": 1.0,
        "ssn": 1.0,
        "dl_num": 1.0,
        "full_address": 1.0
    }
}

Supported Fields

Mistaker handles these fields with field-specific error patterns:

• full_name: Name variations and misspellings
• dob: Date format errors and typos
• phone: Number transpositions and formatting errors
• ssn: Number mistakes preserving SSN patterns
• dl_num: Alphanumeric mistakes for driver's licenses
• email: Username and domain-specific errors
• full_address: Address component errors with street numbers, names, suffixes, unit numbers, and directional prefixes

Advanced Usage

Python API

from mistaker import Generator

# Create generator with defaults
generator = Generator()

# Process a single record
record = {
    'full_name': 'John Smith',
    'dob': '1990-01-01',
    'phone': '555-123-4567',
    'full_address': '123 N Main St Apt 4B'
}

variations = generator.generate(record)  # Returns list with original + variations

# Process multiple records
records = [record1, record2, record3]
for variation in generator.generate_all(records):
    print(variation)

Python API Options

# Initialize with custom settings
generator = Generator(
    min_duplicates=3,
    max_duplicates=6,
    min_chaos=2,
    max_chaos=4
)

# Load from config file
generator = Generator.from_file('config.json')

# Custom configuration
config = {
    'missing_weights': {
        'full_name': 0.05,
        'phone': 0.2
    }
}
generator = Generator(config=config)

CLI Options

mistaker --help

usage: mistaker [input_file] [options]

options:
  -h, --help           show this help message and exit
  -c, --config CONFIG  configuration JSON file path
  --min-duplicates N   minimum number of variations per record
  --max-duplicates N   maximum number of variations per record
  --min-chaos N        minimum number of mistakes per field
  --max-chaos N        maximum number of mistakes per field
  -v, --version        show program's version number and exit

Basic Examples

from mistaker import Word, Name, Date, Number, Address

# Generate word variations
Word("GRATEFUL").mistake()   # => "GRATEFU"
Word("GRATEFUL").mistake()   # => "GRATAFUL"

# Generate name variations with common mistakes
Name("KIM DEAL").mistake()   # => "KIM FEAL"
Name("KIM DEAL").mistake()   # => "KIM DEL"
Name("KIM DEAL").chaos()     # => "DEELLL KIN"

# Generate date formatting errors and typos
Date("09/04/1982").mistake() # => "1928-09-04"
Date("09/04/1982").mistake() # => "0019-82-09"

# Generate numeric transcription errors
Number("12345").mistake()    # => "12335"
Number("12345").mistake()    # => "72345"

# Generate address variations and errors
Address("123 N Main St Apt 4B").mistake()  # => "123 N MANE ST APT 4D"
Address("456 South Oak Avenue").mistake()  # => "456 S OAK AVE"

Detailed Usage

Word and Text Errors

from mistaker import Word, ErrorType

# Create a word instance
word = Word("TESTING")

# Generate specific error types
word.mistake(ErrorType.DROPPED_LETTER)  # => "TESTNG"
word.mistake(ErrorType.DOUBLE_LETTER)   # => "TESSTING"
word.mistake(ErrorType.MISREAD_LETTER)  # => "TEZTING"
word.mistake(ErrorType.MISTYPED_LETTER) # => "TEDTING"
word.mistake(ErrorType.EXTRA_LETTER)    # => "TESTINGS"
word.mistake(ErrorType.MISHEARD_LETTER) # => "TEZDING"

Name Handling

from mistaker import Name

# Create a name instance
name = Name("Robert James Smith")

# Generate name variations
variations = name.get_name_variations()
# Returns variations like:
# - "Smith, Robert"
# - "R James Smith"
# - "Robert Smith"
# - "Smith Robert"

# Generate case variants
cases = name.get_case_variants()
# Returns:
# - "Robert James Smith"
# - "ROBERT JAMES SMITH"
# - "robert james smith"

# Generate multiple errors
name.chaos()  # Applies 1-6 random errors
# John Smith -> JAHN SMEH

Date Handling

from mistaker import Date

# Create a date instance
date = Date("2023-05-15")

# Supports multiple input formats
date = Date("05/15/2023")  # US format
date = Date("15/05/2023")  # UK format

# Generate specific error types
date.mistake(ErrorType.MONTH_DAY_SWAP)    # => "2023-15-05"
date.mistake(ErrorType.ONE_DECADE_DOWN)   # => "2013-05-15"
date.mistake(ErrorType.Y2K)               # => "0023-05-15"

Number Handling

from mistaker import Number

# Create a number instance
number = Number("12345")

# Generate specific error types
number.mistake(ErrorType.ONE_DIGIT_UP)     # => "12346"
number.mistake(ErrorType.ONE_DIGIT_DOWN)   # => "12344"
number.mistake(ErrorType.KEY_SWAP)         # => "21345"
number.mistake(ErrorType.DIGIT_SHIFT)      # => "01234"
number.mistake(ErrorType.MISREAD)          # => "12375"
number.mistake(ErrorType.NUMERIC_KEY_PAD)  # => "12348"

Address Handling

from mistaker import Address

# Create an address instance
address = Address("123 North Main Street Suite 100")

# Generate mistakes with automatic component handling
address.mistake()  # => "123 N MANE ST STE 102"

# Standardize address format
address.standardize()  # => "123 N MAIN ST STE 100"

# Handles various address components:
# - Street numbers
# - Directional prefixes (N, S, E, W, NE, NW, SE, SW)
# - Street names
# - Street suffixes (St, Ave, Rd, etc.)
# - Unit designators (Suite, Apt, Unit, etc.)
# - Unit numbers

Error Types

Text and Name Errors

• Dropped Letters: Missing characters (e.g., "testing" → "testng")
• Double Letters: Repeated characters (e.g., "testing" → "tessting")
• Misread Letters: Similar-looking character substitutions (e.g., "testing" → "tezting")
• Mistyped Letters: Keyboard proximity errors (e.g., "testing" → "tedting")
• Extra Letters: Common suffix additions (e.g., "test" → "tests")
• Misheard Letters: Phonetic errors (e.g., "testing" → "tesding")

Number Errors

• Single Digit Errors: Off-by-one errors
• Key Swaps: Adjacent digit transposition
• Digit Shifts: Decimal/position shifts
• Misread Numbers: Similar-looking number substitution
• Numeric Keypad Errors: Based on number pad layout

Date Errors

• Month/Day Swaps: Common in international formats
• Decade Shifts: Common in manual entry
• Y2K Issues: Two-digit year ambiguity
• All Number-Based Errors: Inherited from number handling

Address Errors

• Component Dropping: Omitting address parts (suffixes, unit numbers)
• Standardization Issues: Inconsistent formatting of prefixes and suffixes
• Number Errors: Street number and unit number mistakes
• Text Errors: Street name misspellings and variations
• Unit Formatting: Inconsistent unit designator abbreviations

Contributing

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• Inspired by real-world data quality challenges in government and enterprise systems
• Error patterns based on extensive analysis of common transcription mistakes
• Designed to support data quality testing and synthetic data generation