deltaapply 0.0.1

Change Data Capture und Anwendung von Inserts, Updates und Deletes

🔄 DeltaApply

Change Data Capture (CDC) with automatic application of inserts, updates, and deletes

DeltaApply is a Python package that simplifies Change Data Capture operations by comparing two data sources and automatically applying the differences (inserts, updates, deletes) to synchronize them.

✨ Features

• Flexible Data Sources: CSV files ↔ pandas DataFrames ↔ Polars DataFrames ↔ Database tables
• Configurable Operations: Choose which CDC operations to apply (insert, update, delete, or combinations)
• Type Preservation: Maintains input/output data types consistently
• Database Integration: Full SQLAlchemy support for database operations
• High Performance: Uses Polars internally for fast data processing
• Type Safe: Complete type hints and comprehensive test coverage

🚀 Quick Start

Installation

uv add deltaapply
# or
pip install deltaapply

Basic Usage

from deltaapply import DeltaApply
import pandas as pd

# Sample data
source_df = pd.DataFrame({
    'id': [1, 2, 3, 4],
    'name': ['Alice', 'Bob Updated', 'Charlie', 'David'],
    'value': [10, 25, 30, 40]
})

target_df = pd.DataFrame({
    'id': [1, 2, 5],
    'name': ['Alice', 'Bob', 'Eve'],
    'value': [10, 20, 50]
})

# Initialize CDC
cdc = DeltaApply(
    source=source_df,
    target=target_df,
    key_columns=['id']
)

# Apply all changes
result = cdc.apply()
print(result)
# Result: DataFrame with synchronized data

📊 Data Source Support

DeltaApply supports multiple data source combinations:

# DataFrame to DataFrame
cdc = DeltaApply(source=df1, target=df2, key_columns=['id'])

# CSV files
cdc = DeltaApply(source='source.csv', target='target.csv', key_columns=['id'])

# Database tables
cdc = DeltaApply(
    source='source_table', 
    target='target_table',
    key_columns=['id'],
    source_connection='postgresql://user:pass@host/db',
    target_connection='postgresql://user:pass@host/db'
)

# Mixed sources
cdc = DeltaApply(
    source=df,                    # pandas DataFrame
    target='target_table',        # Database table
    key_columns=['id'],
    target_connection=engine
)

🔧 Configuration Options

Selective Operations

# Apply only inserts
result = cdc.apply(operations=['insert'])

# Apply only updates
result = cdc.apply(operations=['update'])

# Apply inserts and updates (no deletes)
result = cdc.apply(operations=['insert', 'update'])

# Convenience methods
result = cdc.apply_inserts_only()
result = cdc.apply_updates_only()
result = cdc.apply_deletes_only()

Dry Run & Summary

# Preview changes without applying
changes = cdc.apply(dry_run=True)
print(f"Inserts: {len(changes.inserts)}")
print(f"Updates: {len(changes.updates)}")
print(f"Deletes: {len(changes.deletes)}")

# Get summary statistics
summary = cdc.get_summary()
print(summary)
# Output: {'inserts': 2, 'updates': 1, 'deletes': 1, 'unchanged': 1, ...}

Composite Keys

# Multiple column primary key
cdc = DeltaApply(
    source=df1,
    target=df2,
    key_columns=['dept_id', 'emp_id']  # Composite key
)

🏗️ Architecture

DeltaApply consists of four main components:

1. DataSource - Unified abstraction for different input types
2. CDCOperations - Logic for detecting changes between datasets
3. TargetWriter - Handles applying changes to different output formats
4. DeltaApply - Main orchestration class that ties everything together

📋 Requirements

• Python ≥ 3.12
• polars ≥ 1.32.0
• pandas ≥ 2.3.1
• sqlalchemy ≥ 2.0.42
• pyarrow ≥ 21.0.0

🧪 Development

Setup

# Clone the repository
git clone https://github.com/yourusername/deltaapply.git
cd deltaapply

# Install dependencies
uv sync --dev

# Run tests
uv run pytest

# Run tests with coverage
uv run pytest --cov=src/deltaapply --cov-report=html

Testing

The package includes comprehensive tests covering:

• Unit tests for all components
• Integration tests for end-to-end workflows
• Database integration tests
• CSV file handling tests
• Edge cases and error conditions

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.