core-cdc 2.0.2

This library provides the mechanism to implement Change Data Capture (CDC) services...

core-cdc (CDC a.k.a Change Data Capture)

It provides the core mechanism and required resources to implement “Change Data Capture” services…

---

Installation

Install from PyPI using pip:

pip install core-cdc
uv pip install core-cdc  # Or using UV...

Features

Multi-Database CDC Support

• MySQL Binary Log (BinLog) based change capture

• MongoDB Change Streams for real-time event streaming

• Extensible processor architecture for additional database engines

Comprehensive Event Handling

• DML operations: INSERT, UPDATE, DELETE

• DDL operations: CREATE, ALTER, DROP (schemas and tables)

• Configurable event filtering by operation type

Flexible Target Replication

• MySQL target for database-to-database replication

• Snowflake target for data warehouse integration

• Support for multiple simultaneous targets

Standardized Data Format

• Common Record structure for cross-service integration

• Includes metadata: timestamps, transaction IDs, source position

• JSON serialization support for streaming and messaging systems

Production-Ready Features

• Built-in error handling and retry mechanisms

• Comprehensive logging for monitoring and debugging

• Optional event timestamp column for UPSERT/MERGE operations

Quick Start

Installation

Install the package:

pip install core-cdc
uv pip install core-cdc     # Or using UV...
pip install -e ".[dev]"     # For development...

Setting Up Environment

1. Install required libraries:

pip install --upgrade pip
pip install virtualenv

2. Create Python virtual environment:

virtualenv --python=python3.12 .venv

3. Activate the virtual environment:

source .venv/bin/activate

Install packages

pip install .
pip install -e ".[dev]"

Optional libraries

pip install '.[all]'  # For all...
pip install '.[mysql]'
pip install '.[mongo]'
pip install '.[snowflake]'

Check tests and coverage

python manager.py run-tests
python manager.py run-tests --test-type integration
python manager.py run-coverage

# Having the docker containers up and running you can execute the functional
# tests that ensure the CDC services are working as expected...
python manager.py run-tests --test-type functional --pattern "*.py"

Implemented CDC Engines

The following database engines have CDC implementations:

Fully Implemented

MySQL - Binary Log (BinLog) based CDC

• Uses mysql-replication library

• Captures INSERT, UPDATE, DELETE operations

• Supports DDL events (CREATE, ALTER, DROP)

• Fallback mechanism for column name resolution

• See: core_cdc/processors/mysql/

MongoDB - Change Streams based CDC

• Uses native MongoDB Change Streams

• Captures INSERT, UPDATE, DELETE operations

• Requires replica set configuration

• Real-time event streaming

• See: core_cdc/processors/mongo/

In Development

MS SQL Server - Abstract base class defined

• See: core_cdc/processors/mssql.py

Oracle - Abstract base class defined

• See: core_cdc/processors/oracle.py

Contributing

Contributions are welcome! Please:

1. Fork the repository

2. Create a feature branch

3. Write tests for new functionality

4. Ensure all tests pass: pytest -n auto

5. Run linting: pylint core_cdc

6. Run security checks: bandit -r core_cdc

7. Submit a pull request

License

This project is licensed under the MIT License. See the LICENSE file for details.

Links

• Documentation: https://core-cdc.readthedocs.io/en/latest/

• Repository: https://gitlab.com/bytecode-solutions/core/core-cdc

• Issues: https://gitlab.com/bytecode-solutions/core/core-cdc/-/issues

• Changelog: https://gitlab.com/bytecode-solutions/core/core-cdc/-/blob/master/CHANGELOG.md

• PyPI: https://pypi.org/project/core-cdc/

Support

For questions or support, please open an issue on GitLab or contact the maintainers.

Authors

• Alejandro Cora González - Initial work - [e-mail]