core-cdc 2.0.0

This library provides the mechanism to implement Change Data Capture (CDC) services...

It provides the core mechanism and required resources to implement “Change Data Capture” services…

---

Execution Environment

Install libraries

pip install --upgrade pip
pip install virtualenv

Create the Python Virtual Environment.

virtualenv --python={{python-version}} .venv
virtualenv --python=python3.11 .venv

Activate the Virtual Environment.

source .venv/bin/activate

Install required libraries

pip install .

Optional libraries

pip install '.[all]'  # For all...
pip install '.[mysql]'
pip install '.[mongo]'
pip install '.[snowflake]'

Check tests and coverage

python manager.py run-tests
python manager.py run-tests --test-type integration
python manager.py run-coverage

# Having the docker containers up and running you can execute the functional
# tests that ensure the CDC services are working as expected...
python manager.py run-tests --test-type functional --pattern "*.py"

Engines

The following database engines have CDC implementations:

Fully Implemented

MySQL - Binary Log (BinLog) based CDC

• Uses mysql-replication library

• Captures INSERT, UPDATE, DELETE operations

• Supports DDL events (CREATE, ALTER, DROP)

• Fallback mechanism for column name resolution

• See: core_cdc/processors/mysql/

MongoDB - Change Streams based CDC

• Uses native MongoDB Change Streams

• Captures INSERT, UPDATE, DELETE operations

• Requires replica set configuration

• Real-time event streaming

• See: core_cdc/processors/mongo/

In Development

MS SQL Server - Abstract base class defined

• See: core_cdc/processors/mssql.py

Oracle - Abstract base class defined

• See: core_cdc/processors/oracle.py