Enhanced PostgreSQL to Elasticsearch Data Synchronization
Project description
⚡ pg2elastic
Enhanced PostgreSQL to Elasticsearch Data Synchronization
📚 Description
pg2elastic is fork of official pgsync. Everything that pgsync got, pg2elastic got too.
Welcome to pg2elastic, a fork of the official pgsync package, designed to provide seamless and efficient data synchronization between PostgreSQL databases and Elasticsearch clusters. Building upon the solid foundation of pgsync, pg2elastic inherits all of its powerful capabilities and takes them a step further.
Key Features:
- High-Performance Sync: pg2elastic inherits the robust data synchronization engine from pgsync, ensuring lightning-fast and reliable transfers.
- Real-time Indexing: Seamlessly mirror your PostgreSQL data into Elasticsearch indices, keeping them in sync in real-time.
- Schema Mapping: Easily define and customize the mapping of PostgreSQL schemas to Elasticsearch indexes, giving you full control over the data structure.
- Efficient Data Types Handling: pg2elastic effortlessly handles data type conversions, ensuring accurate representation across platforms.
- Continuous Enhancements: We are committed to actively maintaining and enhancing pg2elastic, incorporating the latest advancements in both PostgreSQL and Elasticsearch technologies.
- Whether you're working on a data-driven application or performing complex data analysis, pg2elastic empowers you with a streamlined and feature-rich solution for harmonizing your PostgreSQL and Elasticsearch ecosystems.
🛠️ Prerequisites
✨ Key Enhancements
- Loguru, Better Logging Module
PG_SCHEMA
environment variable to enhance performance by eliminating the need to scan all schemasREDIS_USERNAME
environment variable to specify redis usernameREDIS_PASSWORD
environment variable to specify redis passwordREDIS_ENDPOINT
environment variable to specify redis connection endpointREDIS_SSL
environment variable to specify if redis connection should use sslREDIS_CLUSTER
environment variable to specify if redis connection is clustered
🚀 Deployment
Manual Deployment
You need to run pg2elastic
command in order to initialize it.
-
Create a .env file using the
cp .env.sample .env
command and replace the existing environment variables with personal configuration settings. -
Download dependencies using
python setup.py develop
-
Start the app in pre-production mode by using
pg2elastic
command for development
If you do not run the full setup, you will get errors when running this package.
✅ Testing
$ export PG_SCHEMA=
$ flake8 pg2elastic tests
$ python setup.py test
🔊 Logs
This project comes with a loguru module for logging, the configurations
for loguru can be found in pg2elastic
bin file.
🚚 Deployment
$ python setup.py sdist bdist_wheel
$ twine upload dist/*
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pg2elastic-0.1.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f8df201fc9bd431362c217dd1876315617436efc3b71300c1524cc006dd1571e |
|
MD5 | b3cd96555c01bf03c4f500e04db17096 |
|
BLAKE2b-256 | 344568836645eaa457480eb6760bf80519b0f3814a3bf0f9d8b14800c8d9d4f3 |