PyPI Version Python Versions License Downloads

PostgreSQL/MySQL/MariaDB to Elasticsearch/OpenSearch sync

PGSync is a middleware for syncing data from PostgreSQL, MySQL, or MariaDB to Elasticsearch or OpenSearch.

Keep your relational database as the source of truth and expose structured denormalized documents in your search engine.

Django integration: Use django-pgsync to integrate PGSync with Django projects.

Key Features

  • Real-time sync via logical decoding (PostgreSQL) or binary log (MySQL/MariaDB)

  • Denormalize complex relational data into nested search documents

  • JSON schema-based configuration

  • Support for one-to-one, one-to-many relationships

  • Plugin system for document transformation

  • Multiple operation modes: daemon, polling, or direct WAL streaming

Requirements

Installation

Install from PyPI:

pip install pgsync

Database Setup

PostgreSQL

Enable logical decoding in your PostgreSQL configuration (postgresql.conf):

wal_level = logical
max_replication_slots = 1

MySQL / MariaDB

Enable binary logging in your MySQL/MariaDB configuration (my.cnf):

server-id = 1
log_bin = mysql-bin
binlog_row_image = FULL
binlog_expire_logs_seconds = 604800

Create a replication user:

CREATE USER 'replicator'@'%' IDENTIFIED WITH mysql_native_password BY 'password';
GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'replicator'@'%';
FLUSH PRIVILEGES;

Configuration

Create a JSON schema file (e.g., schema.json) defining your sync mapping:

[
    {
        "database": "book",
        "index": "book",
        "nodes": {
            "table": "book",
            "columns": ["isbn", "title", "description"],
            "children": [
                {
                    "table": "publisher",
                    "columns": ["name"],
                    "relationship": {
                        "variant": "object",
                        "type": "one_to_one"
                    }
                },
                {
                    "table": "author",
                    "label": "authors",
                    "columns": ["name", "date_of_birth"],
                    "relationship": {
                        "variant": "object",
                        "type": "one_to_many",
                        "through_tables": ["book_author"]
                    }
                }
            ]
        }
    }
]

See the examples directory for more schema examples (airbnb, social, rental, etc.).

Environment Variables

Configure PGSync via environment variables:

# Schema
SCHEMA='/path/to/schema.json'

# PostgreSQL
PG_HOST=localhost
PG_PORT=5432
PG_USER=postgres
PG_PASSWORD=*****

# Elasticsearch / OpenSearch
ELASTICSEARCH_HOST=localhost
ELASTICSEARCH_PORT=9200

# Redis (optional in WAL mode)
REDIS_HOST=localhost
REDIS_PORT=6379

Running

Bootstrap (run once to set up triggers and replication slots):

bootstrap --config schema.json

Run as daemon:

pgsync --config schema.json --daemon

Links

License

MIT License - see LICENSE for details.

Project links

Data verified by PyPI on Aug 13, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for taina from gravatar.com taina

Credits

Author/Maintainer: Tolu Aina

GitHub Statistics

Data verified by PyPI on Aug 13, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

MIT License (MIT)

Requires

Python >=3.10.0

Tags

change data capture elasticsearch opensearch pgsync postgres mysql mariadb

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pgsync-7.3.0.tar.gz (183.6 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

pgsync-7.3.0-py3-none-any.whl (85.5 kB view details)

Uploaded Python 3

File details

Details for the file pgsync-7.3.0.tar.gz.

File metadata

  • Download URL: pgsync-7.3.0.tar.gz
  • Upload date:
  • Size: 183.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for pgsync-7.3.0.tar.gz
Algorithm Hash digest
SHA256 0ea5c355e417dfd5d064593c59fdd7d2d92e978a8a1c110f25168a97ede306bc
MD5 87784b2fa3063eb8586abe112470bb1f
BLAKE2b-256 663a2c5dd1e691a16a7bd30f2c085c9e239e85632e371ed2c1661a7c102975f4

See more details on using hashes here.

Provenance

The following attestation bundles were made for pgsync-7.3.0.tar.gz:

Publisher: python-publish.yml on toluaina/pgsync

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file pgsync-7.3.0-py3-none-any.whl.

File metadata

  • Download URL: pgsync-7.3.0-py3-none-any.whl
  • Upload date:
  • Size: 85.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for pgsync-7.3.0-py3-none-any.whl
Algorithm Hash digest
SHA256 f1b4f8aa4104bd1de14b4c02fdd954947c8b6144e03ae7c2268c511612ece01f
MD5 f143440dae1922b8563c66fde903862b
BLAKE2b-256 567f972c5462f44f6cb2789674743ce9d335019fb7bd20c0bb901d454f61ee7a

See more details on using hashes here.

Provenance

The following attestation bundles were made for pgsync-7.3.0-py3-none-any.whl:

Publisher: python-publish.yml on toluaina/pgsync

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page