pgsync

PostgreSQL/MySQL/MariaDB to Elasticsearch/OpenSearch sync

PGSync is a middleware for syncing data from PostgreSQL, MySQL, or MariaDB to Elasticsearch or OpenSearch.

Keep your relational database as the source of truth and expose structured denormalized documents in your search engine.

Django integration: Use django-pgsync to integrate PGSync with Django projects.

Key Features

Real-time sync via logical decoding (PostgreSQL) or binary log (MySQL/MariaDB)
Denormalize complex relational data into nested search documents
JSON schema-based configuration
Support for one-to-one, one-to-many relationships
Plugin system for document transformation
Multiple operation modes: daemon, polling, or direct WAL streaming

Requirements

Python 3.10+
PostgreSQL 9.6+ or MySQL 8.0.0+ or MariaDB 12.0.0+
Redis 3.1.0+ or Valkey 7.2.0+ (optional in WAL mode)
Elasticsearch 6.3.1+ or OpenSearch 1.3.7+

Installation

Install from PyPI:

pip install pgsync

Database Setup

PostgreSQL

Enable logical decoding in your PostgreSQL configuration (postgresql.conf):

wal_level = logical
max_replication_slots = 1

MySQL / MariaDB

Enable binary logging in your MySQL/MariaDB configuration (my.cnf):

server-id = 1
log_bin = mysql-bin
binlog_row_image = FULL
binlog_expire_logs_seconds = 604800

Create a replication user:

CREATE USER 'replicator'@'%' IDENTIFIED WITH mysql_native_password BY 'password';
GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'replicator'@'%';
FLUSH PRIVILEGES;

Configuration

Create a JSON schema file (e.g., schema.json) defining your sync mapping:

[
    {
        "database": "book",
        "index": "book",
        "nodes": {
            "table": "book",
            "columns": ["isbn", "title", "description"],
            "children": [
                {
                    "table": "publisher",
                    "columns": ["name"],
                    "relationship": {
                        "variant": "object",
                        "type": "one_to_one"
                    }
                },
                {
                    "table": "author",
                    "label": "authors",
                    "columns": ["name", "date_of_birth"],
                    "relationship": {
                        "variant": "object",
                        "type": "one_to_many",
                        "through_tables": ["book_author"]
                    }
                }
            ]
        }
    }
]

See the examples directory for more schema examples (airbnb, social, rental, etc.).

Environment Variables

Configure PGSync via environment variables:

# Schema
SCHEMA='/path/to/schema.json'

# PostgreSQL
PG_HOST=localhost
PG_PORT=5432
PG_USER=postgres
PG_PASSWORD=*****

# Elasticsearch / OpenSearch
ELASTICSEARCH_HOST=localhost
ELASTICSEARCH_PORT=9200

# Redis (optional in WAL mode)
REDIS_HOST=localhost
REDIS_PORT=6379

Running

Bootstrap (run once to set up triggers and replication slots):

bootstrap --config schema.json

Run as daemon:

pgsync --config schema.json --daemon

License

MIT License - see LICENSE for details.

Release files for pgsync 7.3.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pgsync 7.3.0
File	Size	Uploaded
pgsync-7.3.0.tar.gz	183.6 kB	Aug 13, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for pgsync 7.3.0
File	Interpreter	ABI	Platform	Reset
pgsync-7.3.0-py3-none-any.whl 85.5 kB Aug 13, 2026	Python 3	none	any	Details

Total release size: 269.2 kB

Release files / pgsync-7.3.0.tar.gz

Download URL	pgsync-7.3.0.tar.gz
Size	183.6 kB
Tags	Source
SHA-256 checksum How to use checksums	`0ea5c355e417dfd5d064593c59fdd7d2d92e978a8a1c110f25168a97ede306bc`
BLAKE2b-256 checksum How to use checksums	`663a2c5dd1e691a16a7bd30f2c085c9e239e85632e371ed2c1661a7c102975f4`
Upload date	Aug 13, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/toluaina/pgsync
Publishing commit: github.com/toluaina/pgsync/tree/239c9e216275dced2b4f4938f29a9386f5490da7

Workflow

Publishing configuration: github.com/toluaina/pgsync/blob/239c9e216275dced2b4f4938f29a9386f5490da7/.github/workflows/python-publish.yml
Publishing logs: github.com/toluaina/pgsync/actions/runs/31738357666/attempts/1

Artifact

Subject: pgsync-7.3.0.tar.gz
SHA-256 checksum: 0ea5c355e417dfd5d064593c59fdd7d2d92e978a8a1c110f25168a97ede306bc Verifying attestations

Release files / pgsync-7.3.0-py3-none-any.whl

Download URL	pgsync-7.3.0-py3-none-any.whl
Size	85.5 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`f1b4f8aa4104bd1de14b4c02fdd954947c8b6144e03ae7c2268c511612ece01f`
BLAKE2b-256 checksum How to use checksums	`567f972c5462f44f6cb2789674743ce9d335019fb7bd20c0bb901d454f61ee7a`
Upload date	Aug 13, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/toluaina/pgsync
Publishing commit: github.com/toluaina/pgsync/tree/239c9e216275dced2b4f4938f29a9386f5490da7

Workflow

Publishing configuration: github.com/toluaina/pgsync/blob/239c9e216275dced2b4f4938f29a9386f5490da7/.github/workflows/python-publish.yml
Publishing logs: github.com/toluaina/pgsync/actions/runs/31738357666/attempts/1

Artifact

Subject: pgsync-7.3.0-py3-none-any.whl
SHA-256 checksum: f1b4f8aa4104bd1de14b4c02fdd954947c8b6144e03ae7c2268c511612ece01f Verifying attestations