Skip to main content

Postgres to elasticsearch sync

Project description

PostgreSQL to Elasticsearch sync

PGSync <>_ is a middleware for syncing data from Postgres <>_ to Elasticsearch <>.
It allows you to keep Postgres <>
as your source of truth data source and expose structured denormalized documents in Elasticsearch <>_.


  • Python <>_ 3.6+
  • Postgres <>_ 9.4+
  • Redis <>_
  • Elasticsearch <>_ 6.3.1+

Postgres setup

Enable logical decoding <>_ in your Postgres setting.

  • You would also need to set up two parameters in your Postgres config postgresql.conf

    wal_level = logical

    max_replication_slots = 1


You can install PGSync from PyPI <>_:

$ pip install pgsync


Create a schema for the application named e.g schema.json

Example schema <>_

Example spec

.. code-block::

        "database": "[database name]",
        "index": "[elasticsearch index]",
        "nodes": [
                "table": "[table A]",
                "schema": "[table A schema]",
                "columns": [
                    "column 1 from table A",
                    "column 2 from table A",
                    ... additional columns
                "children": [
                        "table": "[table B with relationship to table A]",
                        "schema": "[table B schema]",
                        "columns": [
                          "column 1 from table B",
                          "column 2 from table B",
                          ... additional columns
                        "relationship": {
                            "variant": "object",
                            "type": "one_to_many"
                        ... any other additional children

Environment variables

Setup required environment variables for the application



PG_USER=i-am-root # this must be a postgres superuser



Bootstrap the database (one time only) $ bootstrap --config schema.json Run pgsync as a daemon $ pgsync --config schema.json --daemon

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for pgsync, version 1.1.24
Filename, size File type Python version Upload date Hashes
Filename, size pgsync-1.1.24.tar.gz (73.1 kB) File type Source Python version None Upload date Hashes View
Filename, size pgsync-1.1.24-py2.py3-none-any.whl (37.3 kB) File type Wheel Python version py2.py3 Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page