Postgres to elasticsearch sync
Project description
PostgreSQL to Elasticsearch sync
PGSync <https://pgsync.com>
_ is a middleware for syncing data from Postgres <https://www.postgresql.org>
_ to Elasticsearch <https://www.elastic.co/products/elastic-stack>
.
It allows you to keep Postgres <https://www.postgresql.org>
as your source of truth data source and
expose structured denormalized documents in Elasticsearch <https://www.elastic.co/products/elastic-stack>
_.
Requirements
Python <https://www.python.org>
_ 3.6+Postgres <https://www.postgresql.org>
_ 9.4+Redis <https://redis.io>
_Elasticsearch <https://www.elastic.co/products/elastic-stack>
_ 6.3.1+
Postgres setup
Enable logical decoding <https://www.postgresql.org/docs/current/logicaldecoding.html>
_ in your
Postgres setting.
-
you would also need to set up two parameters in your Postgres config postgresql.conf
wal_level = logical
max_replication_slots = 1
Installation
You can install PGSync from PyPI <https://pypi.org>
_:
$ pip install pgsync
Config
Create a schema for the application named e.g schema.json
Example schema <https://github.com/toluaina/pg-sync/blob/master/examples/airbnb/schema.json>
_
Example spec
.. code-block::
[
{
"database": "[database name]",
"index": "[elasticsearch index]",
"nodes": [
{
"table": "[table A]",
"schema": "[table A schema]",
"columns": [
"column 1 from table A",
"column 2 from table A",
... additional columns
],
"children": [
{
"table": "[table B with relationship to table A]",
"schema": "[table B schema]",
"columns": [
"column 1 from table B",
"column 2 from table B",
... additional columns
],
"relationship": {
"variant": "object",
"type": "one_to_many"
},
...
},
{
... any other additional children
}
]
}
]
}
]
Environment variables
Setup required environment variables for the application
SCHEMA='/path/to/schema.json'
ELASTICSEARCH_HOST=localhost
ELASTICSEARCH_PORT=9200
PG_HOST=localhost
PG_USER=i-am-root # this must be a postgres superuser
PG_PORT=5432
PG_PASSWORD=*****
REDIS_HOST=redis
REDIS_PORT=6379
REDIS_DB=0
REDIS_AUTH=*****
Running
bootstrap the database (one time only) $ bootstrap --config schema.json run pgsync as a daemon $ pgsync --config schema.json --daemon
======= History
1.0.1 (2020-15-01)
- First release on PyPI.
1.0.1 (2020-01-01)
- RC1 release
1.1.0 (2020-04-13)
-
Postgres multi schema support for multi-tennant applications
-
Show resulting Query with verbose mode
-
this release required you to re-bootstrap your database with
- bootstrap -t
- bootstrap
1.1.1 (2020-05-18)
- Fixed authentication with Redis
- Fixed Docker build
1.1.2 (2020-06-11)
- Sync multiple indices in the same schema
- Test for replication or superuser
- Fix PG_NOTIFY error when payload exceeds 8000 bytes limit
1.1.3 (2020-06-14)
- Bug fix when syncing multiple indices in the same schema
1.1.4 (2020-06-15)
- Only create triggers for tables present in schema
1.1.5 (2020-06-16)
- Bug fix when creating multiple triggers in same schema
1.1.6 (2020-07-31)
- Bug fix when tearing down secondary schema
1.1.7 (2020-08-16)
- Fix issue #29: SQLAlchemy err: Neither 'BooleanClauseList' object nor 'Comparator' object has an attribute '_orig'
1.1.8 (2020-08-19)
- Fix issue #30: Traceback AttributeError: id
1.1.9 (2020-08-26)
- Fix issue #33: Unable to set redis port via environment variable.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for pgsync-1.1.9-cp38-cp38-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fc5c90c1b6868f3c360acd8d45a1e833e59b524de5a362e75b3985eb6013b730 |
|
MD5 | 8f5cfeb5d40b9de768fcce7de5b20cdc |
|
BLAKE2b-256 | ad384a2a5fd6c79f1a088526eef08f8ade43dfb3f374192e99d8672e4571eecd |
Hashes for pgsync-1.1.9-cp38-cp38-manylinux2010_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6ca877ec437a0b5f97ef21a85c3aa2f99df324cb2f2bbe41d6561ddd412a8590 |
|
MD5 | 8c1cde7bd0ace6691faaa205b8e2b80c |
|
BLAKE2b-256 | 87bad20fa6ec0368280d52024b2d90bc2d1f4b47df3a3db9b5bda337c523aa77 |
Hashes for pgsync-1.1.9-cp38-cp38-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 17d8f9b930d24d49b41970967902b311fb63dc0979d0b61e82c54f10abbff55c |
|
MD5 | 61d9afb9c23b3027204f522e029ed297 |
|
BLAKE2b-256 | 74a8ef1a65c86382180c629475922909810deb78156bf3ddc2ec8586ac6672ba |
Hashes for pgsync-1.1.9-cp37-cp37m-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | baef282298167f756c5b3569f30d52ab6e26d78ab9a733680deed3589f6282b3 |
|
MD5 | 7cbad3a3a9ed3bfae661e42f6fcfcfaa |
|
BLAKE2b-256 | e0c817e0915d92b89b1bb10d7c0070caca9835723e191edd2e18162bd6aac5c3 |
Hashes for pgsync-1.1.9-cp37-cp37m-manylinux2010_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 959031b71140a069869adf004ee6f33fe98be55157ba03f096551db5afc671a3 |
|
MD5 | 2c926132093b75928747cf51920e3a88 |
|
BLAKE2b-256 | 6cc315a7a296f4146e7fa2c75005759c2f104587d6cacce57cdefab1b885ba99 |
Hashes for pgsync-1.1.9-cp37-cp37m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 17df54bad57373a33c9e55cc67393a5132553ddcab9398aae895c6a6bc037e44 |
|
MD5 | 99b2f17b3ff9c74e941af475cb4ed7f8 |
|
BLAKE2b-256 | d217c3c20ac82b611397dd1c97a45002f4b40ee386f544a93be7d6b9d98ce307 |
Hashes for pgsync-1.1.9-cp37-cp37m-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 68d0718e7685f956e58a097ef9b54cd65cedbd2f339c08f0ae063920cb67e500 |
|
MD5 | d339fae59a44b3bcbef5aab7faade47e |
|
BLAKE2b-256 | c944f61fc01b06dc33d819aa3f589dd1c60f0d48713a44cd32c2651eddcbf44c |
Hashes for pgsync-1.1.9-cp36-cp36m-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3a0177de95eba04ab77458b7eaa77e44a3ad4fe350e75abde379716481bfb1d9 |
|
MD5 | 820a907f5ca6392ab3030c92b6671040 |
|
BLAKE2b-256 | 0923c73d60665cd35c6642ec12f4a436e1e08193dac0d043eed7e149fc4bf1f6 |
Hashes for pgsync-1.1.9-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e1a3edc83f040048ab045f1425fb52ab62566077af2dee2d060d70fcabba3d35 |
|
MD5 | d5b21fc007838512245f1f000a3b5995 |
|
BLAKE2b-256 | 1aaf58f9333462c5a1ba04d70bd0d3b21fa0bbb57799390634515d227b8fbb7d |
Hashes for pgsync-1.1.9-cp36-cp36m-manylinux1_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 699fc8a4dc519d64678652bbcbcaea6a5d4b8a996dff0356e13d3734eb04d6cb |
|
MD5 | 111aefdf1bb44f77d40ec3abb40aa1ca |
|
BLAKE2b-256 | 76992abd21f21d90614ae058534ec3abfd717b2f2f317f2a043ba7f95f120c1f |