Postgres to elasticsearch sync
Project description
PostgreSQL to Elasticsearch sync
PGSync <https://pgsync.com>
_ is a middleware for syncing data from Postgres <https://www.postgresql.org>
_ to Elasticsearch <https://www.elastic.co/products/elastic-stack>
.
It allows you to keep Postgres <https://www.postgresql.org>
as your source of truth data source and
expose structured denormalized documents in Elasticsearch <https://www.elastic.co/products/elastic-stack>
_.
Requirements
Python <https://www.python.org>
_ 3.6+Postgres <https://www.postgresql.org>
_ 9.4+Redis <https://redis.io>
_Elasticsearch <https://www.elastic.co/products/elastic-stack>
_ 6.3.1+
Postgres setup
Enable logical decoding <https://www.postgresql.org/docs/current/logicaldecoding.html>
_ in your
Postgres setting.
-
you would also need to set up two parameters in your Postgres config postgresql.conf
wal_level = logical
max_replication_slots = 1
Installation
You can install PGSync from PyPI <https://pypi.org>
_:
$ pip install pgsync
Config
Create a schema for the application named e.g schema.json
Example schema <https://github.com/toluaina/pg-sync/blob/master/examples/airbnb/schema.json>
_
Example spec
.. code-block::
[
{
"database": "[database name]",
"index": "[elasticsearch index]",
"nodes": [
{
"table": "[table A]",
"schema": "[table A schema]",
"columns": [
"column 1 from table A",
"column 2 from table A",
... additional columns
],
"children": [
{
"table": "[table B with relationship to table A]",
"schema": "[table B schema]",
"columns": [
"column 1 from table B",
"column 2 from table B",
... additional columns
],
"relationship": {
"variant": "object",
"type": "one_to_many"
},
...
},
{
... any other additional children
}
]
}
]
}
]
Environment variables
Setup required environment variables for the application
SCHEMA='/path/to/schema.json'
ELASTICSEARCH_HOST=localhost
ELASTICSEARCH_PORT=9200
PG_HOST=localhost
PG_USER=i-am-root # this must be a postgres superuser
PG_PORT=5432
PG_PASSWORD=*****
REDIS_HOST=redis
REDIS_PORT=6379
REDIS_DB=0
REDIS_AUTH=*****
Running
bootstrap the database (one time only) $ bootstrap --config schema.json run pgsync as a daemon $ pgsync --config schema.json --daemon
======= History
1.0.1 (2020-15-01)
- First release on PyPI.
1.0.1 (2020-01-01)
- RC1 release
1.1.0 (2020-04-13)
-
Postgres multi schema support for multi-tennant applications
-
Show resulting Query with verbose mode
-
this release required you to re-bootstrap your database with
- bootstrap -t
- bootstrap
1.1.1 (2020-05-18)
- Fixed authentication with Redis
- Fixed Docker build
1.1.2 (2020-06-11)
- Sync multiple indices in the same schema
- Test for replication or superuser
- Fix PG_NOTIFY error when payload exceeds 8000 bytes limit
1.1.3 (2020-06-14)
- Bug fix when syncing multiple indices in the same schema
1.1.4 (2020-06-15)
- Only create triggers for tables present in schema
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for pgsync-1.1.4-cp38-cp38-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b1b37aaebe878de54136c5c4c1aeb2ca00ca630bea575110441df77780a338f2 |
|
MD5 | 0ed64dbd7e816eb81239a671e5935e83 |
|
BLAKE2b-256 | 319c31749af962a7872e58bd102af655158d2813bddbee9bcd9b8edcb7f7b12c |
Hashes for pgsync-1.1.4-cp38-cp38-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fa1774e2e780062de3e006a7e77492f3f7a4a33c90811a922deed957d5e1b7a8 |
|
MD5 | 369d11f2feccce52d2886b4d0ee1b756 |
|
BLAKE2b-256 | 9c72359809a72cafef6ffeb5977c3eb1259fd49f6b425a2f988fed0e995b06c4 |
Hashes for pgsync-1.1.4-cp38-cp38-manylinux1_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7f7bc40f5de3acb3b9968c56c24c7040b698d6841f073c738fd804acc30fb5a3 |
|
MD5 | 7eba84c0a33ee1b07b66299b9134ebd5 |
|
BLAKE2b-256 | 3ce9038f376efd07fa122dbac1d99f78dc83f03a2bacc152c38f712bf4f1af7c |
Hashes for pgsync-1.1.4-cp37-cp37m-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 77b8f8eda144cfe2dfafbbe86af473a2d9266df263f18d9442b3287f582df26c |
|
MD5 | 8b622614fe05d9cfbf701b6026ba9146 |
|
BLAKE2b-256 | 0871ad1134ce0f04a382f4b565e0afa2239ee7e7ef26193bc48ab0d1a9482865 |
Hashes for pgsync-1.1.4-cp37-cp37m-manylinux2010_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fb890af1eb2d593cc2d420378d63f7d068168e5dd22225383a8d254154470de9 |
|
MD5 | e7d7e402d726f78ddf97b0e4b642e6f2 |
|
BLAKE2b-256 | 63e1c466bff4c39ad0cafdf746160daa5c7991056f3f68d189d47bbe0e054b18 |
Hashes for pgsync-1.1.4-cp37-cp37m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dbc0927642a1f7d99ed9794170170300ed4f1cb92f0178bf14a222685dcdd224 |
|
MD5 | 279d6e95c996b38607c7ba123b31b367 |
|
BLAKE2b-256 | 24db5cc6ee7362ac2e03bfbc7150d4c2911b727766205333fb4ae3e66c3aefc7 |
Hashes for pgsync-1.1.4-cp37-cp37m-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dd9f00439618efd1479f96387c98c19e30c6ef451458bf3f8212bc1dc91d103e |
|
MD5 | 6074f0c3cb1ded4661e776dfbb9769f3 |
|
BLAKE2b-256 | 896ee9bfab2d5cd4ab3644e8872689c3940985d643afa5059414b050369ccd25 |
Hashes for pgsync-1.1.4-cp36-cp36m-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ea4012fd4b9d011f0dff04f2625c1e694cad8dc306e05b536ef25634be6db627 |
|
MD5 | 00caf2293b5df0ec49a363b027158484 |
|
BLAKE2b-256 | f535390df3a1f954b798e8a769b37e154e52007b5bfafce552aac7e822d42652 |
Hashes for pgsync-1.1.4-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f9d6db8360372db5051a09902486f1b1517c4bddb3bf60c197cb101e50ab4fba |
|
MD5 | d98b96fc08088d670f926246bf1dedbb |
|
BLAKE2b-256 | 40d41458ba4a3a11b643a7c1cc8857c4f968ef0296d3055249659572ff7b32ca |
Hashes for pgsync-1.1.4-cp36-cp36m-manylinux1_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 930970ff132eb1fa200ab94121639d1a33b0e2a3bfb6b3048d34bf2c9ba54c5a |
|
MD5 | ab9ec8f968d7313dbca59c9ba9328546 |
|
BLAKE2b-256 | 5d6abd41ed977433b53370517720ed363b16bb614e9930dfa4346f476e3be942 |