Postgres to elasticsearch sync
Project description
# PostgreSQL to Elasticsearch sync
PGSync is a middleware for syncing data from Postgres to Elasticsearch. It allows you to keep Postgres as your source of truth data source and expose structured denormalized documents in Elasticsearch.
### Requirements
Python 3.6+
Postgres 9.4+
Elasticsearch 6.3.1+
### Postgres setup
Enable logical decoding in your Postgres setting.
you would also need to set up at least two parameters at postgresql.conf
`wal_level = logical`
`max_replication_slots = 1`
### Installation
You can install PGSync from PyPI:
$ pip install pgsync
### Config
Create a schema for the application named e.g schema.json
Example spec
[
{
"index": "[database name]",
"nodes": [
{
"table": "[table A]",
"columns": [
"column 1 from table A",
"column 2 from table A",
... additional columns
],
"children": [
{
"table": "[table B with relationship to table A]",
"columns": [
"column 1 from table B",
"column 2 from table B",
... additional columns
],
"relationship": {
"variant": "object",
"type": "one_to_many"
},
...
},
{
... any other additional children
}
]
}
]
}
]
### Environment variables
Setup required environment variables for the application
SCHEMA=’/path/to/schema.json’
ELASTICSEARCH_HOST=localhost ELASTICSEARCH_PORT=9200
PG_HOST=localhost PG_USER=i-am-root # this must be a postgres superuser PG_PORT=5432 PG_PASSWORD=*****
REDIS_HOST=redis REDIS_PORT=6379 REDIS_DB=0 REDIS_AUTH=*****
## Running
- bootstrap the database (one time only)
$ bootstrap –config schema.json
- run pgsync as a daemon
$ pgsync –config schema.json –daemon
History
1.0.1 (2020-15-01)
First release on PyPI.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for pgsync-1.0.1-cp38-cp38-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ff0645b64c0f4d5a967277b78dc99ca8be5f592e40b7bad3c26c725e4776a263 |
|
MD5 | f658bff9414690ea86a770e1408b393c |
|
BLAKE2b-256 | 34d47868cc9c499d2df49e45d2ac9f11fc89ae66524bf29b8c0a317a36e9eb55 |
Hashes for pgsync-1.0.1-cp38-cp38-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7a880f9c6b682b7d4ce5fb56f95ee2588b6e8e2756ce8f054c1cd2b20af616ff |
|
MD5 | 40321582125c41e7392541576149ce3c |
|
BLAKE2b-256 | d91a63932943e81c6cd112e44054e33af48206a09164cbb51ece49b33fce92dc |
Hashes for pgsync-1.0.1-cp38-cp38-manylinux1_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ca9b4edd0735b55fe099a42ba7d945bbe2c92a8cc145eb38c2d6fe220422ada4 |
|
MD5 | 87eac25022c207523963a785b696b943 |
|
BLAKE2b-256 | ad4e49d96d54871157b0835c9c3a2016c7592bfa6fd7b2536a928047c60311d3 |
Hashes for pgsync-1.0.1-cp37-cp37m-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9527fed0313d14063a1743ff2459607d1708ce45409a7e70250ef841cc47767b |
|
MD5 | a7840fa4f9a93a0c7c2be3e201e5b9b0 |
|
BLAKE2b-256 | 33a716fea44b1e0faa95ed8d41e063f0d34c68f9023954c4e3ddd507235c1454 |
Hashes for pgsync-1.0.1-cp37-cp37m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b9e480dd124949507a31f84128aec1f9a466c50c559176fda89811f4d04e5ff3 |
|
MD5 | 04c45747e0c69893c9454027ab42b058 |
|
BLAKE2b-256 | 8f06a9eef0d67c3d4a674a62d1d5c05d6cb06b1787583bdc1533f47486ae3b4b |
Hashes for pgsync-1.0.1-cp37-cp37m-manylinux1_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 61686de4d38dfe820b5dbfcd41540a8cdfeb55753f20668ddabdbe40c5acc8ed |
|
MD5 | 61d3b1232e35ffadc1d911542a5c70b1 |
|
BLAKE2b-256 | 2078c840b4e689801037dc2b0b14f43691d5690cb164b36307f751eae2470c4d |
Hashes for pgsync-1.0.1-cp37-cp37m-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 310b0e7330d8f426e8dfcca7f53a86eb5d8aa6c57e66a4d344f132a81a33721e |
|
MD5 | f78d8f3c2b3fa1e3bae1e1e768d8da27 |
|
BLAKE2b-256 | ed9af282cd816e116e7cde1b39576519cf97a15d5e74bf7e2178c65c59003934 |
Hashes for pgsync-1.0.1-cp36-cp36m-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d8fac0640124c344e25916d79cef5759e2ed42b6111e1fae7185a6e7e92f24f8 |
|
MD5 | d876c90ca0a9ecda4140081b64d7af46 |
|
BLAKE2b-256 | 5759a312b7547e398fe52df32192312e9e8732d75b166b4e94cfa8aa6b9dd86a |
Hashes for pgsync-1.0.1-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 36effc7a10caa6feaa04ee2932eea525f5249d9459c1ab2dba97e749cfe5b51b |
|
MD5 | 500b0bfa04dcda3f58a5b7f55ea42c60 |
|
BLAKE2b-256 | 7b2eb08d75c8c3e5b3d1f3040008570066750b679265490271b5fdb4b420c979 |
Hashes for pgsync-1.0.1-cp36-cp36m-manylinux1_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ebd7eecdbb80d2a4ecc1b6ea8fe25d42bf11d458a6252c792dd5d70eb41ecfb9 |
|
MD5 | 3feea294baf52ae3469f4dfab120124c |
|
BLAKE2b-256 | 87f38e8134b246b69b1512c42e14d5786742b5454fea51c2adb58f0168741f85 |