Postgres to elasticsearch sync
Project description
PostgreSQL to Elasticsearch sync
PGSync <https://pgsync.com>
_ is a middleware for syncing data from Postgres <https://www.postgresql.org>
_ to Elasticsearch <https://www.elastic.co/products/elastic-stack>
.
It allows you to keep Postgres <https://www.postgresql.org>
as your source of truth data source and
expose structured denormalized documents in Elasticsearch <https://www.elastic.co/products/elastic-stack>
_.
Requirements
Python <https://www.python.org>
_ 3.6+Postgres <https://www.postgresql.org>
_ 9.4+Redis <https://redis.io>
_Elasticsearch <https://www.elastic.co/products/elastic-stack>
_ 6.3.1+
Postgres setup
Enable logical decoding <https://www.postgresql.org/docs/current/logicaldecoding.html>
_ in your
Postgres setting.
-
you would also need to set up two parameters in your Postgres config postgresql.conf
wal_level = logical
max_replication_slots = 1
Installation
You can install PGSync from PyPI <https://pypi.org>
_:
$ pip install pgsync
Config
Create a schema for the application named e.g schema.json
Example schema <https://github.com/toluaina/pg-sync/blob/master/examples/airbnb/schema.json>
_
Example spec
.. code-block::
[
{
"database": "[database name]",
"index": "[elasticsearch index]",
"nodes": [
{
"table": "[table A]",
"schema": "[table A schema]",
"columns": [
"column 1 from table A",
"column 2 from table A",
... additional columns
],
"children": [
{
"table": "[table B with relationship to table A]",
"schema": "[table B schema]",
"columns": [
"column 1 from table B",
"column 2 from table B",
... additional columns
],
"relationship": {
"variant": "object",
"type": "one_to_many"
},
...
},
{
... any other additional children
}
]
}
]
}
]
Environment variables
Setup required environment variables for the application
SCHEMA='/path/to/schema.json'
ELASTICSEARCH_HOST=localhost
ELASTICSEARCH_PORT=9200
PG_HOST=localhost
PG_USER=i-am-root # this must be a postgres superuser
PG_PORT=5432
PG_PASSWORD=*****
REDIS_HOST=redis
REDIS_PORT=6379
REDIS_DB=0
REDIS_AUTH=*****
Running
bootstrap the database (one time only) $ bootstrap --config schema.json run pgsync as a daemon $ pgsync --config schema.json --daemon
======= History
1.0.1 (2020-15-01)
- First release on PyPI.
1.0.1 (2020-01-01)
- RC1 release
1.1.0 (2020-04-13)
-
Postgres multi schema support for multi-tennant applications
-
Show resulting Query with verbose mode
-
this release required you to re-bootstrap your database with
- bootstrap -t
- bootstrap
1.1.1 (2020-05-18)
- Fixed authentication with Redis
- Fixed Docker build
1.1.2 (2020-06-11)
- Sync multiple indices in the same schema
- Test for replication or superuser
- Fix PG_NOTIFY error when payload exceeds 8000 bytes limit
1.1.3 (2020-06-14)
- Bug fix when syncing multiple indices in the same schema
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for pgsync-1.1.3-cp38-cp38-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 58795d7b2125e98c2b9b16adc042ad109eefd53e515d6f75ff70c22a6e8b0aa4 |
|
MD5 | a58fe8566a25ef07673c886d2981f6f8 |
|
BLAKE2b-256 | b0cc7f5edf9f6b358aed620fd3e2981b2b329cacbf14613bab915455ec2ccbe5 |
Hashes for pgsync-1.1.3-cp38-cp38-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 43f8f9c36a0887356d04204296cecea2e1ef878b37429f660eab4e48464ca149 |
|
MD5 | 33093c4ac0bae04626d04cd68b4b3067 |
|
BLAKE2b-256 | 67023d23127a267b64c6defc537fcf88b9de616822ab30754ec8366ed13162fa |
Hashes for pgsync-1.1.3-cp38-cp38-manylinux1_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 44f70561c91690aad668e0339a4ad0551e4388b6374eaa3dcaf9ec23bf7040c1 |
|
MD5 | ec1e91a96c90c2d417fd7795ef317124 |
|
BLAKE2b-256 | c482520767d4da0cc3dab6a6fe03fca3b73bc5e08dcdabb4061064dfbc6b134c |
Hashes for pgsync-1.1.3-cp37-cp37m-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5ea450bd1980a8c1c21eb2ba924ad4e54a4dafe97b43e7bd21ced65d12cfed75 |
|
MD5 | ff2dbcf583e4f4ca7dac902e30670502 |
|
BLAKE2b-256 | 52f7f73fcc2069c04fc64a10a5cde84b699a224482e8fc86119eba5ecd73b5eb |
Hashes for pgsync-1.1.3-cp37-cp37m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 03d8457671ed77b482fce546c6b0e0d22d34138db508419102525b85ff7815f2 |
|
MD5 | a0c79630be89eccb84267960945507f1 |
|
BLAKE2b-256 | ce9d76642f1927a654ab9f04a6a6d2912b7df741a876a7b379c4aff8f7334ef4 |
Hashes for pgsync-1.1.3-cp37-cp37m-manylinux1_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9f131acc974974f95628bb643334e730794a303e40bf1d8f6d8d3e527e6c5858 |
|
MD5 | cade44ded32838f8a457341495412cec |
|
BLAKE2b-256 | 8def767f74770678d27e0dbcac01de9ff17debd37d7304b6c4f2ef4cd85f27f9 |
Hashes for pgsync-1.1.3-cp37-cp37m-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 82041de0501c9f71962e0dc6133392248a1af6b8264ea1c14a018352a57ae6c2 |
|
MD5 | e284606068d0dc63be3296fed0cefa94 |
|
BLAKE2b-256 | 3a54b43e1e3005ce24daf3648ddcfd2c524536944926280305e00d73925e6a9d |
Hashes for pgsync-1.1.3-cp36-cp36m-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a3a67105ec9af6ec8f50e95797679089cb850dd31ccb9d0941e42c45ad4a73cc |
|
MD5 | 3f0dba4da14ee3d9785678f5c28e05af |
|
BLAKE2b-256 | 07ff066c4e3ce21f467861c59ebed8436c9c1c4eccd9959aa142e358e52aa9d9 |
Hashes for pgsync-1.1.3-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6aa8c3282d49a4edf01c206f44277936963b046861d11b373b6413ada42aae87 |
|
MD5 | 7d2ec121e46e86961e0ae4d95a3df2b2 |
|
BLAKE2b-256 | 405b0f36f199296cb08d02275d39171c5c90ae5a0c85c41fd318f5db0eb0a8f7 |
Hashes for pgsync-1.1.3-cp36-cp36m-manylinux1_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f636bff30a629e34c603ce6c824fd5e08daa1c889c2463f817c442502f3ee4b9 |
|
MD5 | df2694a66ee1ca6307cf611c5cc7bc9d |
|
BLAKE2b-256 | c6e10c3c9f2792e7f837281c88983a8ff52ea90e0322492bdbf78bd21fba7cc7 |