Postgres to elasticsearch sync
Project description
PostgreSQL to Elasticsearch sync
PGSync <https://pgsync.com>
_ is a middleware for syncing data from Postgres <https://www.postgresql.org>
_ to Elasticsearch <https://www.elastic.co/products/elastic-stack>
.
It allows you to keep Postgres <https://www.postgresql.org>
as your source of truth data source and
expose structured denormalized documents in Elasticsearch <https://www.elastic.co/products/elastic-stack>
_.
Requirements
Python <https://www.python.org>
_ 3.6+Postgres <https://www.postgresql.org>
_ 9.4+Redis <https://redis.io>
_Elasticsearch <https://www.elastic.co/products/elastic-stack>
_ 6.3.1+
Postgres setup
Enable logical decoding <https://www.postgresql.org/docs/current/logicaldecoding.html>
_ in your
Postgres setting.
-
you would also need to set up two parameters in your Postgres config postgresql.conf
wal_level = logical
max_replication_slots = 1
Installation
You can install PGSync from PyPI <https://pypi.org>
_:
$ pip install pgsync
Config
Create a schema for the application named e.g schema.json
Example schema <https://github.com/toluaina/pg-sync/blob/master/examples/airbnb/schema.json>
_
Example spec
.. code-block::
[
{
"database": "[database name]",
"index": "[elasticsearch index]",
"nodes": [
{
"table": "[table A]",
"schema": "[table A schema]",
"columns": [
"column 1 from table A",
"column 2 from table A",
... additional columns
],
"children": [
{
"table": "[table B with relationship to table A]",
"schema": "[table B schema]",
"columns": [
"column 1 from table B",
"column 2 from table B",
... additional columns
],
"relationship": {
"variant": "object",
"type": "one_to_many"
},
...
},
{
... any other additional children
}
]
}
]
}
]
Environment variables
Setup required environment variables for the application
SCHEMA='/path/to/schema.json'
ELASTICSEARCH_HOST=localhost
ELASTICSEARCH_PORT=9200
PG_HOST=localhost
PG_USER=i-am-root # this must be a postgres superuser
PG_PORT=5432
PG_PASSWORD=*****
REDIS_HOST=redis
REDIS_PORT=6379
REDIS_DB=0
REDIS_AUTH=*****
Running
bootstrap the database (one time only) $ bootstrap --config schema.json run pgsync as a daemon $ pgsync --config schema.json --daemon
======= History
1.0.1 (2020-15-01)
- First release on PyPI.
1.0.1 (2020-01-01)
- RC1 release
1.1.0 (2020-04-13)
-
Postgres multi schema support for multi-tennant applications
-
Show resulting Query with verbose mode
-
this release required you to re-bootstrap your database with
- bootstrap -t
- bootstrap
1.1.1 (2020-05-18)
- Fixed authentication with Redis
- Fixed Docker build
1.1.2 (2020-06-11)
- Sync multiple indices in the same schema
- Test for replication or superuser
- Fix PG_NOTIFY error when payload exceeds 8000 bytes limit
1.1.3 (2020-06-14)
- Bug fix when syncing multiple indices in the same schema
1.1.4 (2020-06-15)
- Only create triggers for tables present in schema
1.1.5 (2020-06-16)
- Bug fix when creating multiple triggers in same schema
1.1.6 (2020-07-31)
- Bug fix when tearing down secondary schema
1.1.7 (2020-08-16)
- Fix issue #29: SQLAlchemy err: Neither 'BooleanClauseList' object nor 'Comparator' object has an attribute '_orig'
1.1.8 (2020-08-19)
- Fix issue #30: Traceback AttributeError: id
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for pgsync-1.1.8-cp38-cp38-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 67e29c11e5babf9fcf15de7f412b8be2e67260b3b542a38840123f6e35e765d1 |
|
MD5 | a7bf5654648c1a4571392cbbfffd227d |
|
BLAKE2b-256 | f4fe8c860a99856a1c24edb90674ce479663b6ea8f372ef1507e052fac45d26f |
Hashes for pgsync-1.1.8-cp38-cp38-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0b0cd9bdbffda67595cc71eea07dd674448311a62e4d8a56c66dfe8ed2e365f0 |
|
MD5 | 098ec9bf012499322b121eaaa34c9192 |
|
BLAKE2b-256 | 07f3e6dd10ce12ba70163408c8cc9b1ee0fb5e6388ae0a589c75fea6d8adc7f6 |
Hashes for pgsync-1.1.8-cp38-cp38-manylinux1_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8480a24c70d0945ea1e87534bbe17ae53c8522a060d4998efd52ad5b9ddb9c34 |
|
MD5 | 029d463f06dc0472b49bd63417691d9d |
|
BLAKE2b-256 | 358e0bdbc8648976043cc144461e542110a78b2084535aaa76aaf5ace0c40b48 |
Hashes for pgsync-1.1.8-cp37-cp37m-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3ac5b34333648e5285faf5537ae81362bc6d1167840d8717d453b690d5461bcc |
|
MD5 | 5cc4643064b08cea2f5802ed79b3506f |
|
BLAKE2b-256 | c4e5d7f444a4d4c359ca6d929e21dc4937eb3437e54d5d82fbd98db0cffd8e97 |
Hashes for pgsync-1.1.8-cp37-cp37m-manylinux2010_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7a57ad44d16acfec015fe9165a6ee50366d7e0ea796fcdf669219746c0ed4eee |
|
MD5 | e2238cd59724af15249e3a01247db95d |
|
BLAKE2b-256 | e783582516bb79307d67e0252bfed561cf4b52ccaa96f9e0a6d345d2487fecad |
Hashes for pgsync-1.1.8-cp37-cp37m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d25c21e2b38e60fd9a18797e4933cead2aa7f23b8a3a532cfb1e66dfa7ef0675 |
|
MD5 | 7f1b8ca750b7b913f3caf26140ab9156 |
|
BLAKE2b-256 | a560bd416889c56d635f26d1821e574d14f59809004db492c367730dce12d416 |
Hashes for pgsync-1.1.8-cp37-cp37m-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 669cb3f92f13e76f06e3fad38385b11498a7f469eede635050f385cc8cc189f2 |
|
MD5 | f8f72df13b7a497f56a1cc983a56baa4 |
|
BLAKE2b-256 | 11149440c41515e25b82941c2b2e26fef7ba5d11a049f06c5fa124bd36334ea2 |
Hashes for pgsync-1.1.8-cp36-cp36m-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7056a3c4e526505bbe0e31fb4d57ca6236f4b47377e8e2f59b1142228a5954c2 |
|
MD5 | f9f1327c0f26c61737816088d61f0ed0 |
|
BLAKE2b-256 | 7027950a029ff2b90d795085999ac38410cd503d7ca3d78304a742e7159a6542 |
Hashes for pgsync-1.1.8-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ede83507bfb9e74feff566aa3b11126cd48dabe8c35b9ab7bb6e6b5195b90b6a |
|
MD5 | 45f7def519a0ea0443db6db165462a00 |
|
BLAKE2b-256 | 253be74c21e981dbb4a4f93663c4bb49359f481eafb6a5cdf22cc40b2c494275 |
Hashes for pgsync-1.1.8-cp36-cp36m-manylinux1_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 67a674d683269acd6b3258ede23d191232098a287b59b5f6f4247358afb65c5d |
|
MD5 | a265f970fc337c928cf10e8719be3f97 |
|
BLAKE2b-256 | 742fe8ef8ab81ac08f4ecb7ea2c300c0f0c2e9794c30c14028792ca240de7d55 |