Postgres to elasticsearch sync
Project description
PostgreSQL to Elasticsearch sync
PGSync <https://pgsync.com>
_ is a middleware for syncing data from Postgres <https://www.postgresql.org>
_ to Elasticsearch <https://www.elastic.co/products/elastic-stack>
.
It allows you to keep Postgres <https://www.postgresql.org>
as your source of truth data source and
expose structured denormalized documents in Elasticsearch <https://www.elastic.co/products/elastic-stack>
_.
Requirements
Python <https://www.python.org>
_ 3.6+Postgres <https://www.postgresql.org>
_ 9.4+Redis <https://redis.io>
_Elasticsearch <https://www.elastic.co/products/elastic-stack>
_ 6.3.1+
Postgres setup
Enable logical decoding <https://www.postgresql.org/docs/current/logicaldecoding.html>
_ in your
Postgres setting.
-
you would also need to set up two parameters in your Postgres config postgresql.conf
wal_level = logical
max_replication_slots = 1
Installation
You can install PGSync from PyPI <https://pypi.org>
_:
$ pip install pgsync
Config
Create a schema for the application named e.g schema.json
Example schema <https://github.com/toluaina/pg-sync/blob/master/examples/airbnb/schema.json>
_
Example spec
.. code-block::
[
{
"database": "[database name]",
"index": "[elasticsearch index]",
"nodes": [
{
"table": "[table A]",
"schema": "[table A schema]",
"columns": [
"column 1 from table A",
"column 2 from table A",
... additional columns
],
"children": [
{
"table": "[table B with relationship to table A]",
"schema": "[table B schema]",
"columns": [
"column 1 from table B",
"column 2 from table B",
... additional columns
],
"relationship": {
"variant": "object",
"type": "one_to_many"
},
...
},
{
... any other additional children
}
]
}
]
}
]
Environment variables
Setup required environment variables for the application
SCHEMA='/path/to/schema.json'
ELASTICSEARCH_HOST=localhost
ELASTICSEARCH_PORT=9200
PG_HOST=localhost
PG_USER=i-am-root # this must be a postgres superuser
PG_PORT=5432
PG_PASSWORD=*****
REDIS_HOST=redis
REDIS_PORT=6379
REDIS_DB=0
REDIS_AUTH=*****
Running
bootstrap the database (one time only) $ bootstrap --config schema.json run pgsync as a daemon $ pgsync --config schema.json --daemon
======= History
1.0.1 (2020-15-01)
- First release on PyPI.
1.0.1 (2020-01-01)
- RC1 release
1.1.0 (2020-04-13)
-
Postgres multi schema support for multi-tennant applications
-
Show resulting Query with verbose mode
-
this release required you to re-bootstrap your database with
- bootstrap -t
- bootstrap
1.1.1 (2020-05-18)
- Fixed authentication with Redis
- Fixed Docker build
1.1.2 (2020-06-11)
- Sync multiple indices in the same schema
- Test for replication or superuser
- Fix PG_NOTIFY error when payload exceeds 8000 bytes limit
1.1.3 (2020-06-14)
- Bug fix when syncing multiple indices in the same schema
1.1.4 (2020-06-15)
- Only create triggers for tables present in schema
1.1.5 (2020-06-16)
- Bug fix when creating multiple triggers in same schema
1.1.6 (2020-07-31)
- Bug fix when tearing down secondary schema
1.1.7 (2020-08-16)
- Fix issue #29: SQLAlchemy err: Neither 'BooleanClauseList' object nor 'Comparator' object has an attribute '_orig'
1.1.8 (2020-08-19)
- Fix issue #30: Traceback AttributeError: id
1.1.9 (2020-08-26)
- Fix issue #33: Unable to set Redis port via environment variable.
1.1.10 (2020-08-29)
- Support Amazon RDS #16
- Optimize database reflection on startup
- Show elapsed time
1.1.11 (2020-09-07)
- Support specify Elasticsearch field data type
1.1.12 (2020-09-08)
- Add support for SSL TCP/IP connection mode
1.1.13 (2020-09-09)
- Show version details with --version argument
- Fixed airbnb examples docker build
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for pgsync-1.1.13-cp38-cp38-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b720851052c48c4f950b5e7e4595a6e23a2152867726b986002cb2d7bdbd7475 |
|
MD5 | 50e9ddb83967b08cb71deb9fb04cd686 |
|
BLAKE2b-256 | 5ff787544c931a263ff927f035b9d2bd44723bef3c5d05752b36d877e773a2b6 |
Hashes for pgsync-1.1.13-cp38-cp38-manylinux2010_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f83ffb1949797ede7d66a886d980eb14c3c65fd78f4903fcd848d9d688708286 |
|
MD5 | 75a2bed7d367f37aefcce55f7601f940 |
|
BLAKE2b-256 | 20551e80d5c9999b6823065e11817e9e7151124f0314ec114c2b003478abbd32 |
Hashes for pgsync-1.1.13-cp38-cp38-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 969de665bbf54f5ae0b52d49ae282ae019404045c8688d0ec13028cabac7d639 |
|
MD5 | 2f563444cd10242dbfcece6c6e3b74b7 |
|
BLAKE2b-256 | 9d9cc15e6150c2fa5a06fd3b2bfe70b85145451f8039f2d61651a43d956efd41 |
Hashes for pgsync-1.1.13-cp37-cp37m-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5bdf42340343600f049b07d2c27f280546fe7eed2329e177a1cbf8407f1fd388 |
|
MD5 | 28a3dd119e9d6bf4c35c078811684287 |
|
BLAKE2b-256 | b8e4b1a6214f892d4bfcb20dc1909b784fa2cd04938831711275a7dceafec04f |
Hashes for pgsync-1.1.13-cp37-cp37m-manylinux2010_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9baf7417ad736349dba46ff99eb13eb8755db6aeb2b03fd3de588477104ebcd2 |
|
MD5 | 5e513f92bc6733ebf2392ed91b20704d |
|
BLAKE2b-256 | 927a6cd1335e48d2c4a6cf814ab40b30df8cc4c463488d99ac92af57f4612839 |
Hashes for pgsync-1.1.13-cp37-cp37m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fb55b9b8e3b04e207dc5c510e6f20faf02f33135129de17e5f274b1c5b703c1d |
|
MD5 | d0ad21102cbddcd3ddee28bc03cc84fb |
|
BLAKE2b-256 | ce8de50ee565c20ab97d97a49dfd5803d7164004aae2a80272b2afde394367a7 |
Hashes for pgsync-1.1.13-cp37-cp37m-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 92bdaa21a8c661484db34c27748d0536407bed1c07e7f80603d4eaafeebcf4df |
|
MD5 | 94b50f57487ef2778d5f99a8c726147d |
|
BLAKE2b-256 | 2c1e5de013ed70b459d97c9147b27fdb1563ee9accdadd67ce053b62b08f25ca |
Hashes for pgsync-1.1.13-cp36-cp36m-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 18e2ea3069da185f5ecf1e2cac1654400596db726f637f1ca02c10d2b9af87e3 |
|
MD5 | aeb9d2384a43594d5774972108a8035b |
|
BLAKE2b-256 | 4a847c0ae890b00d2a8e0df72aac1f43c164e08978f0945e8be69d6e34693754 |
Hashes for pgsync-1.1.13-cp36-cp36m-manylinux2010_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9b262b5e9bb47771433592ece8b2b62fbc8b332b06ac26e57f42ccf3f5e0a82a |
|
MD5 | 8d51f5d1231ef6234ed6baa78ddb97be |
|
BLAKE2b-256 | e2831b42d1dd18b31d1b430e3166d35d545673135ba167b401b27cf5f73ed7e5 |
Hashes for pgsync-1.1.13-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6aa8b332e10c0b544190bdde48764fb0481b4bff6bcc132a9c47cf589b49239f |
|
MD5 | 9a5113d3a47ac36194784c9db36bc063 |
|
BLAKE2b-256 | b5758242fccd468ae4359e2d80eb27947a274200857beba16b1997d34dc0c0ef |