Postgres to elasticsearch sync
Project description
PostgreSQL to Elasticsearch sync
PGSync <https://pgsync.com>
_ is a middleware for syncing data from Postgres <https://www.postgresql.org>
_ to Elasticsearch <https://www.elastic.co/products/elastic-stack>
.
It allows you to keep Postgres <https://www.postgresql.org>
as your source of truth data source and
expose structured denormalized documents in Elasticsearch <https://www.elastic.co/products/elastic-stack>
_.
Requirements
Python <https://www.python.org>
_ 3.6+Postgres <https://www.postgresql.org>
_ 9.4+Redis <https://redis.io>
_Elasticsearch <https://www.elastic.co/products/elastic-stack>
_ 6.3.1+
Postgres setup
Enable logical decoding <https://www.postgresql.org/docs/current/logicaldecoding.html>
_ in your
Postgres setting.
-
you would also need to set up two parameters in your Postgres config postgresql.conf
wal_level = logical
max_replication_slots = 1
Installation
You can install PGSync from PyPI <https://pypi.org>
_:
$ pip install pgsync
Config
Create a schema for the application named e.g schema.json
Example schema <https://github.com/toluaina/pg-sync/blob/master/examples/airbnb/schema.json>
_
Example spec
.. code-block::
[
{
"database": "[database name]",
"index": "[elasticsearch index]",
"nodes": [
{
"table": "[table A]",
"schema": "[table A schema]",
"columns": [
"column 1 from table A",
"column 2 from table A",
... additional columns
],
"children": [
{
"table": "[table B with relationship to table A]",
"schema": "[table B schema]",
"columns": [
"column 1 from table B",
"column 2 from table B",
... additional columns
],
"relationship": {
"variant": "object",
"type": "one_to_many"
},
...
},
{
... any other additional children
}
]
}
]
}
]
Environment variables
Setup required environment variables for the application
SCHEMA='/path/to/schema.json'
ELASTICSEARCH_HOST=localhost
ELASTICSEARCH_PORT=9200
PG_HOST=localhost
PG_USER=i-am-root # this must be a postgres superuser
PG_PORT=5432
PG_PASSWORD=*****
REDIS_HOST=redis
REDIS_PORT=6379
REDIS_DB=0
REDIS_AUTH=*****
Running
bootstrap the database (one time only) $ bootstrap --config schema.json run pgsync as a daemon $ pgsync --config schema.json --daemon
======= History
1.0.1 (2020-15-01)
- First release on PyPI.
1.0.1 (2020-01-01)
- RC1 release
1.1.0 (2020-04-13)
-
Postgres multi schema support for multi-tennant applications
-
Show resulting Query with verbose mode
-
this release required you to re-bootstrap your database with
- bootstrap -t
- bootstrap
1.1.1 (2020-05-18)
- Fixed authentication with Redis
- Fixed Docker build
1.1.2 (2020-06-11)
- Sync multiple indices in the same schema
- Test for replication or superuser
- Fix PG_NOTIFY error when payload exceeds 8000 bytes limit
1.1.3 (2020-06-14)
- Bug fix when syncing multiple indices in the same schema
1.1.4 (2020-06-15)
- Only create triggers for tables present in schema
1.1.5 (2020-06-16)
- Bug fix when creating multiple triggers in same schema
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for pgsync-1.1.5-cp38-cp38-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 67c7e4e5ca058140fff36cfcb720d035d7c15b6d6e7478f07043af71fd82f6dd |
|
MD5 | beb52386f9fe29b5c70b379aa1dd8fc1 |
|
BLAKE2b-256 | b8bd68c3d784a45339fc21a7bc85a96f8b5e9f1a3fb28f8c87dbd7ab4b0b2a22 |
Hashes for pgsync-1.1.5-cp38-cp38-manylinux2010_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 058707480822487c398f3f0e9b3359e1f11b943443a074d40bb2530ebef49694 |
|
MD5 | 7e2106c42fcab83e3bf244f463d9be9c |
|
BLAKE2b-256 | a95c308c0ef056106a4cd88951345c479efc80a409048aaa9addcac8ebf1da6b |
Hashes for pgsync-1.1.5-cp38-cp38-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d89f860d240d68585051b4ba644727a476d6c9e630dfcf22104296609406725b |
|
MD5 | 0552660152535d9fe09fe9ea5f987ab1 |
|
BLAKE2b-256 | 4548e8ab3093e75b13ba6bafa0e99a22e91728359f270a7d5f632aefe1e863f0 |
Hashes for pgsync-1.1.5-cp37-cp37m-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 261a252c4f4a884b30abeefda5d8849456359d0781176858891b0e5c5470a8db |
|
MD5 | e1c5b339ec087117d52bc4d8e89620f2 |
|
BLAKE2b-256 | 500652f483ce15456da99da4006948fb39b168c1d894fc0590e123aa6813f89a |
Hashes for pgsync-1.1.5-cp37-cp37m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 759234f9eda9f8e97d329faa80bedb32da7037fbcb4e9b747392c87b287ef303 |
|
MD5 | a87be78d334e9aea55959331d496347f |
|
BLAKE2b-256 | 135a06a152ed0a27dc3cd23f8fb22621fa8b15dd02ddf4927f170f6c6a2717e6 |
Hashes for pgsync-1.1.5-cp37-cp37m-manylinux1_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 58afb86a91b4eb6d4b55f7dd22a81191e1dd1623078a33bfee1e6791e6273cd2 |
|
MD5 | 16cbf03fb1021b718c52b813c54e3f4c |
|
BLAKE2b-256 | f25c7f4ef8e0f16a6e647d27358940ef8bb2ce8784783009996c62af27ce3a4d |
Hashes for pgsync-1.1.5-cp37-cp37m-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d984074c4c7197fa83ffa9a59d137a1a45a7f373b8ff6d40777036fe1ec4ea0d |
|
MD5 | 17d568f755e412d1280dd0cd9b600668 |
|
BLAKE2b-256 | d5cdae0342a03d788857a42f4c6e148e49852afb9422db51f538fea0444eb377 |
Hashes for pgsync-1.1.5-cp36-cp36m-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4fe3fedc17606ad58bb2dfeb7b6984cbe462efc51c9a5239607a904235f26e9a |
|
MD5 | 72c188842e348b8ee74c41b9c45b9f45 |
|
BLAKE2b-256 | 3d0e0edd8b61f9033dff4e6d0fb8ed3e8c6ab2325c7c88098eea21ddbd289e71 |
Hashes for pgsync-1.1.5-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 41e9e25d6ac8382779749c373b29b27a416136710e58a04e9d1247761261962b |
|
MD5 | ba1bac4b29061827280580fc92a7bf6e |
|
BLAKE2b-256 | ca261e53b3e1b33ce991f2f93664c684a46fdfde8a3c3c738feba0029b353705 |
Hashes for pgsync-1.1.5-cp36-cp36m-manylinux1_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 89fbc6785fbe4cbd356887d9dc6a4d3b120ac9d775005bd05d779b48ccb21c6e |
|
MD5 | 9a5242c77ff647db8b890abd0f4ca2d9 |
|
BLAKE2b-256 | 94012e911d7a6ce34ce69b2ca7f36808e2eda67b07e515d87b0ab76e97576869 |