Postgres to elasticsearch sync
Project description
PostgreSQL to Elasticsearch sync
PGSync <https://pgsync.com>
_ is a middleware for syncing data from Postgres <https://www.postgresql.org>
_ to Elasticsearch <https://www.elastic.co/products/elastic-stack>
.
It allows you to keep Postgres <https://www.postgresql.org>
as your source of truth data source and
expose structured denormalized documents in Elasticsearch <https://www.elastic.co/products/elastic-stack>
_.
Requirements
Python <https://www.python.org>
_ 3.6+Postgres <https://www.postgresql.org>
_ 9.4+Redis <https://redis.io>
_Elasticsearch <https://www.elastic.co/products/elastic-stack>
_ 6.3.1+
Postgres setup
Enable logical decoding <https://www.postgresql.org/docs/current/logicaldecoding.html>
_ in your
Postgres setting.
-
you would also need to set up two parameters in your Postgres config postgresql.conf
wal_level = logical
max_replication_slots = 1
Installation
You can install PGSync from PyPI <https://pypi.org>
_:
$ pip install pgsync
Config
Create a schema for the application named e.g schema.json
Example schema <https://github.com/toluaina/pg-sync/blob/master/examples/airbnb/schema.json>
_
Example spec
.. code-block::
[
{
"database": "[database name]",
"index": "[elasticsearch index]",
"nodes": [
{
"table": "[table A]",
"schema": "[table A schema]",
"columns": [
"column 1 from table A",
"column 2 from table A",
... additional columns
],
"children": [
{
"table": "[table B with relationship to table A]",
"schema": "[table B schema]",
"columns": [
"column 1 from table B",
"column 2 from table B",
... additional columns
],
"relationship": {
"variant": "object",
"type": "one_to_many"
},
...
},
{
... any other additional children
}
]
}
]
}
]
Environment variables
Setup required environment variables for the application
SCHEMA='/path/to/schema.json'
ELASTICSEARCH_HOST=localhost
ELASTICSEARCH_PORT=9200
PG_HOST=localhost
PG_USER=i-am-root # this must be a postgres superuser
PG_PORT=5432
PG_PASSWORD=*****
REDIS_HOST=redis
REDIS_PORT=6379
REDIS_DB=0
REDIS_AUTH=*****
Running
bootstrap the database (one time only) $ bootstrap --config schema.json run pgsync as a daemon $ pgsync --config schema.json --daemon
======= History
1.0.1 (2020-15-01)
- First release on PyPI.
1.0.1 (2020-01-01)
- RC1 release
1.1.0 (2020-04-13)
-
Postgres multi schema support for multi-tennant applications
-
Show resulting Query with verbose mode
-
this release required you to re-bootstrap your database with
- bootstrap -t
- bootstrap
1.1.1 (2020-05-18)
- Fixed authentication with Redis
- Fixed Docker build
1.1.2 (2020-06-11)
- Sync multiple indices in the same schema
- Test for replication or superuser
- Fix PG_NOTIFY error when payload exceeds 8000 bytes limit
1.1.3 (2020-06-14)
- Bug fix when syncing multiple indices in the same schema
1.1.4 (2020-06-15)
- Only create triggers for tables present in schema
1.1.5 (2020-06-16)
- Bug fix when creating multiple triggers in same schema
1.1.6 (2020-07-31)
- Bug fix when tearing down secondary schema
1.1.7 (2020-08-16)
- Fix issue #29: SQLAlchemy err: Neither 'BooleanClauseList' object nor 'Comparator' object has an attribute '_orig'
1.1.8 (2020-08-19)
- Fix issue #30: Traceback AttributeError: id
1.1.9 (2020-08-26)
- Fix issue #33: Unable to set Redis port via environment variable.
1.1.10 (2020-08-29)
- Support Amazon RDS #16
- Optimize database reflection on startup
- Show elapsed time
1.1.11 (2020-09-07)
- Support specify Elasticsearch field data type
1.1.12 (2020-09-08)
- Add support for SSL TCP/IP connection mode
1.1.13 (2020-09-09)
- Show version details with --version argument
- Fixed airbnb examples docker build
1.1.14 (2020-10-07)
- Support Elasticsearch settings for adding mapping and analyzers
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for pgsync-1.1.14-cp38-cp38-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 41cca984b1f23439a0143e8e17bf3ed0bdbc53cab27a35fdb4ed76e3d85e8c10 |
|
MD5 | a0894c37b239bb15d63e450f2f63ed07 |
|
BLAKE2b-256 | ff0eb0446aa95d110fd8e275e016a25f8875051649678ab9dd86422ba2a789ce |
Hashes for pgsync-1.1.14-cp38-cp38-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 91dc0406c26201944258c1866197e9c42c72c76515c2901ad2e54ec5dc924724 |
|
MD5 | 11f38fa0b757244d99084edf2642834e |
|
BLAKE2b-256 | 7a648c6ed47039f5386d9e03a51fb15fa1414b5816103121c4871c8b02906566 |
Hashes for pgsync-1.1.14-cp38-cp38-manylinux1_x86_64.manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d393287b76ad0c1e3a29d2c34056f5d012af99a69ecd386085cd96fdb62eff8d |
|
MD5 | 4970a8c6825ff1e560f61398a9228cdb |
|
BLAKE2b-256 | ceed61ef7804f7037864215165d1446ef08d9e9ce0ffb273cde1718c9ff78aa8 |
Hashes for pgsync-1.1.14-cp37-cp37m-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5ccc12b3feb081522b1aa4b569378668198f5141dca7c5aa9fe42f784cb4f89d |
|
MD5 | fec4ed9445978955daa2d4b6ae6842fb |
|
BLAKE2b-256 | 091827b6003d2ff30c4043058baa9a17f0b083502de1462c110e2cf7da8372ba |
Hashes for pgsync-1.1.14-cp37-cp37m-manylinux2010_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8b032e387ba7e6924ac479fd36a058fde2612d918d63d2052e8c6eb0c56a9289 |
|
MD5 | 1ad3046d0320efaa72f0bf963bdfbcd2 |
|
BLAKE2b-256 | 124b6acccff180e971f28933bfa73ea1a51d34af18eac15b5eeb166904a767c5 |
Hashes for pgsync-1.1.14-cp37-cp37m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c7d7080cd8174141caaf70497f44b7a21f9a00d2cb7900b1fe4dd981cb64b962 |
|
MD5 | f9d63cb2ca150bb2819d555595ee3df3 |
|
BLAKE2b-256 | 5c7ca3b6e9e37f43b2c3219ab58668265e518be24631f8cd8beaedfc93f13ab8 |
Hashes for pgsync-1.1.14-cp37-cp37m-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e1df94f68ce33222cd3b6a64a8ad102656a1b336b6d9e0e712e6da3acfbbc623 |
|
MD5 | e14a83bd663fc967acb1aac9b8e9f030 |
|
BLAKE2b-256 | 22ac6fa427ba576418d56e7138adeac03ec7cc83700d985b7ccb5e1b3cbcd862 |
Hashes for pgsync-1.1.14-cp36-cp36m-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c5cf05699d1eeb4ac8a521d719b168d94d0d36e77b03c7f6cdebb7e9f886f892 |
|
MD5 | df14011e181c47e563463776a69111f7 |
|
BLAKE2b-256 | a0404a9a92326a68f1c518d6fbcab97de305bc40a06093f2306a5762f3d4a0fe |
Hashes for pgsync-1.1.14-cp36-cp36m-manylinux2010_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 28081c4fa8d144482fa09de0fbf27c699069726b61b6b4f9fd4b25a7741e54cf |
|
MD5 | b6de992bd1f7ecb38d1c954a72047279 |
|
BLAKE2b-256 | 63e658597542eeb029748864cb5facc0795c2d2ddf340bbc73cbf7da26247960 |
Hashes for pgsync-1.1.14-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 482c6c4517c06f8cf0d694723af970781bd7c0a6bd0e2b6835f7e48807bc0e74 |
|
MD5 | 64de0ff6249d850712c3fd0056eb38ac |
|
BLAKE2b-256 | db388758e783997373e7b9f9b504dd1fae142bec25567289508b9285fdc57eed |