Postgres to elasticsearch sync
Project description
PostgreSQL to Elasticsearch sync
PGSync <https://pgsync.com>
_ is a middleware for syncing data from Postgres <https://www.postgresql.org>
_ to Elasticsearch <https://www.elastic.co/products/elastic-stack>
.
It allows you to keep Postgres <https://www.postgresql.org>
as your source of truth data source and
expose structured denormalized documents in Elasticsearch <https://www.elastic.co/products/elastic-stack>
_.
Requirements
Python <https://www.python.org>
_ 3.6+Postgres <https://www.postgresql.org>
_ 9.4+Redis <https://redis.io>
_Elasticsearch <https://www.elastic.co/products/elastic-stack>
_ 6.3.1+
Postgres setup
Enable logical decoding <https://www.postgresql.org/docs/current/logicaldecoding.html>
_ in your
Postgres setting.
-
you would also need to set up two parameters in your Postgres config postgresql.conf
wal_level = logical
max_replication_slots = 1
Installation
You can install PGSync from PyPI <https://pypi.org>
_:
$ pip install pgsync
Config
Create a schema for the application named e.g schema.json
Example schema <https://github.com/toluaina/pg-sync/blob/master/examples/airbnb/schema.json>
_
Example spec
.. code-block::
[
{
"database": "[database name]",
"index": "[elasticsearch index]",
"nodes": [
{
"table": "[table A]",
"schema": "[table A schema]",
"columns": [
"column 1 from table A",
"column 2 from table A",
... additional columns
],
"children": [
{
"table": "[table B with relationship to table A]",
"schema": "[table B schema]",
"columns": [
"column 1 from table B",
"column 2 from table B",
... additional columns
],
"relationship": {
"variant": "object",
"type": "one_to_many"
},
...
},
{
... any other additional children
}
]
}
]
}
]
Environment variables
Setup required environment variables for the application
SCHEMA='/path/to/schema.json'
ELASTICSEARCH_HOST=localhost
ELASTICSEARCH_PORT=9200
PG_HOST=localhost
PG_USER=i-am-root # this must be a postgres superuser
PG_PORT=5432
PG_PASSWORD=*****
REDIS_HOST=redis
REDIS_PORT=6379
REDIS_DB=0
REDIS_AUTH=*****
Running
bootstrap the database (one time only) $ bootstrap --config schema.json run pgsync as a daemon $ pgsync --config schema.json --daemon
======= History
1.0.1 (2020-15-01)
- First release on PyPI.
1.0.1 (2020-01-01)
- RC1 release
1.1.0 (2020-04-13)
-
Postgres multi schema support for multi-tennant applications
-
Show resulting Query with verbose mode
-
this release required you to re-bootstrap your database with
- bootstrap -t
- bootstrap
1.1.1 (2020-05-18)
- Fixed authentication with Redis
- Fixed Docker build
1.1.2 (2020-06-11)
- Sync multiple indices in the same schema
- Test for replication or superuser
- Fix PG_NOTIFY error when payload exceeds 8000 bytes limit
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for pgsync-1.1.2-cp38-cp38-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 16c404e28e78f04130d7d04d5efa641e71ce9a00b7d48fa453eeaf310031ec9d |
|
MD5 | 20911e63b32fdb0a5f01fa95bc653dd1 |
|
BLAKE2b-256 | 45f9a8bdab9aec23ff043aa3fde3b8ec8e74908cefc8b20ce869b9fd0fc98bf1 |
Hashes for pgsync-1.1.2-cp38-cp38-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 28cbb6a575219867131cdfd9e7e47a8940c18120ad47a7ccf44beaf7d0260d71 |
|
MD5 | 2401ed422ede0971f2c073bd9ad11887 |
|
BLAKE2b-256 | 46e91e04d9b70659d66a394754ea3788517bb807676e67c08bc26f0bf264b021 |
Hashes for pgsync-1.1.2-cp37-cp37m-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c0412859b5f147f0f4e3e570af3e553a54e8471c4b6c7b323107c6820ddfb213 |
|
MD5 | 8ff94574f49a6e927995b6ea67f247ac |
|
BLAKE2b-256 | f64effce32d31a86c8e319f5f68c06fdfc59841630773a6e2e35ae3da6fd0ee9 |
Hashes for pgsync-1.1.2-cp37-cp37m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 49217860ea10df68d09959e4b9f6fe2cdbb020731231e0d3f0a1ab4934882e33 |
|
MD5 | f62739d083ea080a937efc1e8fe2e9e6 |
|
BLAKE2b-256 | b0b35bf457bcb207b58efab6a1d51b63b600e34176883a608fa0d879197ab60e |
Hashes for pgsync-1.1.2-cp37-cp37m-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ef51047ffb8e203ac6e411baf46e8ba463d47dac71f2eab7ee121a860267a119 |
|
MD5 | 2b3adf5b44e2be0e64fdf4f1e2c15f63 |
|
BLAKE2b-256 | bb1d9bdc9daa57cf4eadac8a1c8fe0e0a3bd4f40d6caba750777c2ccab5db6d1 |
Hashes for pgsync-1.1.2-cp36-cp36m-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 14b04efdf3058b657684cae74b33d296f5a8a23a8e8207f0cb0e243c346ba89f |
|
MD5 | e272f7813dd8f0fadc762bdd0772db34 |
|
BLAKE2b-256 | cd28907ba7925d5e6b0a7c3e0d210c61843b913d23afdd15f265cccc3344ddaa |
Hashes for pgsync-1.1.2-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 00a6972dba89ec62a7abaec52b8818b4b6b4e514c5290e104c8ec50e95c775a8 |
|
MD5 | 8447d86b7802962915d475b341cdedfd |
|
BLAKE2b-256 | fcb4935c6facd7c04c3ff5db531b909fba95f0185733351dd908c329af2e2e69 |