Postgres to elasticsearch sync
Project description
PostgreSQL to Elasticsearch sync
PGSync <https://pgsync.com>
_ is a middleware for syncing data from Postgres <https://www.postgresql.org>
_ to Elasticsearch <https://www.elastic.co/products/elastic-stack>
.
It allows you to keep Postgres <https://www.postgresql.org>
as your source of truth data source and
expose structured denormalized documents in Elasticsearch <https://www.elastic.co/products/elastic-stack>
_.
Requirements
Python <https://www.python.org>
_ 3.6+Postgres <https://www.postgresql.org>
_ 9.4+Redis <https://redis.io>
_Elasticsearch <https://www.elastic.co/products/elastic-stack>
_ 6.3.1+
Postgres setup
Enable logical decoding <https://www.postgresql.org/docs/current/logicaldecoding.html>
_ in your
Postgres setting.
-
you would also need to set up two parameters in your Postgres config postgresql.conf
wal_level = logical
max_replication_slots = 1
Installation
You can install PGSync from PyPI <https://pypi.org>
_:
$ pip install pgsync
Config
Create a schema for the application named e.g schema.json
Example schema <https://github.com/toluaina/pg-sync/blob/master/examples/airbnb/schema.json>
_
Example spec
.. code-block::
[
{
"database": "[database name]",
"index": "[elasticsearch index]",
"nodes": [
{
"table": "[table A]",
"schema": "[table A schema]",
"columns": [
"column 1 from table A",
"column 2 from table A",
... additional columns
],
"children": [
{
"table": "[table B with relationship to table A]",
"schema": "[table B schema]",
"columns": [
"column 1 from table B",
"column 2 from table B",
... additional columns
],
"relationship": {
"variant": "object",
"type": "one_to_many"
},
...
},
{
... any other additional children
}
]
}
]
}
]
Environment variables
Setup required environment variables for the application
SCHEMA='/path/to/schema.json'
ELASTICSEARCH_HOST=localhost
ELASTICSEARCH_PORT=9200
PG_HOST=localhost
PG_USER=i-am-root # this must be a postgres superuser
PG_PORT=5432
PG_PASSWORD=*****
REDIS_HOST=redis
REDIS_PORT=6379
REDIS_DB=0
REDIS_AUTH=*****
Running
bootstrap the database (one time only) $ bootstrap --config schema.json run pgsync as a daemon $ pgsync --config schema.json --daemon
======= History
1.0.1 (2020-15-01)
- First release on PyPI.
1.0.1 (2020-01-01)
- RC1 release
1.1.0 (2020-04-13)
-
Postgres multi schema support for multi-tennant applications
-
Show resulting Query with verbose mode
-
this release required you to re-bootstrap your database with
- bootstrap -t
- bootstrap
1.1.1 (2020-05-18)
- Fixed authentication with Redis
- Fixed Docker build
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for pgsync-1.1.1-cp38-cp38-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4850a6043b70cbadf083017a8807a6310a8dae13fa04f4ddd0a1702e39db1ce4 |
|
MD5 | 0553d1343766962dbeae5e1a535c08a6 |
|
BLAKE2b-256 | 406d7e79b70f2b7070ff0ca8a3018ab72e0ebf94e96d6e44e64159c537c63d19 |
Hashes for pgsync-1.1.1-cp38-cp38-manylinux2010_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 275cddc955dfa708f900d127c919d79f44df8b7b1bf4bbb316888e55fff342bb |
|
MD5 | 36fe173a56bdf49824d4e8254f78bcbc |
|
BLAKE2b-256 | 9aea49b29036c4dac720062b1cff8591457984d92274585cfcbef488be399b30 |
Hashes for pgsync-1.1.1-cp38-cp38-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e3159a31f47a941a929e93494ca92ee5b13df0ccca58923395063f18f3a72810 |
|
MD5 | 28cbc689968cd047d7632e7ce5f1ca00 |
|
BLAKE2b-256 | dbcc10173a2b5c1d97c8ca5c2b293ac90b3968d9a7eed32523f8e30e640a3c60 |
Hashes for pgsync-1.1.1-cp37-cp37m-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ceaced05d874dfb271e4dc6a7c8389193e71717cec6764d6ba6c4c9cab969faa |
|
MD5 | 6db2380f12dbc7b3e029b46d741733bd |
|
BLAKE2b-256 | ffcb0af5f6fda2d996cd04b9cb1a064e1e251cc6ff188be9c6897269c955ec12 |
Hashes for pgsync-1.1.1-cp37-cp37m-manylinux2010_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 014c62c60b4283936cbc5fa04ad06a364deb5bee56d2b43b6ed5923b3e351db3 |
|
MD5 | 863ee9283c77fb9fb7e46cc66d0cdfa3 |
|
BLAKE2b-256 | 2286c89c6d8a93e7764b97e038c00f5ae7b76a2b9d82f8c268e209c80249a2b5 |
Hashes for pgsync-1.1.1-cp37-cp37m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 39a63fc79b04dad97ba9a3ca0af6f3ba397f5d530a8cd063852cbd79aefb3345 |
|
MD5 | b4b4139f8c6fc451048c9aea6c97adaa |
|
BLAKE2b-256 | 9e99931ab3765852ecdf79317ec568369c0301d924db8d4048f15d07e796873b |
Hashes for pgsync-1.1.1-cp37-cp37m-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3e540b8a19a648f489e1e1535488ff088dea9aad155588c4b5168a6d27e6fd40 |
|
MD5 | 519e71c53aeea4de5b38464fef2995d9 |
|
BLAKE2b-256 | 9b36b5931706f2dc0b1c022191effdbdcd8c85dacf30d279be0ad8002ca8bd86 |
Hashes for pgsync-1.1.1-cp36-cp36m-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 86e6ae16778debbfd13f8458ab7b0ceefee2eb10929983545abd1b55084e9d60 |
|
MD5 | 20382996c222a4bc558f931f620cb1df |
|
BLAKE2b-256 | 63a7dc1872a33fa0fd0c81f1c1a5e20f107e70fa2759d83f58e836ef3b2a17e8 |
Hashes for pgsync-1.1.1-cp36-cp36m-manylinux2010_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d63525eccab0ceeff27ba1dc2dfc19785d777b5d3e0f84eb61f6fd470d9442fc |
|
MD5 | 16e90d0da1313a0e378c62d930b09ede |
|
BLAKE2b-256 | ceb5ffc0fd6fcd550c97deb53f264620a5bd74cb53819c4cc0e35e4654b90d8a |
Hashes for pgsync-1.1.1-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 93350bc41ed76c3bb1e1c991b50f68903358f371d003990961153ccf29c9dd32 |
|
MD5 | ed36542ae4fc43bb8c8e0fcef93a10da |
|
BLAKE2b-256 | 6d9c5c0ba674dd2fbbb3390db8168759e015cc9dada472790bd3f71b6d63e64c |