Postgres to elasticsearch sync
Project description
PostgreSQL to Elasticsearch sync
PGSync <https://pgsync.com>
__ is a middleware for syncing data from Postgres <https://www.postgresql.org>
__ to Elasticsearch <https://www.elastic.co/products/elastic-stack>
.
It allows you to keep Postgres <https://www.postgresql.org>
as your source of truth data source and
expose structured denormalized documents in Elasticsearch <https://www.elastic.co/products/elastic-stack>
__.
Requirements
Python <https://www.python.org>
__ 3.6+Postgres <https://www.postgresql.org>
__ 9.4+Redis <https://redis.io>
__Elasticsearch <https://www.elastic.co/products/elastic-stack>
__ 6.3.1+
Postgres setup
Enable logical decoding <https://www.postgresql.org/docs/current/logicaldecoding.html>
__ in your
Postgres setting.
-
you would also need to set up two parameters in your Postgres config postgresql.conf
wal_level = logical
max_replication_slots = 1
Installation
You can install PGSync from PyPI <https://pypi.org>
__:
$ pip install pgsync
Config
Create a schema for the application named e.g schema.json
Example schema <https://github.com/toluaina/pg-sync/blob/master/examples/airbnb/schema.json>
__
Example spec
.. code-block::
[
{
"database": "[database name]",
"index": "[elasticsearch index]",
"nodes": [
{
"table": "[table A]",
"schema": "[table A schema]",
"columns": [
"column 1 from table A",
"column 2 from table A",
... additional columns
],
"children": [
{
"table": "[table B with relationship to table A]",
"schema": "[table B schema]",
"columns": [
"column 1 from table B",
"column 2 from table B",
... additional columns
],
"relationship": {
"variant": "object",
"type": "one_to_many"
},
...
},
{
... any other additional children
}
]
}
]
}
]
Environment variables
Setup required environment variables for the application
SCHEMA='/path/to/schema.json'
ELASTICSEARCH_HOST=localhost
ELASTICSEARCH_PORT=9200
PG_HOST=localhost
PG_USER=i-am-root # this must be a postgres superuser
PG_PORT=5432
PG_PASSWORD=*****
REDIS_HOST=redis
REDIS_PORT=6379
REDIS_DB=0
REDIS_AUTH=*****
Running
bootstrap the database (one time only) $ bootstrap --config schema.json run pgsync as a daemon $ pgsync --config schema.json --daemon
======= History
1.0.1 (2020-15-01)
- First release on PyPI.
1.0.1 (2020-01-01)
- RC1 release
1.1.0 (2020-04-13)
-
Postgres multi schema support for multi-tennant applications
-
Show resulting Query with verbose mode
-
this release required you to re-bootstrap your database with
- bootstrap -t
- bootstrap
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for pgsync-1.1.0-cp37-cp37m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5938e368ea0fc14d336cc7a4034fb20101a4000954f5d95e3c0123bfa77cf316 |
|
MD5 | bee22868ef426af664cf9557946de245 |
|
BLAKE2b-256 | 6e97ceda295639e08f0f1ed80735b5c4025ab529b0e50fd08785a0b6d85fea42 |
Hashes for pgsync-1.1.0-cp37-cp37m-macosx_10_15_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 77832ff634e18a8b97b55a485e0585a127a5c60bee944f73aca4e6dca7830424 |
|
MD5 | abd925303d4e05f3f1384fa65b984538 |
|
BLAKE2b-256 | e9c06f56445ebffe4f8bbce63448594583295f87947124ba38cee9ddd411af3c |
Hashes for pgsync-1.1.0-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f6ff1430dd08d62dc345644ff6cdea3f25a2bab04a6d5701c3f4267644d16fc1 |
|
MD5 | 5d60d79ae813612536ada3c21a05305e |
|
BLAKE2b-256 | 1f75685d5e888b7f955f257c728dd4044c8b6af0125fbc09d621617163bd9eeb |