Postgres to Elasticsearch/OpenSearch sync
Project description
PostgreSQL to Elasticsearch/OpenSearch sync
- PGSync is a middleware for syncing data from Postgres to Elasticsearch/OpenSearch or OpenSearch.
- It allows you to keep Postgres as your source of truth data source and expose structured denormalized documents in Elasticsearch/OpenSearch.
Requirements
- Python 3.8+
- Postgres 9.6+
- Redis 3.1.0
- Elasticsearch 6.3.1+ or OpenSearch 1.3.7+
- SQlAlchemy 1.3.4+
Postgres setup
Enable logical decoding in your Postgres setting.
-
You also need to set up two parameters in your Postgres config postgresql.conf
wal_level = logical
max_replication_slots = 1
Installation
You can install PGSync from PyPI:
$ pip install pgsync
Config
Create a schema for the application named e.g schema.json
Example spec
.. code-block::
[
{
"database": "[database name]",
"index": "[Elasticsearch or OpenSearch index]",
"nodes": {
"table": "[table A]",
"schema": "[table A schema]",
"columns": [
"column 1 from table A",
"column 2 from table A",
... additional columns
],
"children": [
{
"table": "[table B with relationship to table A]",
"schema": "[table B schema]",
"columns": [
"column 1 from table B",
"column 2 from table B",
... additional columns
],
"relationship": {
"variant": "object",
"type": "one_to_many"
},
...
},
{
... additional children
}
]
}
}
]
Environment variables
Setup environment variables required for the application
SCHEMA='/path/to/schema.json'
ELASTICSEARCH_HOST=localhost
ELASTICSEARCH_PORT=9200
PG_HOST=localhost
PG_USER=i-am-root # this must be a postgres superuser or replication user
PG_PORT=5432
PG_PASSWORD=*****
REDIS_HOST=redis
REDIS_PORT=6379
REDIS_DB=0
REDIS_AUTH=*****
Running
Bootstrap the database (one time only)
- $ bootstrap --config schema.json
Run pgsync as a daemon
- $ pgsync --config schema.json --daemon
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pgsync-3.2.1.tar.gz
(114.3 kB
view hashes)
Built Distribution
pgsync-3.2.1-py3-none-any.whl
(61.1 kB
view hashes)