Crea una conexión de una base de datos postgres lo gurda en cache y en elasticsearch
Project description
Redis ElasticSearch
version 1.0.1 (Stable release)
Summary
This Python code uses various libraries to connect to and manipulate data in different databases. Specifically, it imports the ast, psycopg2, functools, redis, threading, time, json, and pandas libraries, as well as the Elasticsearch and bulk modules of the elasticsearch library.
The Settings_redis_elastic class uses the connection parameters to a Redis server and an Elasticsearch server to create methods that allow you to get data from a PostgreSQL database and send it to an Elasticsearch index through Redis.
In addition, the class uses a decorator to cache the results of a Redis function and another method to update the cache periodically. It also defines a method to convert data stored in Redis to a Pandas DataFrame object.
Finally, the code runs an infinite loop to send the data from Redis to Elasticsearch with a certain time interval. This process is done in a separate thread, while another thread takes care of updating the cache periodically.
recommendations
Use docker for testing, You can deploy the official containers
Example by execution
from Warehouse.redis_elastic import Settings_redis_elastic
postgres = {
"host": "127.0.0.1",
"port": "5432",
"user": "etl",
"password": "123",
"database": "customer"
}
query = """Select rp.id,rp.name,rp.code from res_partner rp"""
my_object = Settings_redis_elastic("localhost", "6379", "http://localhost:9200")
my_object.fusion(query, "_test", "test", 10, 20, postgres)
Documentation
- The first step is to create an object, and connect redis and elasticsearch
my_object = Settings_redis_elastic("redis_ip", "port_redis", "http://ip_elasticserch:port")
- Variables description
-
query : It is the query that we make from the database, in the first version it works with postgres
-
redis_name : Is the name of the collection that Redis saves
-
elasticsearch_name : It is the way to feed the Elasticsearch API
-
time_redis : The time the SQL query is saved to Redis
-
time_elastic : It is the timeout from Redis to elasticsearch, Recommendation: It should be bigger than <<time_redis>>
-
database : It is the connection to the database, it must be a dictionary
Position in the method fusion
my_object.fusion(query, redis_name , elasticsearch_name, time_redis, time_elastic, database)
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.