Pumpwood Miscellaneous

Pumpwood miscellaneous functions and classes.

Pumpwood is a native brasilian tree which has a symbiotic relation with ants (Murabei)

Description

This is a module with miscellaneous functions and classes used in Pumpwood System. These helps from adjusting static file provider (AWS S3, Google Bucket, Azure blob), connect, read and publish messages in RabbitMQ, and other functionalities.

pumpwood_miscellaneous.database

Functions to help connect and build SQLAlchemy query strings.

SQLAlchemyPostGres

Class to substitute SQLAlchemy/Flask database, it integrates with Pumpwood Communication pumpJsonDump that serilize complex objects like shapely BaseGeometry, pandas dataframes and Numpy arrays. It also requery a pre_ping request for each query, invalidating not functional connections.

from flask import Flask, jsonify
from pumpwood_miscellaneous.models import FlaskPumpWoodBaseModel
from pumpwood_communication.serializers import PumpWoodJSONEncoder


db = SQLAlchemyPostGres(model_class=FlaskPumpWoodBaseModel)


def create_app(config_dict, test=False):
    """Create and return app."""
    app = Flask(__name__)
    app.json_encoder = PumpWoodJSONEncoder

    db.app = app
    db.init_app(app)

pumpwood_miscellaneous.models

Class and function to help definition of flask models.

FlaskPumpWoodBaseModel

Add a default a BigInteger field id for all models.

pumpwood_miscellaneous.query

Convert dictionary payload to a SQLAlchemy query. Make similar query to Django filter, exclude and order_by ORM API.

from models import User
from pumpwood_miscellaneous.query import SqlalchemyQueryMisc

query_result = SqlalchemyQueryMisc.sqlalchemy_kward_query(
    object_model=User,
    filter_dict={
        "name__icontains": "Jonh",
        "age__gte": 30
    },
    exclude_dict={
      "company__in": ["Murabei", "Data Random Corp"]
    },
    order_by=["name", "-age"]
).limit(50).all()

pumpwood_miscellaneous.rabbitmq

Manage connections with rabbitmq.

PumpWoodRabbitMQ

Help sending msg to RabbitMQ queues, it is possible to "lazy" set constructor variables using init function. Send function uses PumpWoodJSONEncoder, making it possible to send complex objects as msg payload.

from pumpwood_miscellaneous.rabbitmq import PumpWoodRabbitMQ

not_lazy_worker = PumpWoodRabbitMQ(
  queue="some-queue",
  username="some-user",
  password="with-a-strong-pass"
  host="1.2.3.4"
  port=5672)
not_lazy_worker.send({
    "pandas": pd.DataFrame({
        "process_data": [1, 2, 3, 4],
        "time": pd.to_datatime([
            "2021-01-01", "2021-02-01", "2021-03-01", "2021-04-01"])
      }),
    "ok": "so-ok"})

lazy_worker = PumpWoodRabbitMQ()
lazy_worker.init(
  queue="some-queue",
  username="some-user",
  password="with-a-strong-pass"
  host="1.2.3.4"
  port=5672
)

pumpwood_miscellaneous.storage

Make the interaction with different storage backends using the same API. So far Google bucket and AWS S3 were implemented.

PumpWoodStorage

from pumpwood_miscellaneous.storage import PumpWoodStorage

storage_google = PumpWoodStorage(
  storage_type="google_bucket",
  base_path="base_path/for/files/"
  bucket_name="some_bucket")

file_data = storage_google.read_file("file_path/file.joblib")
self.storage_google.delete_file("file_path/file.joblib")


storage_google = PumpWoodStorage(
  storage_type="aws_s3",
  base_path="base_path/for/files/"
  bucket_name="some_s3")

file_data = storage_google.read_file("file_path/file_2.joblib")
self.storage_google.delete_file("file_path/file_2.joblib")

allowed_extension

Check if file extension is in a list.

allowed_extension(
    filename="file.xlsx", allowed_extensions=["xls", "xlsx", "parquet"],
    exception=Exception)