Pumpwood Flask Views

Assists in the creation of Pumpwood views in Flask.

pumpwood-flaskviews .

Pumpwood is a native Brazilian tree which has a symbiotic relation with ants (Murabei)

Objective and motivation

Flask view layer for Pumpwood microservices: CRUD routes, serializers, row-permission filters, file storage, and data-view endpoints (pivot, bulk save).

Why this exists

Pumpwood services share the same REST patterns (list, retrieve, save, actions, options). This package implements those patterns once so each Flask app registers model views instead of reimplementing routing, auth, and serialization.

How it is used

Import view base classes, define a subclass with model_class and serializer, then register routes with register_pumpwood_view. Data lake and estimation workers use PumpWoodDataFlaskView for high-volume bulk insert and pivot queries.

Scope

Covers Flask views, Marshmallow fields, and query helpers. Database models, workers, and front-end clients live in sibling repositories.

Quick start

Install from the Murabei package index (Poetry example):

poetry add pumpwood-flaskviews

Register a view on the Flask app:

from pumpwood_flaskviews.views import PumpWoodFlaskView
from pumpwood_flaskviews.views.register import register_pumpwood_view


class PersonView(PumpWoodFlaskView):
    description = "Person"
    dimensions = {"service": "my-service", "type": "person"}
    db = db
    model_class = Person
    serializer = PersonSerializer
    microservice = microservice
    storage_object = storage_object


register_pumpwood_view(app=app, view=PersonView.as_view())

License

BSD-3-Clause License. See repository metadata and pyproject.toml.

Environment variables

• MICROSERVICE_URL (str): Base URL of the Pumpwood registration and auth microservice. Used by AuthFactory for authorization checks. Prefer this over deprecated AuthFactory.set_server_url.
• PUMPWOOD_FLASKVIEWS__INFO_CACHE_EXPIRE (int): Default 10 minutes (600). Cache TTL for relatively static metadata such as field descriptions and fill options.
• PUMPWOOD_FLASKVIEWS__SERIALIZER_FK_CACHE_EXPIRE (int): Default 5 minutes (300). Cache TTL for foreign-key objects fetched from other microservices on retrieve and list calls. Used by MicroserviceForeignKeyField.
• PUMPWOOD_FLASKVIEWS__AUTHORIZATION_CACHE_EXPIRE (int): Default 1 minute (60). Cache TTL for authorization and row-permission checks.

pumpwood_flaskviews.action

Expose model functions through the API. It is possible to expose normal and classmethods; the first argument for each should respect the convention self and cls respectively.

from pumpwood_flaskviews.action import action


class Person(db.Model):
    """Citizen of a far far way country."""

    name = db.Column(db.String(3), nullable=True, doc=(
        "csv field delimiter when using file input"))
    birth_date = db.DateTime(db.String(3), nullable=True, doc=(
        "csv decimal delimiter when using file input"))

    __tablename__ = 'person'

    @action(info='Marry person to another.')
    def marry(self, person_id: int, contract: str = None) -> bool:
        """Marry person to another one.

        Args:
            person_id (int):
                Id of the person to whom the object should be married.
            contract (str):
                Set the legal contract used.

        Returns:
            bool:
                Returns true if it was possible to process the action.
        """
        ...
        return True

    @classmethod
    @action(info="Send cards to today's birthday.")
    def process_birthday_cards(cls, reference_date: datetime.datetime) -> int:
        """Send birthday cards to every one with birthday today.

        If reference_date if reference_date is passed it will be used instead
        of today as reference.

        Args:
            reference_date (datetime.datetime):
                Reference date to check for birthdays.

        Returns:
            int:
                Number of birthday cards sent.
        """
        ...
        return True

from pumpwood_communication.microservices import PumpWoodMicroService

microservice = PumpWoodMicroService(
    server_url="http://0.0.0.0:8080/",
    username="pumpwood", password="pumpwood")
microservice.login()

# List the actions available for the model Person, it will include information
# about the parameters and also the doc string associated with the function.
person_actions = microservice.list_actions(model_class="Person")

# Return the serialized person object, the parameters used at the action and
# the result
results = microservice.execute_action(
  model_class="Person", pk=3, action="marry",
  parameters={"person_id": 1})

# process_birthday_cards is a classfunction so it is not associated with an
# object
microservice.execute_action(
  model_class="Person", action="process_birthday_cards")

pumpwood_flaskviews.auth

Integrate pumpwood authentication with flask end-points. After setting the Auth host it is possible to call check_authorization, it will pass request headers to auth end-point and check if user is authenticated.

AuthFactory.set_server_url(server_url=config_dict['AUTH_SERVER'])

def flask_end_point():
  AuthFactory.check_authorization()
  return ...

pumpwood_flaskviews.fields

Extend Marshmallow fields for common Pumpwood serializer patterns.

General and audit fields

• GeometryField: Serialize Shapely geometry fields.
• ChoiceField: Serialize sqlalchemy_utils.ChoiceType fields.
• PrimaryKeyField: Serialize primary keys as Base64 strings.
• CreatedByIdField, ModifiedByIdField, CreatedAtField, ModifiedAtField: Audit fields with superuser overwrite support.
• EncryptedField: Encrypt and decrypt sensitive values.
• ReadOnlyChoiceField: Choice field restricted on deserialize.
• RowPermissionField: Integer field for row-permission values.

Related object fields (read-only, fail-soft)

• LocalForeignKeyField: Serialize a local FK using default_query_get.
• LocalRelatedField: Serialize related local objects as a list.
• MicroserviceForeignKeyField: Serialize a remote FK using microservice list_one.
• MicroserviceRelatedField: Serialize related remote objects as a list.
• AutoFillFieldLocal, AutoFillFieldMicroservice: Fill a field from a related object on save. Set apply_user_permission=True to enforce row-permission filters on the lookup (default_query_get locally; request auth_header on microservice retrieve). For microservice autofill with user permission, use a client without credentials so the current request token is forwarded.

These read-only related fields use the fail-soft pattern documented below. They return error metadata inside the field when a related object cannot be retrieved.

Foreign key validation fields (write, fail-hard)

• ValidateForeignKeyFieldLocal: Integer FK field for local models. On deserialize it validates that the referenced row exists and is visible through default_query_get (row-permission filters apply).
• ValidateForeignKeyFieldMicroservice: Integer FK field for remote models. On deserialize it validates access through microservice retrieve, forwarding request auth and base_filter_skip.

Use validation fields on save serializers when the client sends a raw integer foreign key. Use read-only related fields on retrieve/list serializers when the API must embed related object data.

Example for a local model:

from pumpwood_flaskviews.fields import ValidateForeignKeyFieldLocal


class JobSerializer(PumpWoodSerializer):
    person_id = ValidateForeignKeyFieldLocal(
        model_class='models.Person')

Example for a remote model:

from pumpwood_flaskviews.fields import (
    ValidateForeignKeyFieldMicroservice)


class TaskSerializer(PumpWoodSerializer):
    owner_id = ValidateForeignKeyFieldMicroservice(
        model_class='Person',
        not_logged_microservice=microservice)

When validation fails, pumpwood-communication exceptions are raised (for example PumpWoodObjectDoesNotExist, PumpWoodForbidden). The microservice field caches serialized errors in request scope and replays them with raise_from_dict.

Example for autofill with row-permission enforcement:

from pumpwood_flaskviews.fields import (
    AutoFillFieldLocal, AutoFillFieldMicroservice)


class JobSerializer(PumpWoodSerializer):
    row_permission_id = AutoFillFieldLocal(
        model_class='models.DataInput',
        source='datainput_origin_id',
        fill_field='row_permission_id',
        apply_user_permission=True)

    owner_name = AutoFillFieldMicroservice(
        microservice=anonymous_microservice,
        model_class='Person',
        source='owner_id',
        fill_field='name',
        apply_user_permission=True)

pumpwood_flaskviews.serializers

Define a base serializer for pumpwood models which always return at least pk and model_class.

pumpwood_flaskviews.views

Define pumpwood basic views. They have always the same pattern:

PumpWoodFlaskView

Class Attributes

description [str]:

Description of the model; this can be used to display model navigation on the sidebar. During calls to endpoint information, this attribute will be passed to i18n for translation.

dimensions [dict]:

Dictionary of tag/value; this will be registered at the model class route.

icon [str]:

String setting icon name to be display at the frontend.

db [SQLAlchemy Database]:

Connection to database.

model_class [SQLAlchemy Model]:

Model class of Flask SQLAlchemy.

storage_object [PumpWoodStorage]:

Storage object to connect to S3, blog, GCP storage, etc...

microservice [PumpWoodMicroService]:

Microservice used to connect to other microservices in pumpwood. This microservice must be authenticated.

serializer [PumpWoodSerializer]:

Serializer to be used to serialize model objects at endpoints.

list_fields [list(str)]:

List of fields that will be considered as default to be displayed when calling list with default fields.

It is possible to modify the function get_list_fields to make list_fields adapt to the request.

foreign_keys [dict]:

A dictionary to describe the relation of this model with other models in Pumpwood. This is informational data and does not verify if the model actually exists on Pumpwood.

gui_retrieve_fieldset [dict]:

Set a dictionary for rendering frontend, it specify groups of fields to be displayed together. It also permits pass to front the ordering of the fields. Dictionary structure:

# List of the field sets that will be passed to front-end
# permitting setting them in an order
gui_retrieve_fieldset = [
    {
        # Name of the field set
        "name": "Main",
        # Fields that will be returned on this field set,
        # with permits ordering the fields on front end.
        "fields": ['name', 'birth_date']
    }, {
        # It can be passed many relations such as children_set
        "name": "Relations",
        "fields": ['married_to_id', 'children_set']
    }, {
        "name": "Jobs",
        "fields": ['job_set']
    }
]

It is possible to modify function get_gui_retrieve_fieldset to make list_fields to adapt to request.

gui_verbose_field [str]:

Permit pass to front end how this object should be displayed on retrieve view. It is set to substitute using data from the model, example:

# This will help front end to display title of retrieve page
# as '5 | John Doe' substituting values of pk and name.
gui_verbose_field = '{pk} | {name}'

It is possible to modify function get_gui_verbose_field to make list_fields to adapt to request.

gui_readonly [list(str)]:

Set a list of fields that will be considered as read-only on front-end, but can be modified using API. This is useful since some data might be only modified thought API such as jobs start and end time.

# This will set birth_date as read-only on field description,
# although it is still possible to modify it using the API
gui_readonly = ["birth_date"]

It is possible to modify function get_gui_readonly to make list_fields to adapt to request.

End-points

• list (/rest/[model_class]/list/): List objects using query parameters passed as dictionary payload, paginate by 50.
• list_without_pag (/rest/[model_class]/list-without-pag/): Same as list, but return all objects.
• list_one (/rest/[model_class]/list-one/): List one object using list serialize (fewer fields).
• retrieve (/rest/[model_class]/retrieve/[pk]): Return all information from one object.
• object_template (/rest/[model_class]/retrieve/): Return an empty object template.
• retrieve_file (/rest/[model_class]/retrieve-file/[pk]?file-field=[field]): retrieve file which path is saved on file-field argument.
• remove_file_field (/rest/[model_class]/remove-file-field/[pk]?file-field=[field]): delete file which path is saved on file-field argument using streaming.
• retrieve_file_streaming (/rest/[model_class]/retrieve-file-streaming/[pk]?file-field=[field]): retrieve file which path is saved on file-field argument using streaming.
• save_file_streaming (/rest/[model_class]/save-file-streaming/[pk]?file-field=[field]): Save a file in file-field using streaming.
• save (/rest/[model_class]/save/) Save file object passed in payload.
• delete (/rest/[model_class]/delete/[pk]): Remove an object from database.
• delete_many (/rest/[model_class]/delete/): Delete many objects using query dictionary.
• list_actions (/rest/[model_class]/actions/): List actions available for model_class.
• execute_action (/rest/[model_class]/actions/[action name]/[pk]): Run action over object pk.
• search_options (GET /rest/[model_class]/options/): Retrieve information for list fields and filters.
• fill_options (POST /rest/[model_class]/options/): Pass an incomplete object as payload and receive field validation and updated choice possibilities. Callable serializer and SQLAlchemy defaults (list, dict, now) are normalized to JSON-safe values in responses.
• aggregate (/rest/[model_class]/aggregate/): Group and aggregate rows with pandas. Accepts show_deleted to include soft-deleted rows when the model has a deleted column.

PumpWoodDataFlaskView

• Same as PumpWoodFlaskView...
• pivot (/rest/[model_class]/pivot/): Retrieve data using query dict, but instead of using serializers use pandas data frame and parse result with to_dict. It is possible to pivot data using the columns.
• bulk_save (/rest/[model_class]/bulk-save/): Bulk save data on database. Requires expected_cols_bulk_save on the view (see below).

Bulk save configuration

Set expected_cols_bulk_save on PumpWoodDataFlaskView to declare columns sent in the payload and columns filled server-side before bulk_insert_mappings. Plain strings are pass-through columns from the client. Use typed entries from pumpwood_communication.type:

• BulkSaveLocalAutoFillField: Fill a column from a local related model. object_fk_column must be a column present in the payload (for example datainput_origin_id), not the target field being filled.
• BulkSaveMicroserviceAutoFillField: Same pattern using a remote model through the view microservice.
• BulkSaveDefaultField: Apply a default when the column is missing or null.

Example:

from pumpwood_communication.type import (
    BulkSaveDefaultField, BulkSaveLocalAutoFillField)


class DataBaseVariableView(PumpWoodDataFlaskView):
    expected_cols_bulk_save = [
        BulkSaveLocalAutoFillField(
            field='row_permission_id',
            object_fk_column='datainput_origin_id',
            fill_model_class=DataInputDatabaseVariable,
            fill_col='row_permission_id'),
        BulkSaveDefaultField(field='deleted', default=False),
        'time', 'modeling_unit_id', 'value', 'datainput_origin_id',
    ]

Misconfigured autofill (missing object_fk_column on the payload) raises PumpWoodDataLoadingException with column hints in the payload.

PumpWoodDimensionsFlaskView

• Same as PumpWoodFlaskView...
• list_dimensions (/rest/[model_class]/list-dimensions/): List available dimensions in objects resulting from the query.
• list_dimension_values (/rest/[model_class]/list-dimension-values/): List values associated with dimensions in objects resulting from the query.

Example for defining a PumpWoodFlaskView:

from pumpwood_flaskviews.views import PumpWoodFlaskView
from models import Person
from serializers import PersonSerializer
from singletons import storage_object, microservice


class PersonView(PumpWoodFlaskView):
    description = "Person"

    # Used when registering end-point on routes
    dimensions = {
        "service": "test-service",
        "type": "human",
    }
    icon = None

    db = db
    model_class = Person
    storage_object = storage_object
    microservice = microservice
    serializer = PersonSerializer

    foreign_keys = {
        'married_to_id': {
          'model_class': 'Person', 'many': False,
          'display_field': 'name'},
        'children_set': {
            'model_class': 'Children', 'many': True,
            'foreign_key': 'parent_id', 'read_only': True},
        'job_set': {
            'model_class': 'Job', 'many': True,
            'foreign_key': 'person_id', 'read_only': False},
    }

    #######
    # Gui #
    list_fields = [
        'pk', 'model_class', 'name', 'birthday', 'married_to_id']
    gui_retrieve_fieldset = [
        {
            "name": "Main",
            "fields": ['name', 'birth_date']
        }, {
            "name": "Relations",
            "fields": ['married_to_id', 'children_set']
        }, {
            "name": "Jobs",
            "fields": ['job_set']
        }
    ]
    gui_verbose_field = '{pk} | {name}'
    gui_readonly = ["birth_date"]
    #######


### Fail-Soft Serialization
Pumpwood Flask Views implements a "fail-soft" pattern for read-only
related fields (`MicroserviceForeignKeyField`, `LocalRelatedField`,
etc.). If a related object cannot be retrieved (for example 404 Not
Found or 403 Forbidden), the serializer does not crash. Instead, it
returns a standardized error object within the field:

```json
{
  "model_class": "Person",
  "__error__": "PumpWoodObjectDoesNotExist",
  "__display_field__": "Object not found",
  "payload": {
    "pk": 123
  }
}

This ensures that the main object can still be serialized and returned to the client even if some secondary relations are temporarily unavailable. Sensitive user information is stripped from these error returns and is only available in the server-side logs.

Validation fields (ValidateForeignKeyFieldLocal, ValidateForeignKeyFieldMicroservice) follow the opposite pattern on save: invalid or inaccessible foreign keys raise exceptions instead of returning embedded error metadata.