pumpwood-deploy-complex-datalake

Satellite deploy package for the Pumpwood Complex Datalake microservice on Kubernetes. It generates manifests for the API app, the complex-database dataloader worker, and secrets, then hands them to pumpwood-deploy for apply.

Developed by Murabei Data Science. BSD-3-Clause.

Pumpwood is a native Brazilian tree with a symbiotic relation to ants (Murabei)

---

Objective and motivation

This package deploys Complex Datalake as a satellite of pumpwood-deploy: one Python class renders Kubernetes YAML; the core orchestrator applies it.

Why this exists

Complex Datalake used to live inside the monolithic pumpwood-deploy tree. Splitting it keeps image pins, worker count, and database credentials next to this service without shipping unused annotation workers.

How it is used

A platform deploy script constructs PumpWoodComplexDatalakeMicroservice, registers it with DeployPumpWood.add_microservice, then calls create_deploy_files / deploy_microservices. Operators pass image tags (app_version, datalake_dataloader_version); the PyPI package version is in VERSION (see changelog.md).

Scope

In scope: Secret, app Deployment + Service, and the complex-database dataloader worker.

Out of scope: Postgres/PgBouncer, RabbitMQ, storage ConfigMap, auth, Kong, and annotation workers. Those come from pumpwood-deploy, pumpwood-deploy-auth, or a follow-up satellite.

---

What it deploys

Manifest	Kubernetes resources
pumpwood_complex__datalake__secrets	Secret pumpwood-complex-datalake
pumpwood_complex__datalake__deploy	Deployment + Service pumpwood-complex-datalake-app
pumpwood_complex__datalake_dataloader_worker	Deployment pumpwood-complex-datalake-worker-complex-database

The app serves HTTP APIs. The worker consumes RabbitMQ and uploads rows in parallel chunks to Postgres.

flowchart LR
    subgraph pkg [pumpwood-deploy-complex-datalake]
        A[PumpWoodComplexDatalakeMicroservice]
    end
    subgraph core [pumpwood-deploy]
        B[DeployPumpWood]
    end
    subgraph cluster [Cluster]
        S[pumpwood-complex-datalake Secret]
        APP[pumpwood-complex-datalake-app]
        W[complex-database worker]
        RMQ[rabbitmq-main]
    end
    A --> B
    B --> S
    B --> APP
    B --> W
    RMQ --> APP
    RMQ --> W

---

Prerequisites

This package does not stand alone. Before pods can start, the cluster must already provide:

Resource	Provided by
storage ConfigMap	StandardMicroservices in pumpwood-deploy
general-secrets	StandardMicroservices
rabbitmq-main-secrets	StandardMicroservices
Storage keys (GCP / Azure / AWS)	DeployPumpWood storage config
Postgres for complex datalake	PostgresDatabase + PGBouncerDatabase
Auth (typical)	pumpwood-deploy-auth

Storage bucket name and type are read from the cluster storage ConfigMap. They are not constructor arguments of PumpWoodComplexDatalakeMicroservice.

For local or CI databases, deploy Postgres/PgBouncer from pumpwood-deploy. Embedded test-database parameters were removed from this satellite.

---

Installation

pip install pumpwood-deploy-complex-datalake

Requires pumpwood-deploy and Python 3.6+.

---

Quick start

import os
import simplejson as json
from dotenv import load_dotenv
from pumpwood_deploy.deploy import DeployPumpWood
from pumpwood_deploy.microservices.postgres.deploy import (
    PostgresDatabase, PGBouncerDatabase)
from pumpwood_deploy_complex_datalake import (
    PumpWoodComplexDatalakeMicroservice)

with open("secrets/production.json", "r") as file:
    secrets = json.loads(file.read())
load_dotenv()

deploy = DeployPumpWood(
    model_user_password=secrets["microservices--model"],
    rabbitmq_secret=secrets["rabbitmq_secret"],
    hash_salt=secrets["hash_salt"],
    storage_type="aws_s3",
    storage_deploy_args={
        "storage_bucket_name": "my-pumpwood-bucket",
        "access_key_id": secrets["aws_access_key_id"],
        "secret_access_key": secrets["aws_secret_access_key"],
    },
    k8_provider="aws",
    k8_deploy_args={
        "region": "us-east-1",
        "cluster_name": "my-cluster",
    },
    k8_namespace="pumpwood",
)

deploy.add_microservice(
    PostgresDatabase(
        db_username="pumpwood",
        db_password=secrets["postgres_password"],
        name="postgres-main",
        disk_name="postgres-disk",
        disk_size="150Gi",
    ))

deploy.add_microservice(
    PGBouncerDatabase(
        name="pgbouncer-pumpwood-complex-datalake",
        postgres_database="pumpwood_complex_datalake",
        postgres_secret="postgres-main",
        postgres_host="postgres-main",
    ))

deploy.add_microservice(
    PumpWoodComplexDatalakeMicroservice(
        app_version=os.getenv("PUMPWOOD_COMPLEX_DATALAKE_APP"),
        datalake_dataloader_version=os.getenv(
            "PUMPWOOD_COMPLEX_DATALAKE_WORKER"),
        repository="my-registry.example.com",
        db_host="pgbouncer-pumpwood-complex-datalake",
        db_database="pumpwood_complex_datalake",
        db_password=secrets["postgres_password"],
        microservice_password=secrets[
            "microservice--complex-datalake"],
        app_replicas=1,
        app_debug="FALSE",
        datalake_dataloader_replicas=1,
        datalake_dataloader_n_parallel=4,
    ))

deploy.create_deploy_files()
deploy.deploy_microservices()

Environment variables

These are container image tags, not the PyPI package version:

PUMPWOOD_COMPLEX_DATALAKE_APP=2.0.69
PUMPWOOD_COMPLEX_DATALAKE_WORKER=2.0.25

If the rendered manifest matches the cluster, kubectl apply is a no-op — safe for rolling image updates.

---

Configuration reference

Constructor: PumpWoodComplexDatalakeMicroservice in src/pumpwood_deploy_complex_datalake/deploy.py.

Required

Parameter	Description
app_version	Image tag for pumpwood-complex-datalake-app
datalake_dataloader_version	Image tag for pumpwood-complex-datalake-worker-complex-database

Database and registry

Parameter	Default	Description
db_host	pgbouncer-pumpwood-complex-datalake	Postgres host (PgBouncer in prod)
db_port	5432	Postgres port
db_database	pumpwood	Database name
db_username	pumpwood	Database user
db_password	pumpwood	Database password
microservice_password	microservice--complex-datalake	Service user password
repository	gcr.io/repositorio-geral-170012	Docker registry for app and worker

Application

Parameter	Default	Description
app_replicas	1	Number of app pods
app_debug	FALSE	Debug flag (TRUE / FALSE)
app_workers	10	Granian workers (GRANIAN_WORKERS)
app_timeout	300	Request timeout in seconds
app_limits_memory	60Gi	Memory limit
app_limits_cpu	12000m	CPU limit
app_requests_memory	20Mi	Memory request
app_requests_cpu	1m	CPU request

Complex-database dataloader worker

Parameter	Default	Description
datalake_dataloader_replicas	1	Worker pod count
datalake_dataloader_debug	FALSE	Worker debug flag
datalake_dataloader_n_parallel	4	Parallel upload requests
datalake_dataloader_chunk_size	1000	Rows per parallel batch
datalake_dataloader_query_limit	1000000	Max rows per upload cycle
markitdown_ocr_enabled	FALSE	LLM Vision OCR for scanned PDFs (MARKITDOWN_OCR_ENABLED)
markitdown_llm_model	gpt-4o	OCR model (MARKITDOWN_LLM_MODEL)
openai_api_key	(empty)	Stored in Secret; mounted as OPENAI_API_KEY
datalake_dataloader_limits_memory	60Gi	Worker memory limit
datalake_dataloader_limits_cpu	12000m	Worker CPU limit
datalake_dataloader_requests_memory	20Mi	Worker memory request
datalake_dataloader_requests_cpu	1m	Worker CPU request

OCR is off by default. Set markitdown_ocr_enabled="TRUE" and pass openai_api_key to extract text from scanned PDFs (worker image 2.0.23+).

---

Health check

The app Deployment readiness probe is:

GET /health-check/pumpwood-complex-datalake-app/  (port 5000)

Use the same path for ingress and load balancer checks.

---

Migration from monolithic pumpwood-deploy

Older scripts imported from the core package:

# Before
from pumpwood_deploy.microservices.pumpwood_complex_datalake.deploy import (
    PumpWoodComplexDatalakeMicroservice)

# After
from pumpwood_deploy_complex_datalake import (
    PumpWoodComplexDatalakeMicroservice)

Constructor changes in this satellite:

• Worker image tag is datalake_dataloader_version (not worker_datalake_dataloader_version)
• bucket_name — use the cluster storage ConfigMap
• test_db_* — use PostgresDatabase / PGBouncerDatabase
• worker_simple_dataloader_version and worker_complex_dataloader_version — annotation workers are not in this package

The monolith rendered five manifests (three workers). This satellite renders three. Add annotation workers in another package if needed.

---

Related packages

Package	Role
pumpwood-deploy	Orchestrator, Kong, RabbitMQ, Postgres
pumpwood-deploy-datalake	Standard datalake microservice
pumpwood-deploy-auth	Authorization microservice

Platform docs: Murabei Open Source — pumpwood-deploy.

Release history: changelog.md.

---

Development

pip install -e ../pumpwood-deploy
pip install -e .

PYTHONPATH="src:../pumpwood-deploy/src" \
  python3 -m unittest discover \
  -s src/pumpwood_deploy_complex_datalake/tests -p "test_*.py" -v

ruff check src/

---

License

BSD-3-Clause — see LICENSE.