pumpwood-deploy-crawler-bigdata

Satellite deploy package for the Pumpwood Crawler BigData microservice on Kubernetes. It generates manifests for the API application, person and company workers, and crawler secrets — then hands them to pumpwood-deploy for apply.

Developed by Murabei Data Science. BSD-3-Clause.

Pumpwood is a native Brazilian tree with a symbiotic relation to ants (Murabei)

---

Objective and motivation

This package deploys the Crawler BigData app and its two RabbitMQ workers onto a Pumpwood Kubernetes cluster.

Why this exists

Queue CRUD lives in the API image; BigData fetch and ETL live in the worker image. The cluster needs one app Service and two worker Deployments that share the worker image and differ only by queue.

How it is used

A cluster deploy script imports PumpWoodCrawlerBigdataMicroservice and calls DeployPumpWood.add_microservice. PgBouncer is added in that same script via PGBouncerDatabase from pumpwood-deploy.

Scope

In scope: Secret, app Deployment + Service, person worker, company worker.

Out of scope: Postgres, PgBouncer, Kong, Auth, Datalake, Graph Datalake, and the database itself.

---

What it deploys

Manifest	Kubernetes resources
pumpwood_crawler_bigdata__secrets	Secret pumpwood-crawler-bigdata
pumpwood_crawler_bigdata__deploy	Deployment + Service pumpwood-crawler-bigdata-app
pumpwood_crawler_bigdata_person__worker	Deployment pumpwood-crawler-bigdata-worker-person
pumpwood_crawler_bigdata_company__worker	Deployment pumpwood-crawler-bigdata-worker-company

flowchart LR
    subgraph pkg [pumpwood-deploy-crawler-bigdata]
        A[PumpWoodCrawlerBigdataMicroservice]
    end
    subgraph core [pumpwood-deploy]
        B[DeployPumpWood]
    end
    subgraph cluster [Cluster]
        S[crawler Secret]
        APP[crawler-bigdata-app]
        WP[person worker]
        WC[company worker]
        RMQ[rabbitmq-main]
    end
    A --> B
    B --> S
    B --> APP
    B --> WP
    B --> WC
    APP --> RMQ
    RMQ --> WP
    RMQ --> WC

---

Prerequisites

This package does not stand alone. Before crawler pods can start, the cluster must already provide:

Resource	Provided by
storage ConfigMap	StandardMicroservices in pumpwood-deploy
general-secrets	StandardMicroservices
rabbitmq-main-secrets	StandardMicroservices
Storage keys (GCP / Azure / AWS)	DeployPumpWood storage config
Postgres for crawler	PGBouncerDatabase in the cluster script
Auth	pumpwood-deploy-auth
Datalake	pumpwood-deploy-datalake
Graph Datalake	graph-datalake satellite

Storage bucket name and type are read from the cluster storage ConfigMap — they are not passed to PumpWoodCrawlerBigdataMicroservice.

The database named in db_database must already exist on the external Postgres. PGBouncerDatabase only pools connections.

---

Installation

pip install pumpwood-deploy-crawler-bigdata

Requires pumpwood-deploy.

---

Quick start

import os
import simplejson as json
from dotenv import load_dotenv
from pumpwood_deploy.deploy import DeployPumpWood
from pumpwood_deploy.microservices.postgres.deploy import (
    PGBouncerDatabase)
from pumpwood_deploy_crawler_bigdata import (
    PumpWoodCrawlerBigdataMicroservice)

with open("secrets/production.json", "r") as file:
    secrets = json.loads(file.read())
load_dotenv()

deploy.add_microservice(
    PGBouncerDatabase(
        name="pgbouncer-pumpwood-crawler-bigdata",
        postgres_database="pumpwood_crawler_bigdata",
        postgres_secret="postgres-general",
        postgres_host=POSTGRES_HOST,
    ))

deploy.add_microservice(
    PumpWoodCrawlerBigdataMicroservice(
        app_version=os.getenv("PUMPWOOD_CRAWLER_BIGDATA_APP"),
        worker_version=os.getenv(
            "PUMPWOOD_CRAWLER_BIGDATA_WORKER"),
        repository="my-registry.example.com",
        db_host="pgbouncer-pumpwood-crawler-bigdata",
        db_database="pumpwood_crawler_bigdata",
        db_password=secrets["postgres_password"],
        microservice_password=secrets[
            "microservice--crawler-bigdata"],
        bigdata_access_token=secrets["bigdata_access_token"],
        app_replicas=1,
        app_debug="FALSE",
        worker_person_replicas=1,
        worker_company_replicas=1,
    ))

deploy.create_deploy_files()
deploy.deploy_microservices()

Environment variables

PUMPWOOD_CRAWLER_BIGDATA_APP=0.0.5
PUMPWOOD_CRAWLER_BIGDATA_WORKER=0.0.17

If the rendered manifest matches what is already on the cluster, kubectl apply produces no changes — safe for rolling image updates.

---

Configuration reference

Required

Parameter	Description
app_version	Image tag for pumpwood-crawler-bigdata-app
worker_version	Image tag for pumpwood-crawler-bigdata-worker

Workers need a non-empty bigdata_access_token to call BigData Corp.

Database

Parameter	Default	Description
db_host	postgres-pumpwood-crawler-bigdata	Postgres host (PgBouncer in prod)
db_port	5432	Postgres port
db_database	pumpwood	Database name
db_username	pumpwood	Database user
db_password	pumpwood	Database password
microservice_password	microservice--crawler-bigdata	Service user password
repository	GCR default	Docker registry for app and worker

Application

Parameter	Default	Description
app_replicas	1	Number of app pods
app_debug	FALSE	Debug flag
app_workers	10	Granian workers (GRANIAN_WORKERS)
app_timeout	300	Request timeout (seconds)
app_limits_memory	60Gi	Memory limit
app_limits_cpu	12000m	CPU limit
app_requests_memory	20Mi	Memory request
app_requests_cpu	1m	CPU request

Workers

Person queue: CrawlerBigDataPersonQueue__on_queue.
Company queue: CrawlerBigDataCompanyQueue__on_queue.

Parameter	Default	Description
worker_person_replicas	1	Person worker pod count
worker_company_replicas	1	Company worker pod count
worker_debug	FALSE	Worker debug flag
worker_person_datasets	image default	JSON list of person datasets
worker_company_datasets	image default	JSON list of company datasets
worker_limits_memory	60Gi	Worker memory limit
worker_limits_cpu	12000m	Worker CPU limit
worker_requests_memory	20Mi	Worker memory request
worker_requests_cpu	1m	Worker CPU request

---

Health check

The app Deployment exposes a readiness probe at:

GET /health-check/pumpwood-crawler-bigdata-app/  (port 5000)

Use this path for ingress and load balancer health checks.

---

Related packages

Package	Role
pumpwood-deploy	Orchestrator, Kong, RabbitMQ, Postgres
pumpwood-deploy-auth	Authorization microservice
pumpwood-deploy-datalake	Datalake app and dataloader

---

Development

pip install -e ../pumpwood-deploy
pip install -e .

ruff check src/

---

License

BSD-3-Clause — see LICENSE.