pumpwood-deploy-crawler-escavador

Satellite deploy package for the Pumpwood Crawler Escavador microservice on Kubernetes. It generates manifests for the API application, the legal proceeding worker, and crawler secrets — then hands them to pumpwood-deploy for apply.

Developed by Murabei Data Science. BSD-3-Clause.

Pumpwood is a native Brazilian tree with a symbiotic relation to ants (Murabei)

---

Objective and motivation

This package deploys the Crawler Escavador app and its legal proceeding RabbitMQ worker onto a Pumpwood Kubernetes cluster.

Why this exists

Queue CRUD lives in the API image; Escavador fetch and ETL live in the worker image. The cluster needs one app Service and one worker Deployment that consume CrawlerEscavadorLegalProceedingQueue__on_queue.

How it is used

A cluster deploy script imports PumpWoodCrawlerEscavadorMicroservice and calls DeployPumpWood.add_microservice. PgBouncer is added in that same script via PGBouncerDatabase from pumpwood-deploy.

Scope

In scope: Secret, app Deployment + Service, legal proceeding worker.

Out of scope: Postgres, PgBouncer, Kong, Auth, Datalake, Complex Datalake, Graph Datalake, Models LLM, and the database itself.

---

What it deploys

Manifest	Kubernetes resources
pumpwood_crawler_escavador__secrets	Secret pumpwood-crawler-escavador
pumpwood_crawler_escavador__deploy	Deployment + Service pumpwood-crawler-escavador-app
pumpwood_crawler_escavador__worker	Deployment pumpwood-crawler-escavador-worker

flowchart LR
    subgraph pkg [pumpwood-deploy-crawler-escavador]
        A[PumpWoodCrawlerEscavadorMicroservice]
    end
    subgraph core [pumpwood-deploy]
        B[DeployPumpWood]
    end
    subgraph cluster [Cluster]
        S[crawler Secret]
        APP[crawler-escavador-app]
        W[legal proceeding worker]
        RMQ[rabbitmq-main]
    end
    A --> B
    B --> S
    B --> APP
    B --> W
    APP --> RMQ
    RMQ --> W

---

Prerequisites

This package does not stand alone. Before crawler pods can start, the cluster must already provide:

Resource	Provided by
storage ConfigMap	StandardMicroservices in pumpwood-deploy
general-secrets	StandardMicroservices
rabbitmq-main-secrets	StandardMicroservices
Storage keys (GCP / Azure / AWS)	DeployPumpWood storage config
Postgres for crawler	PGBouncerDatabase in the cluster script
Auth	pumpwood-deploy-auth
Datalake	pumpwood-deploy-datalake
Complex Datalake	complex-datalake satellite
Graph Datalake	graph-datalake satellite
Models LLM	models-llm satellite (document extraction)

Storage bucket name and type are read from the cluster storage ConfigMap — they are not passed to PumpWoodCrawlerEscavadorMicroservice.

The database named in db_database must already exist on the external Postgres. PGBouncerDatabase only pools connections.

The worker queue name is fixed in the worker image as CrawlerEscavadorLegalProceedingQueue__on_queue.

---

Installation

pip install pumpwood-deploy-crawler-escavador

Requires pumpwood-deploy.

---

Quick start

import os
import simplejson as json
from dotenv import load_dotenv
from pumpwood_deploy.deploy import DeployPumpWood
from pumpwood_deploy.microservices.postgres.deploy import (
    PGBouncerDatabase)
from pumpwood_deploy_crawler_escavador import (
    PumpWoodCrawlerEscavadorMicroservice)

with open("secrets/production.json", "r") as file:
    secrets = json.loads(file.read())
load_dotenv()

deploy.add_microservice(
    PGBouncerDatabase(
        name="pgbouncer-pumpwood-crawler-escavador",
        postgres_database="pumpwood_crawler_escavador",
        postgres_secret="postgres-general",
        postgres_host=POSTGRES_HOST,
    ))

deploy.add_microservice(
    PumpWoodCrawlerEscavadorMicroservice(
        app_version=os.getenv("PUMPWOOD_ESCAVADOR_CRAWLER_APP"),
        worker_version=os.getenv(
            "PUMPWOOD_ESCAVADOR_CRAWLER_WORKER"),
        repository="my-registry.example.com",
        db_host="pgbouncer-pumpwood-crawler-escavador",
        db_database="pumpwood_crawler_escavador",
        db_password=secrets["postgres_password"],
        microservice_password=secrets[
            "microservice--crawler-escavador"],
        escavador_auth_token=secrets["escavador_auth_token"],
        openai_api_key=secrets["openai_api_key"],
        app_replicas=1,
        app_debug="FALSE",
        worker_replicas=1,
    ))

deploy.create_deploy_files()
deploy.deploy_microservices()

Environment variables

PUMPWOOD_ESCAVADOR_CRAWLER_APP=0.0.3
PUMPWOOD_ESCAVADOR_CRAWLER_WORKER=0.0.10

If the rendered manifest matches what is already on the cluster, kubectl apply produces no changes — safe for rolling image updates.

---

Configuration reference

Required

Parameter	Description
app_version	Image tag for pumpwood-crawler-escavador-app
worker_version	Image tag for pumpwood-crawler-escavador-worker

Workers need a non-empty escavador_auth_token to call Escavador.

Database

Parameter	Default	Description
db_host	postgres-pumpwood-crawler-escavador	Postgres host (PgBouncer in prod)
db_port	5432	Postgres port
db_database	pumpwood	Database name
db_username	pumpwood	Database user
db_password	pumpwood	Database password
microservice_password	microservice--crawler-escavador	Service user password
repository	GCR default	Docker registry for app and worker

Application

Parameter	Default	Description
app_replicas	1	Number of app pods
app_debug	FALSE	Debug flag
app_workers	10	Granian workers (GRANIAN_WORKERS)
app_timeout	300	Request timeout (seconds)
app_limits_memory	4Gi	Memory limit
app_limits_cpu	3000m	CPU limit
app_requests_memory	1Gi	Memory request
app_requests_cpu	500m	CPU request

Worker

Queue: CrawlerEscavadorLegalProceedingQueue__on_queue (hardcoded in the worker image).

Parameter	Default	Description
worker_replicas	1	Worker pod count
worker_debug	FALSE	Worker debug flag
worker_query_limit	100000	ETL batch size (QUERY_LIMIT)
escavador_auth_token	""	Escavador API token
openai_api_key	""	OpenAI key for MarkItDown
worker_limits_memory	4Gi	Worker memory limit
worker_limits_cpu	2000m	Worker CPU limit
worker_requests_memory	20Mi	Worker memory request
worker_requests_cpu	1m	Worker CPU request

---

Health check

The app Deployment exposes a readiness probe at:

GET /health-check/pumpwood-crawler-escavador-app/  (port 5000)

Use this path for ingress and load balancer health checks.

---

Related packages

Package	Role
pumpwood-deploy	Orchestrator, Kong, RabbitMQ, Postgres
pumpwood-deploy-auth	Authorization microservice
pumpwood-deploy-datalake	Datalake app and dataloader
pumpwood-deploy-crawler-bigdata	Sister crawler satellite

---

Development

pip install -e ../pumpwood-deploy
pip install -e .

ruff check src/

---

License

BSD-3-Clause — see LICENSE.