recurvedata-lib 0.1.561

Common Library for ReOrc Data Platform

Recurve Libraries

For the unified maintenance of public components of the Recurve platform, these codes may be used in both Server and Executor (Worker) environments.

Only Python 3.11+ are supported.

Components

This code repository consists of the following core components:

Core

The foundation of the Recurve platform that provides:

• Base classes and interfaces for platform components
• Jinja2 templating engine integration
• Core configuration management
• Common platform abstractions

Utils

A comprehensive utility library offering:

• Time handling and date manipulation
• Concurrent processing tools
• File system operations and path handling
• String manipulation and text processing
• Logging and error handling utilities
• Data validation helpers

Connectors

A robust data connectivity layer supporting:

• Database connections (MySQL, PostgreSQL, Redshift, BigQuery, etc.)
• Cloud storage (S3, GCS, Azure Blob Storage)
• Messaging services and APIs
• Custom connector development framework

Note: Run make update-connector-schema after updating connector config schemas to regenerate config_schema.py

Schedulers

Airflow integration components including:

• Custom Airflow operators and sensors
• DAG generation utilities
• Workflow scheduling interfaces
• Task dependency management

Operators

Task-specific operators for:

• Data extraction and loading
• Data transformation and processing
• Running Python code
• Running SQL code
• Building and running DBT jobs

Client

A flexible client interface providing:

• Platform API abstractions
• Authentication handling
• Resource management
• Extensible base classes for custom clients
• Connection pooling and retry logic

Executors

Core job execution engine that:

• Manages job submissions and execution flows
• Orchestrates task execution on infrastructure
• Handles job lifecycle and state management
• Provides infrastructure abstraction layer

Development Workflow

Requirements management

We use uv to manage Python package dependencies. The workflow is:

1. Update source requirements in .in files:

   • requirements.in - All dependencies
   • requirements/worker.in - Worker-specific dependencies
   • requirements/dbt.in - DBT-specific dependencies
   • requirements-dev.in - Development dependencies

2. Compile locked requirements:

   make compile-requirements  # Compiles all requirements files
   

   Or compile individual files:

   make compile-worker  # Just worker requirements
   make compile-dbt    # Just DBT requirements
   

3. After compiling requirements, update optional dependencies in pyproject.toml:

   make update-optional-deps
   

This ensures consistent dependencies across development and production environments.

Release Process

1. Update version number in recurvedata/__version__.py
2. Build and publish package:

   make publish
   

   This will clean build artifacts, build new package, and publish to Recurve PyPI.

Available Commands

The following make commands are available:

Build and Publishing:

• make clean - Remove build artifacts (dist directory)
• make build - Clean and build the package
• make publish - Build and publish package to Recurve PyPI

Requirements Management:

• make upgrade-uv - Upgrade the uv package installer
• make compile-worker - Compile worker-specific requirements
• make compile-dbt - Compile DBT-specific requirements
• make compile-requirements - Compile all requirements files and sync environment
• make install-requirements - Install requirements files

Maintenance Scripts:

• make update-optional-deps - Update optional dependencies in pyproject.toml
• make update-connector-schema - Update connector configuration schemas

GitLab CI/CD Pipeline

Overview

This repository uses GitLab CI for build and release. The primary stages (in order) are:

• python_internal: build and publish internal PyPI package (develop)
• python_public: build and publish public PyPI package (main, manual)
• copy_dockerfiles: prepare Docker build context and version metadata
• docker_internal: build and push internal Docker images (develop / release/*)
• docker_official: build and push public Docker images (main, manual)
• scan: SAST/Sonar (disabled by default)

Configuration files:

• .gitlab-ci.yml (stages, rules, job orchestration)
• .gitlab/ci/templates.yml (shared templates and login configuration)
• .gitlab/ci/variables.yml (shared variables)
• dockerfiles/build_push_images.sh (image build and push script)

Branch rules and triggers

• develop branch

  • Runs: check_version → package_python_internal → copy_dockerfiles → build_docker_internal
  • Environment: ENVIRONMENT=test
  • Targets: internal private registry + Aliyun internal namespace

• release/* branches

  • Runs: copy_dockerfiles → build_docker_internal
  • Environment: ENVIRONMENT=staging
  • Targets: internal private registry + Aliyun internal namespace

• main branch

  • Runs (manual): package_python_public, build_docker_public, tag
  • Environment: ENVIRONMENT=production
  • Targets: Docker Hub public registry + Aliyun public namespace

Note: the tag job creates an annotated tag after the public image build, including deployment metadata.

Image build and naming

Build script: dockerfiles/build_push_images.sh

• Image names:
  • Production: recurve-<service> (current service = worker)
  • Non-production: recurve-<service>-<environment> (e.g., recurve-worker-test, recurve-worker-staging)
• Tags:
  • version: ${VERSION_PACKAGE} (from recurvedata/__version__.py)
  • latest

Push targets

• Internal private registry (login handled by job type)

  • Registry: $DOCKER_REPOSITORY_URL (internal jobs)
  • Example path: docker.tool.reorc.cloud/<image_name>:<tags>

• Public Docker Hub (public jobs)

  • Namespace: recurvedata/<image_name>:<tags>

• Aliyun Container Registry (pushed in addition for all Docker build jobs)

  • Registry: reorc-registry-cn-registry-vpc.cn-shenzhen.cr.aliyuncs.com
  • Namespace mapping:
    • Non-production (test/staging): internal
    • Production: public
  • Full path: <registry>/<namespace>/<image_name>:<tags>

Login and authentication

Templates in .gitlab/ci/templates.yml:

• .docker_internal_configuration: login to internal private registry and also login to Aliyun
• .docker_public_configuration: login to Docker Hub and also login to Aliyun

Both templates execute docker logout || true once in before_script, then log in to the respective registry and Aliyun so a single job can push to multiple registries.

Required CI variables (examples)

Configure the following as Masked/Protected variables in GitLab CI/CD settings:

# Nexus / PyPI
NEXUS_USERNAME, NEXUS_PASSWORD, NEXUS_REPOSITORY_URL, NEXUS_PACKAGE_URL
PYPI_USERNAME, PYPI_PASSWORD, PYPI_REPOSITORY_URL (optional)

# Private registry (internal jobs)
DOCKER_REPOSITORY_URL, DOCKER_USERNAME, DOCKER_PASSWORD

# Docker Hub (public jobs)
DOCKER_OFFICIAL_REPOSITORY_URL, DOCKER_OFFICIAL_USERNAME, DOCKER_OFFICIAL_PASSWORD

# Aliyun registry (additional push in all Docker jobs)
ALIYUN_REGISTRY_URL=reorc-registry-cn-registry-vpc.cn-shenzhen.cr.aliyuncs.com
ALIYUN_USERNAME, ALIYUN_PASSWORD

# Namespaces declared in repo variables (adjust if needed)
ALIYUN_NAMESPACE_INTERNAL=internal
ALIYUN_NAMESPACE_PUBLIC=public

Manual triggers and verification

• On the main branch, package_python_public and build_docker_public are manual; click “Play” in the GitLab Pipeline UI.
• Verify images:
  • Internal: docker.tool.reorc.cloud/<image_name>:<tag>
  • Docker Hub: recurvedata/<image_name>:<tag>
  • Aliyun: reorc-registry-cn-registry-vpc.cn-shenzhen.cr.aliyuncs.com/<namespace>/<image_name>:<tag>