dehelpers 0.2.0

Lightweight, production-hardened Python utilities for data engineering pipelines.

What It Does

• 🌐 Resilient HTTP client for ETL pipelines with bounded retries and exponential backoff.
• 🗄️ PostgreSQL helper with safe pooling, sessions, and auto-rollback.
• 📝 Structured JSON logging with automatic deep secret redaction.

~60 tests · ~94% coverage · CI on Python 3.10–3.13 · Linted with ruff · Type-checked with mypy

Quickstart

pip install "dehelpers[all]"

A complete pipeline in under 15 lines:

from dehelpers import ResilientClient, DatabaseManager, get_logger

log = get_logger("my_pipeline", job_id="daily-sync")
client = ResilientClient()

# Connects automatically via DATABASE_URL env var
with DatabaseManager() as db, client:
    users = client.get("https://jsonplaceholder.typicode.com/users").json()
    log.info("Fetched users", extra={"count": len(users)})

    with db.session() as session:
        for user in users:
            session.execute(
                "INSERT INTO users (id, name) VALUES (:id, :name) ON CONFLICT DO NOTHING",
                {"id": user["id"], "name": user["name"]}
            )
    log.info("Ingestion complete")

Documentation & Links

• 📚 Documentation: Installation, Getting Started, and FAQ
• 📖 API Reference: Full details on every class and function
• 💡 Examples: Runnable scripts for HTTP, DB, and Logging
• 🗺️ Roadmap: Planned features for v0.2 and beyond
• 📝 Medium Article: The story behind building this library

Architecture & Flow

(For an interactive version of this diagram, see the Architecture Docs)

Boundaries & Capabilities

Here is exactly what this package is and what it is not:

Comparison with Standard Setup

How this package compares to a standard DIY setup:

Configuration

Security

Automatic Redaction

The logger and API client automatically redact values for these keys in log output:

password, secret, token, api_key, authorization, dsn, connection_string, credential, passphrase, private_key, client_secret

Matching is case-insensitive substring — e.g. db_password matches password.

You can extend the redaction list:

from dehelpers._redact import redact_dict

result = redact_dict(
    {"my_custom_secret": "value"},
    extra_sensitive_keys=frozenset({"my_custom_secret"}),
)

⚠️ Never Embed Secrets in URLs

URL query parameter values are redacted, but path segments are not. Never construct URLs like:

https://api.example.com/v1/token/abc123/data  # BAD — token in path

Instead, pass secrets via headers or request body.

Fork Safety (Airflow / Multiprocessing)

If you use DatabaseManager in a forked environment (e.g. Airflow workers, multiprocessing), you must either:

1. Create the DatabaseManager inside each worker process, or
2. Call db.dispose() before forking.

SQLAlchemy connection pools are not safe to share across forked processes.

Testing

Unit tests (no PostgreSQL required)

pip install -e ".[dev,dataframe]"
pytest -v --tb=short -m "not postgres"

PostgreSQL integration tests

# Start a local PostgreSQL
docker run -d --name pg-test -e POSTGRES_PASSWORD=test -p 5432:5432 postgres:16

# Run integration tests
DATABASE_URL="postgresql+psycopg://postgres:test@localhost:5432/postgres" \
    pytest -m postgres -v

Coverage

pytest --cov=dehelpers --cov-report=term-missing -m "not postgres"

Code Style & Tools

We use automated tooling to enforce consistent code quality:

# Set up after cloning
pip install -e ".[dev,dataframe]"
pre-commit install

Developer Resources & Standards

To ensure the library remains production-grade, reliable, and easily maintainable, we enforce the following open-source standards:

• CONTRIBUTING.md: Guidelines for cloning the fork, setting up local editable environments, running unit tests, and opening PRs.
• CODE_OF_CONDUCT.md: Our pledge to foster an inclusive, welcoming, and harassment-free community.
• SECURITY.md: Responsible disclosure policy for reporting vulnerabilities.
• ROADMAP.md: Planned features and design direction.
• CHANGELOG.md: Structured history of features, bugfixes, and breaking changes.
• LICENSE: Permissive MIT License.

Maintenance & Support

This library is actively maintained by the author for personal and internal data engineering workflows. It is built with production-minded practices — comprehensive tests, CI, type checking, and structured documentation — but it is not a certified enterprise product.

Release policy: Semantic-ish versioning (v0.x.y) for incremental features and fixes.

Issues, bug reports, and pull requests are welcome. See CONTRIBUTING.md to get started.

License

Distributed under the MIT License. See LICENSE for more information.