dpetl 0.6.0

No project description provided

dpetl — Data package ETL

dpetl is a command-line interface (CLI) tool designed to assist in running the three phases of the ETL (Extract, Transform, Load) process (although currently only the extract phase is implemented).

It is designed to work alongside the Data Package standard specification.

Installation

Install using pip:

pip install dpetl

Usage

Use the --help flag to inspect the CLI documentation:

dpetl --help

Currently, only the extract command is available:

# Run extract using the default datapackage.yaml descriptor
dpetl extract

# Specify a descriptor explicitly
dpetl extract -d path/to/datapackage.yaml
# or
dpetl extract --descriptor path/to/datapackage.yaml

How It Works

The CLI loads a Data Package descriptor (via the frictionless-py Python package) and iterates over its resources.

For each resource, dpetl extract comand reads the custom property:

dptel_extract:

The key mode determines which extractor will run.

Currently available modes:

• api.
• email.

Example Data Package Configuration

# datapackage.yaml
resources:
  - name: invoices
    path: data/invoices.csv
    sources:
      - method: get
        path: https://api.example.com/invoices
		dptel_extract:
			mode: api

  - name: payroll_from_email
    path: data/payroll.xlsx
		dptel_extract:
			mode: email
			mailbox: INBOX  # optional (defaults to INBOX)
			criteria:
				subject: "Payroll Report" # optional (defaults to resource name)

Extractors

Email Extractor

• Connects to an IMAP server using environment variables:

  • EMAIL_USER.
  • EMAIL_PWD.
  • EMAIL_IMAP.
    • HTTP_PROXY[^1].

[^1]: Just in case you're running the command behind a corporate network that demands proxy configuration. The HTTPS_PROXY, http_proxy and https_proxy are equally acceptable. See this Issue's comment to understand why maybe you'll have to add authentication (http://<user>:<pwd>@<host>:<port>) on PROXY address.

• Reads configuration from:

dptel_extract:
  mode: email
  mailbox: INBOX        # optional (default: INBOX)
  criteria:             # optional
    subject: "Report"   # optional (default: resource.name)
    from_: "[e-mail]" # optional
    date_gte: 2024-01-01 #optional

Behavior:

• If dptel_extract.mailbox is not provided, INBOX is used.
• If dptel_extract.criteria.subject is not provided, it defaults to the resource name.
• The extractor searches for the most recent matching email.
• All e-mail attachments are saved to resource.path.

API Extractor

• Reads resource.sources.
• Searches for a source containing a method.
• Downloads the file.
• Saves it to resource.path.

Design Philosophy

The dpetl package follows a convention over configuration philosophy, treating the Data Package descriptor as the single source of truth for ETL process.

Each resource declares how it should be processed through structured metadata, enabling reproducible, declarative, and version-controlled data workflows.

The goal is to keep the CLI simple while allowing flexible strategies driven entirely by configuration rather than imperative scripting.