dpetl 0.4.0

No project description provided

dpetl — Simple ETL CLI using Frictionless

dpetl is a command-line interface (CLI) tool designed to assist in running the three phases of the ETL (Extract, Transform, Load) process — although currently only the extract phase is implemented.

It is designed to work alongside the Data Package standard specification.

Installation

Install using pip:

pip install dpetl

Usage

Use the --help flag to inspect the CLI documentation:

dpetl --help

Currently, only the extract command is available:

# Run extract using the default datapackage.yaml descriptor
dpetl extract

# Specify a descriptor explicitly
dpetl extract -d path/to/datapackage.yaml
# or
dpetl extract --descriptor path/to/datapackage.yaml

How It Works

The CLI loads a Data Package descriptor (via the frictionless-py Python package) and iterates over its resources.

For each resource, dpetl extract comand reads the custom property:

dptel_extract:

The key mode determines which extractor will run.

Currently available modes:

• api.
• email.

Example Data Package Configuration

# datapackage.yaml
resources:
  - name: invoices
    path: data/invoices.csv
    sources:
      - method: get
        path: https://api.example.com/invoices
    custom:
      dptel_extract:
        mode: api

  - name: payroll_from_email
    path: data/payroll.xlsx
    custom:
      dptel_extract:
        mode: email
        mailbox: INBOX  # optional (defaults to INBOX)
        criteria:
          subject: "Payroll Report" # optional (defaults to resource name)

Extractors

Email Extractor

• Connects to an IMAP server using environment variables:

  • EMAIL_USER.
  • EMAIL_PWD.
  • EMAIL_SMTP.

• Reads configuration from:

dptel_extract:
  mode: email
  mailbox: INBOX        # optional (default: INBOX)
  criteria:             # optional
    subject: "Report"   # optional (default: resource.name)
    from_: "[e-mail]" # optional
    date_gte: 2024-01-01 #optional

Behavior:

• If mailbox is not provided, INBOX is used.
• If criteria.subject is not provided, it defaults to the resource name.
• The extractor searches for the most recent matching email.
• All e-mail attachments are saved to resource.path.

API Extractor

• Reads resource.sources.
• Searches for a source containing a method.
• Downloads the file.
• Saves it to resource.path.

Design Philosophy

The dpetl package follows a convention over configuration philosophy, treating the Data Package descriptor as the single source of truth for ETL process.

Each resource declares how it should be processed through structured metadata, enabling reproducible, declarative, and version-controlled data workflows.

The goal is to keep the CLI simple while allowing flexible strategies driven entirely by configuration rather than imperative scripting.