QuickELT

 ██████╗ ██╗   ██╗██╗ ██████╗██╗  ██╗███████╗██╗  ████████╗
██╔═══██╗██║   ██║██║██╔════╝██║ ██╔╝██╔════╝██║  ╚══██╔══╝
██║   ██║██║   ██║██║██║     █████╔╝ █████╗  ██║     ██║   
██║▄▄ ██║██║   ██║██║██║     ██╔═██╗ ██╔══╝  ██║     ██║   
╚██████╔╝╚██████╔╝██║╚██████╗██║  ██╗███████╗███████╗██║   
 ╚══▀▀═╝  ╚═════╝ ╚═╝ ╚═════╝╚═╝  ╚═╝╚══════╝╚══════╝╚═╝   

QuickELT is a modern, fast CLI tool to scaffold production-ready Lakehouse projects in seconds. It provides data ingestion templates, standardizes the Medallion Architecture, and automates infrastructure setup.

🚀 Quick Start

You can run quickelt without installing it permanently using uvx!

uvx --from quickelt quickelt init my_lakehouse

Alternatively, install it globally via uv:

uv tool install quickelt
quickelt init my_lakehouse

🛠️ Usage

Interactive Mode (Wizard)

By simply running quickelt init, you'll be greeted with an interactive wizard that guides you through the project setup:

quickelt init

Non-Interactive Mode

For CI/CD environments or power users, you can bypass the wizard by providing flags. Use the --yes (-y) flag to automatically accept default values for any omitted configurations.

quickelt init my_project --engine polars --storage adls --quality gx --iac terraform --yes

💻 Commands

Available Flags for init:

• --storage / -s: Cloud / Storage Target (s3, adls, gcs, minio)
• --engine / -e: Execution Engine (polars, duckdb)
• --quality / -q: Data Quality (pandera, gx, none)
• --iac / -i: Infrastructure (terraform, none)
• --yes / -y: Skip confirmations and use defaults when something is omitted.

🏗️ Project Structure (Medallion Architecture)

The generated project follows best practices for Data Engineering, structured around the Medallion Architecture:

graph TD
    A[my_lakehouse/] --> B[config/]
    A --> C[pipelines/]
    A --> D[contracts/]
    A --> E[infrastructure/]
    
    C --> C1[bronze/]
    C --> C2[silver/]
    C --> C3[gold/]
    
    C1 -. Raw Data Ingestion .-> C2
    C2 -. Cleansing & Joins .-> C3
    C3 -. Business Aggregations .-> F[(Analytics / BI)]
    
    D -. Data Quality Schemas .-> C2

Directory breakdown:

• config/: Environment and project settings (.env, settings.py).
• pipelines/: Data transformation logic separated into Bronze, Silver, and Gold layers.
• contracts/: Data quality schemas (Pandera or Great Expectations).
• infrastructure/: Infrastructure as Code (Terraform) to provision cloud resources.

🤝 Contributing

Contributions are welcome! To develop quickelt locally:

1. Clone the repository and install dependencies using uv:

   git clone https://github.com/mpraes/quickelt.git
   cd quickelt
   uv sync
   

2. Make your changes and test the CLI:

   uv run quickelt init test_project
   

3. Run tests:

   uv run pytest
   

4. Submit a Pull Request describing your changes.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.