easy-data-loader 0.1.1

Data transfer utilities between files and databases

Easy Data Loader 🚀

Easy Data Loader is a flexible, modular Python library designed to streamline ETL (Extract, Transform, Load) processes between various data sources (CSV, Excel, Parquet) and SQL databases (MSSQL, PostgreSQL, and others).

✨ Key Features

• Declarative Configuration: Manage connections and pipelines through simple python files and .env resources.
• Integrated CLI: Initialize a standardized project structure with a single command.
• Custom Transformation Hooks: Inject your own Pandas transformation logic directly into the pipeline execution.
• Performance Optimized: Built-in support for chunked loading and writing to handle large datasets efficiently.
• Extensible Architecture: Uses a Factory Pattern for database connectors, making it easy to support new drivers.

---

📦 Installation

Install directly via pip or uv:

pip install easy_data_loader

🚀 Getting Started

1. Initialize a new project structure to generate template configurations:

   easy-loader init
   

2. Review the generated config/ folders for sample resources and pipelines.
3. Run all discovered pipelines across the active configurations:

   easy-loader run_all
   

🏗️ Architecture & Process Flow

The system is designed with modularity in mind. Object dependencies and their standard instantiation lifecycle are executed through the Configuration singleton and the CONNECTOR_FACTORY.

graph TD
    %% Main Execution Flow
    CLI[CLI Application] -->|Instantiates| Pipeline[Pipeline: Load, Procedure, Orchestrator]
    
    %% Configuration and Definitions
    Pipeline -->|Requests Definition & Resources| Config[Configuration Singleton]
    Config -->|Reads| PipelineDef[Pipeline Definitions .py]
    Config -->|Reads| ResourcesEnv[Resource Configs .env]
    
    %% Instantiation of Operations
    Pipeline -->|Uses ResourceConfig| Factory[CONNECTOR_FACTORY]
    Factory -->|Creates| DBConn[DatabaseConnector: SqlServer, SQLite]
    DBConn -->|Provides Engine to| DBOps[DatabaseOperations]
    
    Pipeline -->|Uses FileSettings| FileOps[FileOperations]
    
    %% Pipeline dependencies
    Pipeline -->|Contains| DBOps
    Pipeline -->|Contains| FileOps
    
    %% Audit
    Pipeline -->|Uses| AuditOps[Audit DatabaseOperations]