etl-package-maiyo008 0.0.1

An ETL package that loads data, cleans and loads it to a postgres database

etl_package_maiyo008

---

A simple yet extensible Python ETL (Extract, Transform, Load) pipeline that supports loading data from Parquet files, performing basic transformations, and writing to a PostgreSQL database. Designed to be easily reusable and ideal for automation tasks.

📦 Installation

To install the package from PyPI:

pip install etl-package-maiyo008

🧠 Features

• Extract data from Parquet files
• Transform data by:
  • Removing duplicates
  • Dropping rows with blank fields
• Load data into PostgreSQL with:
  • Chunked inserts for performance
  • Progress feedback via tqdm

📁 Project Structure

etl_package_maiyo008/
├── extract.py
├── transform.py
├── load.py
├── __init__.py

🚀 Quickstart Guide

1. Import the package

from etl_package_maiyo008.etl import Extract, Transform, Load

2. Extract data

df = Extract.load_parquet("data/sample_data.parquet")

3. Transform data

df_clean = Transform.remove_duplicates(df)
df_clean = Transform.remove_blanks(df_clean)

4. Load Data to PostgreSQL

conn = Load.connect_postgres(
    database="mydb",
    host="localhost",
    user="myuser",
    password="mypassword"
)

Load.write_to_db(df_clean, table_name="cleaned_data", conn=conn)

📚 Method Reference

Extract Extract.load_parquet(path: str)

• Loads a Parquet file into a pandas DataFrame.

Transform Transform.remove_duplicates(df: pd.DataFrame)

• Removes duplicate rows from the DataFrame.

Transform.remove_blanks(df: pd.DataFrame)

• Removes rows with missing/blank values.

Load Load.connect_postgres(database, host, user, password, port=5432)

• Creates a connection to a PostgreSQL database using SQLAlchemy.

Load.write_to_db(df, table_name, conn, chunk_size=1000)

• Writes the DataFrame to a PostgreSQL table in chunks, providing progress feedback.

⚠️ Requirements

• pandas
• psycopg2-binary
• SQLAlchemy
• tqdm
• pyarrow or fastparquet (for Parquet support)

Install requirements with:

pip install pandas psycopg2-binary SQLAlchemy tqdm pyarrow

✅ License

• MIT License