havinosh-data-loader 0.0.3

A Python package to dynamically load CSV files into PostgreSQL tables.

📦 havinosh_data_loader: CSV to PostgreSQL Ingestion

Seamlessly load CSV files into PostgreSQL dynamically.

---

🚀 Overview

havinosh_data_loader is a powerful Python package designed to automatically detect CSV structure, create tables, and insert data into PostgreSQL with minimal effort.
Simply drop your CSV files into a folder, and havinosh_data_loader will handle everything dynamically.

---

📌 Features

✅ Automatic Table Creation – Reads CSV headers and maps them to PostgreSQL columns
✅ Dynamic Schema Detection – Infers data types automatically
✅ Batch Data Insertion – Efficiently loads large datasets
✅ Error Handling – Logs errors and provides detailed debugging
✅ Environment Configuration – Uses .env file for database credentials
✅ Modular & Extensible – Well-structured for easy modifications

---

📂 Project Structure

havinosh_data_loader/
│── havinosh_data_loader/                   # Main package directory
│   ├── __init__.py           # Package initialization
│   ├── config.py             # Database configuration
│   ├── process_csv.py        # CSV processing logic (OOP-based)
│   ├── db_utils.py           # PostgreSQL connection utilities
│   ├── exception.py          # Custom exception handling
│   ├── logger.py             # Logging setup
│── scripts/                  # Command-line scripts
│   ├── ingest.py             # Main ingestion script (CLI)
│── tests/                    # Unit tests
│   ├── test_process_csv.py
│── csv_files/                # CSV storage (ignored in production)
│── setup.py                  # Package setup script
│── requirements.txt          # Dependencies
│── README.md                 # Documentation
│── .gitignore                # Ignore unnecessary files
│── LICENSE                   # License details

---

🛠 Installation

1️⃣ Install via pip

Once published to PyPI, you can install havinosh_data_loader using:

pip install havinosh_data_loader

2️⃣ Install from Source

Clone the repository and install dependencies:

git clone https://github.com/IAMDSVSSANGRAL/havinosh_data_loader.git  
cd havinosh_data_loader  
pip install -r requirements.txt  

---

⚙ Usage

1️⃣ Set Up Your Environment

🔹 Using a .env File (Recommended)

Create a .env file in the root directory and add:

DB_NAME=your_database
DB_USER=your_username
DB_PASSWORD=your_password
DB_HOST=localhost
DB_PORT=5432

💡 Note: This package requires .env for database credentials.

---

2️⃣ Using the CLI

Basic Ingestion (Uses .env for DB credentials)

python scripts/ingest.py --folder csv_files

---

3️⃣ Using the Python Package in Your Script

You can also use havinosh_data_loader programmatically within a Python script:

from havinosh_data_loader import Database, CSVProcessor, Config

# Load database configuration from .env
config = Config()
db = Database(config.get_config())

# Define the folder where CSV files are stored
csv_folder = "csv_files/"

# Initialize the processor
processor = CSVProcessor(db_instance=db, csv_folder=csv_folder)

# Start the ingestion process
processor.process_csv()

---

🧪 Testing

Run unit tests using:

pytest tests/

---

🔥 Limitations & Future Enhancements

• 🚧 Does not support nested JSON in CSV fields
• 🚀 Planned: Add parallel processing for large datasets

---

📜 License

This project is licensed under the MIT License – see the LICENSE file for details.

---

🤝 Contributing

Contributions are welcome! Please follow these steps:

1. Fork the repository
2. Create a branch (feature-xyz)
3. Commit your changes
4. Push and submit a Pull Request

---

📬 Contact

📧 Email: [e-mail]
🌐 GitHub: VISHAL SINGH SANGRAL

---

🔹 What's New in This Version?

This version includes major improvements to enhance usability and clarity:
✅ Refactored to OOP-based structure – CSVProcessor, Database classes added
✅ Improved CLI Usage – Removed incorrect --db_name argument, making .env mandatory
✅ Better Documentation – Clarified import structure for easier package use
✅ Enhanced Logging & Exception Handling – Better debugging experience
✅ More Detailed Testing Instructions – Now includes a dedicated testing section
✅ Added Future Enhancements Section – Transparency on upcoming features

---

🚀 Happy Data Ingestion! 🎉