snowforge-package 0.2.4

A Python package for supporting migration from on-prem to cloud

🚀 Snowforge - Powerful Data Integration

Snowforge is a Python package designed to streamline data integration and transfer between AWS, Snowflake, and various on-premise database systems. It provides efficient data extraction, logging, configuration management, and AWS utilities to support robust data engineering workflows.

---

✨ Features

• AWS Integration: Manage AWS S3 and Secrets Manager operations.
• Snowflake Connection: Establish and manage Snowflake connections effortlessly.
• Advanced Logging: Centralized logging system with colored output for better visibility.
• Configuration Management: Load and manage credentials from a TOML configuration file.
• Data Mover Engine: Parallel data processing and extraction strategies for efficiency.
• Extensible Database Extraction: Uses a strategy pattern to support multiple on-prem database systems (e.g., Netezza, Oracle, PostgreSQL, etc.).

---

📥 Installation

Install Snowforge using pip:

pip install snowforge

---

⚙️ Configuration

Snowforge requires a configuration file (snowforge_config.toml) to manage credentials for AWS and Snowflake. The package searches for the config file in the following locations:

1. Path specified in SNOWFORGE_CONFIG_PATH environment variable.
2. Current working directory.
3. ~/.config/snowforge_config.toml
4. Package directory.

Example snowforge_config.toml File

[AWS]
[default]
AWS_ACCESS_KEY = "your-access-key"
AWS_SECRET_KEY = "your-secret-key"
REGION = "us-east-1"

[SNOWFLAKE]
[default]
USERNAME = "your-username"
ACCOUNT = "your-account"

---

🚀 Quick Start

🔹 Initialize AWS Integration

from snowforge.AWSIntegration import AWSIntegration

AWSIntegration.initialize(profile="default", verbose=True)

🔹 Connect to Snowflake

from snowforge.SnowflakeConnect import SnowflakeConnection

conn = SnowflakeConnection.establish_connection(user_name="your-user", account="your-account")

🔹 Use Logging

from snowforge.Logging import Debug

Debug.log("This is an info message", level='INFO')
Debug.log("This is an error message", level='ERROR')

🔹 Extract Data from an On-Prem Database

from snowforge.Extractors.NetezzaExtractor import NetezzaExtractor

extractor = NetezzaExtractor()
query = extractor.extract_table_query("database.schema.table_name", "date_column", "01.01.2024")
print(query)

Since Snowforge follows a strategy pattern, it can be easily extended to support other on-prem database systems by implementing new extractor classes that conform to the ExtractorStrategy interface.

---

🤝 Contributing

Contributions are welcome! To contribute, follow these steps:

1. Fork the repository.
2. Create a new branch (feature/new-feature).
3. Commit your changes.
4. Push to your fork and submit a pull request.

For major changes, please open an issue first to discuss what you would like to change.

---

📜 License

This project is licensed under the MIT License.

---

👤 Author

Developed by [Your Name or Organization]. Feel free to reach out for support, suggestions, or collaboration!