A multiheaded modern data bridging package based on pipeline manifests to integrate between any modern (and old) data stack tools
Project description
Modern Data Integration Tool
A multiheaded modern data bridging package based on pipeline manifests to integrate between any modern (and old) data stack tools
Setup
Quick Install
python -m pip install mdit
Build from source
Clone the repository
git clone https://github.com/Broomva/mdit.git
Install the package
cd mdit && make install
Build manually
After cloning, create a virtual environment
conda create -n mdit python=3.10
conda activate mdit
Install the requirements
pip install -r requirements.txt
Run the python installation
python setup.py install
Usage
The deployment requires a .env file created under local folder:
touch .env
It should have a schema like this:
databricks_experiment_name=''
databricks_experiment_id=''
databricks_host=''
databricks_token=''
databricks_username=''
databricks_password=''
databricks_cluster_id=''
import mdit
# Create a Snowpark session
spark = DatabricksSparkSession().get_session()
# Connect to MLFLow Artifact Server
mlflow_session = DatabricksMLFlowSession().get_session()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for modern_data_integration_tool-0.1.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 52b3901bf71e238d08fe8f7e86c7c1b79c445f4e1f673064e67c1fbb18ae70ee |
|
MD5 | e42e0951f9aa194c4e33be4c90bc2c3d |
|
BLAKE2b-256 | 86a28ac1b2292d5cc256ad56cabdd739077ea0b26f66dd8087099aca782680d8 |
Close
Hashes for modern_data_integration_tool-0.1.0-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7180ed9b7fe44e13c2f3dc787e8943732e07b173d48d41f5f274e236ab1563c5 |
|
MD5 | e961ea788b39473d703f3d17d4f880c5 |
|
BLAKE2b-256 | e7ebbfad234089015c45f561757640dbeaeb6ca833ed3a7f6cbb1cbd4119e632 |