A multiheaded modern data bridging package based on pipeline manifests to integrate between any modern (and old) data stack tools
Project description
Modern Data Integration Tool
A multiheaded modern data bridging package based on pipeline manifests to integrate between any modern (and old) data stack tools
Setup
Quick Install
python -m pip install mdit
Build from source
Clone the repository
git clone https://github.com/Broomva/mdit.git
Install the package
cd mdit && make install
Build manually
After cloning, create a virtual environment
conda create -n mdit python=3.10
conda activate mdit
Install the requirements
pip install -r requirements.txt
Run the python installation
python setup.py install
Usage
The deployment requires a .env file created under local folder:
touch .env
It should have a schema like this:
databricks_experiment_name=''
databricks_experiment_id=''
databricks_host=''
databricks_token=''
databricks_username=''
databricks_password=''
databricks_cluster_id=''
import mdit
# Create a Spark session
spark = DatabricksSparkSession().get_session()
# Connect to MLFLow Artifact Server
mlflow_session = DatabricksMLFlowSession().get_session()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for modern_data_integration_tool-0.1.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | ce10115957aa09a21d55eb89e1e481320ee21b444a1d1ed8a3f37c8dff490a02 |
|
MD5 | 0df903415bb25b4be5a4c3755c0fd9a9 |
|
BLAKE2b-256 | 37620161748abf2d862b5f670023745e6e56ad9d394285d82fc062af0a634457 |
Close
Hashes for modern_data_integration_tool-0.1.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2342eaa8e576e9c361894993d8628adf84cfcaeca95fde93dc7a1bbf1cfa87e3 |
|
MD5 | a53fbd4a828247919b10a6612bea0c13 |
|
BLAKE2b-256 | 6cfb98e15f61a0645a75b9d00543b9067fa3106ee5cdd06c1614522d42a7b4c4 |