Snowpark Session Util

A simple utility for spark and mlflow session objects

Setup

Quick Install

python -m pip install snowpark_session

Build from source

Clone the repository

git clone https://github.com/Broomva/snowpark_session.git

Install the package

cd snowpark_session && make install

Build manually

After cloning, create a virtual environment

conda create -n snowpark_session python=3.8
conda activate snowpark_session
conda install snowflake-snowpark-python pandas

Install the requirements

pip install -r requirements.txt

Run the python installation

python setup.py install

Usage

The deployment requires a .env file created under local folder:

touch .env

It should have a schema like this:

snowflake_account=''
snowflake_user=''
snowflake_password=''
snowflake_user_role=''
snowflake_warehouse=''
snowflake_database=''
snowflake_schema=''

# Optional
azureml_experiment_name=''
azureml_experiment_id=''
azureml_subscription_id=''
azureml_resource_group=''
azureml_workspace_name=''

import snowpark_session as ss

# Create a Spark session
spark = ss.SnowparkSession().get_session()

# Connect to MLFLow Artifact Server
mlflow_session = ss.AzureMLFlowSession().get_session()