azdsdr 1.221005.1

No project description provided

AZDSDR - Data Scientist's Data Reader

• AZDSDR - Data Scientist's Data Reader
  • Installation
  • Use Dremimo Reader
    • Step 1. Install Dremio Connector
    • Step 2. Generate a Personal Access Token(PAT)
    • Step 3. Configure driver
    • Dremio Sample Query
  • Use Kusto Reader
    • Azure CLI Authentication
    • Kusto Sample Query
  • Thanks

This package includes data reader for DS to access data in a easy way.

Covered data platforms

• Dremio
• Kusto

Installation

Use pip to install the package and all of the dependences

pip install -U azdsdr

The -U will help update your old version to the newest

Or, you can clone the repository and copy over the readers.py file to your project folder.

Use Dremimo Reader

Step 1. Install Dremio Connector

You will need to install the Dremio ODBC driver first to use DremioReader from this package.

For Windows user

Please download the dremio-connector file from the drivers folder.

Step 2. Generate a Personal Access Token(PAT)

• Recommend storing this personal access token in a safe location, such as a user environment variable on your local machine.
• Start Menu -> “Edit Environment variables For Your Account”.
• Click “New” under environment variables.
• Enter a new variable with name “DREMIO_TOKEN” and set the value to the PAT you generated earlier.

Step 3. Configure driver

• Go to Start Menu -> “ODBC Data Sources (64-bit)”.
• Under User DSN, click “Add”.
• Add Dremio Connector
• Configure as follows, with your own @host.com as the username.

• Click Ok/Save

For Linux and Mac User

You can download the driver from Dremio's ODBC Driver page.

Dremio Sample Query

from azdsdr.readers import DremioReader
import os

username    = "[e-mail]"
#token       = "token string"
token       = os.environ.get("DREMIO_TOKEN") 
dr          = DremioReader(username=username,token=token)

sql = '''
select 
    * 
from 
    [workspace].[folder].[tablename]
limit 10
'''
r = dr.run_sql(sql)

Use Kusto Reader

Azure CLI Authentication

Before running the kusto query, please use

az login

To login into Azure using AAD authentication. An authentication refresh token is generated by Azure and stored in your local machine. This token will be revoked after 90 days of inactivity.

For More details, read Sign in with Azure CLI.

After successufuly authenticated with AAD, you should be able to run the following code without any pop up auth request.

Kusto Sample Query

from readers import KustoReader

cluster = "https://help.kusto.windows.net"
db      = "Samples"
kr = KustoReader(cluster=cluster,db=db)

kql = "StormEvents | take 10"
r = kr.run_kql(kql)

The Kusto Reader is test in Windows 10, in theroy should also work in Linux and Mac.

Thanks

The Dremio ODBC Reader solution is origin from KC Munnings. Glory and credits belong to KC.