No project description provided
Project description
AZDSDR - Data Scientist's Data Reader
This package includes data reader for DS to access data in a easy way.
Covered data platforms
- Dremio
- Kusto
Installation
Use pip to install the package and all of the dependences
pip install -U azdsdr
The -U
will help update your old version to the newest
Or, you can clone the repository and copy over the readers.py
file to your project folder.
Use Dremimo Reader
Step 1. Install Dremio Connector
You will need to install the Dremio ODBC driver first to use DremioReader
from this package.
For Windows user
Please download the dremio-connector file from the drivers folder.
Step 2. Generate a Personal Access Token(PAT)
- Recommend storing this personal access token in a safe location, such as a user environment variable on your local machine.
- Start Menu -> “Edit Environment variables For Your Account”.
- Click “New” under environment variables.
- Enter a new variable with name “DREMIO_TOKEN” and set the value to the PAT you generated earlier.
Step 3. Configure driver
- Go to Start Menu -> “ODBC Data Sources (64-bit)”.
- Under User DSN, click “Add”.
- Add Dremio Connector.
- Configure as follows, with your own @host.com as the username.Do remember to replace the dremio host with your own host string.
- Click Ok/Save
For Linux and Mac User
You can download the driver from Dremio's ODBC Driver page.
Dremio Sample Query
from azdsdr.readers import DremioReader
import os
username = "name@host.com"
#token = "token string"
token = os.environ.get("DREMIO_TOKEN")
dr = DremioReader(username=username,token=token)
sql = '''
select
*
from
[workspace].[folder].[tablename]
limit 10
'''
r = dr.run_sql(sql)
Use Kusto Reader
Azure CLI Authentication
Before running the kusto query, please use
az login
To login into Azure using AAD authentication. An authentication refresh token is generated by Azure and stored in your local machine. This token will be revoked after 90 days of inactivity.
For More details, read Sign in with Azure CLI.
After successufuly authenticated with AAD, you should be able to run the following code without any pop up auth request.
Kusto Sample Query
from readers import KustoReader
cluster = "https://help.kusto.windows.net"
db = "Samples"
kr = KustoReader(cluster=cluster,db=db)
kql = "StormEvents | take 10"
r = kr.run_kql(kql)
The Kusto Reader is test in Windows 10, in theroy should also work in Linux and Mac.
Thanks
The Dremio ODBC Reader solution is origin from KC Munnings. Glory and credits belong to KC.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.