Dask + Snowflake intergration
Project description
Dask-Snowflake
Installation
dask-snowflake can be installed with pip:
pip install dask-snowflake
or with conda:
conda install -c conda-forge dask-snowflake
Usage
dask-snowflake provides read_snowflake and to_snowflake methods
for parallel IO from Snowflake with Dask.
>>> from dask_snowflake import read_snowflake
>>> example_query = '''
... SELECT *
... FROM SNOWFLAKE_SAMPLE_DATA.TPCH_SF1.CUSTOMER;
... '''
>>> ddf = read_snowflake(
... query=example_query,
... connection_kwargs={
... "user": "...",
... "password": "...",
... "account": "...",
... },
... )
>>> from dask_snowflake import to_snowflake
>>> to_snowflake(
... ddf,
... name="my_table",
... connection_kwargs={
... "user": "...",
... "password": "...",
... "account": "...",
... },
... )
See their docstrings for further API information.
Tests
Running tests requires a Snowflake account and access to a database.
The test suite will automatically look for specific SNOWFLAKE_*
environment variables (listed below) that must be set.
It's recommended (though not required) to store these environment variables
in a local .env file in the root of the dask-snowflake repository.
This file will be automatically ignored by git, reducing the risk of accidentally
commiting it.
Here's what an example .env file looks like:
SNOWFLAKE_USER="<test user name>"
SNOWFLAKE_PASSWORD="<test_user_password>"
SNOWFLAKE_ACCOUNT="<account>.<region>.aws"
SNOWFLAKE_WAREHOUSE="<test warehouse>"
SNOWFLAKE_ROLE="<test role>"
SNOWFLAKE_DATABASE="<test database>"
SNOWFLAKE_SCHEMA="<test schema>"
You may then source .env or install pytest-dotenv
to automatically set these environment variables.
Note: If you run the tests and get an
MemoryErrormentioning "write+execute memory for ffi.callback()", you probably have stale build ofcffifrom conda-forge. Remove it and install the version usingpip:conda remove cffi --force pip install cffi
License
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file dask-snowflake-0.3.4.tar.gz.
File metadata
- Download URL: dask-snowflake-0.3.4.tar.gz
- Upload date:
- Size: 13.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.12.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
0b415ebfc8bc70ab388ea0f47728d1e87559ad3db219ca79ee87e9f8060d68ef
|
|
| MD5 |
bf3dae74be61ff8ee9cd1bf4675ce3a7
|
|
| BLAKE2b-256 |
f60b131e7bd95b07970e9c43089a837f8742c1d1ee4d423e569a437fa5093a61
|
Provenance
The following attestation bundles were made for dask-snowflake-0.3.4.tar.gz:
Publisher:
wheels.yml on dask-contrib/dask-snowflake
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
dask_snowflake-0.3.4.tar.gz -
Subject digest:
0b415ebfc8bc70ab388ea0f47728d1e87559ad3db219ca79ee87e9f8060d68ef - Sigstore transparency entry: 347588712
- Sigstore integration time:
-
Permalink:
dask-contrib/dask-snowflake@d844e6bb4a2759f3340abbda94af829776b796cb -
Branch / Tag:
refs/tags/0.3.4 - Owner: https://github.com/dask-contrib
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
wheels.yml@d844e6bb4a2759f3340abbda94af829776b796cb -
Trigger Event:
push
-
Statement type:
File details
Details for the file dask_snowflake-0.3.4-py3-none-any.whl.
File metadata
- Download URL: dask_snowflake-0.3.4-py3-none-any.whl
- Upload date:
- Size: 9.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.12.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
3f4b4a2472c34962a8ad20969d5aae60910141b6dee77d1c2f8f17ee158c79a5
|
|
| MD5 |
7f0a8179cf43894fb8b419ca7508b75c
|
|
| BLAKE2b-256 |
52661ef9afc8fd0c3752c482d707e376f226c39f433f044d38c2df9fcd500156
|
Provenance
The following attestation bundles were made for dask_snowflake-0.3.4-py3-none-any.whl:
Publisher:
wheels.yml on dask-contrib/dask-snowflake
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
dask_snowflake-0.3.4-py3-none-any.whl -
Subject digest:
3f4b4a2472c34962a8ad20969d5aae60910141b6dee77d1c2f8f17ee158c79a5 - Sigstore transparency entry: 347588727
- Sigstore integration time:
-
Permalink:
dask-contrib/dask-snowflake@d844e6bb4a2759f3340abbda94af829776b796cb -
Branch / Tag:
refs/tags/0.3.4 - Owner: https://github.com/dask-contrib
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
wheels.yml@d844e6bb4a2759f3340abbda94af829776b796cb -
Trigger Event:
push
-
Statement type:
