Snowflake Spark Connect Python Client
pip install snowflake-spark-connect
Supports Python 3.10, 3.11, and 3.12.
Usage
from snowflake.sparkconnect import SparkSession
spark = SparkSession.builder.getOrCreate()
spark.sql("SELECT 1").show()
spark.stop()
getOrCreate() starts a Spark Connect session in Snowflake, returning once the session is ready for us.
Builder options
Authentication
Use exactly one of the following authentication options when creating a session to customize the connection.
| Method | What it does |
|---|---|
| (nothing) | getOrCreate() returns the default connection in the environment |
.snowpark_session(session) |
Use an existing Snowpark session object |
.connection("prod") |
Named profile from ~/.snowflake/connections.toml |
.remote(account=..., ...) |
Snowflake connection parameters (see below) |
Once a Spark Connect session is established, token refresh happens in the background to ensure the session stays authenticated.
Session lifetime
| Option | Meaning |
|---|---|
.idlettl(timedelta(...)) |
Stop the Spark server after this much idle time |
.sessionttl(timedelta(...)) |
Hard cap on total session lifetime |
.tokenttl(timedelta(...)) |
Lifetime of each Spark Connect JWT used for authentication |
spark_idlettl / spark_sessionttl / spark_tokenttl inside .remote(...) |
Same TTLs, passed as Snowflake kwargs |
Database and schema
Spark Connect sessions must be started with a database and schema. .database(name) and .schema(name) on the builder can be used to specify a database and schema combination to use; an exception is raised if only one is provided. When neither is set, the user's personal database USER$<current user>.PUBLIC is used by default.
| Option | Meaning |
|---|---|
.database(name) |
Use this Snowflake database for the Spark Connect server |
.schema(name) |
Use this Snowflake schema for the Spark Connect server |
Explicit .database(...) and .schema(...) values always override database/schema from a connection, configuration, or supplied/active Snowpark session.
Artifact repository and Python packages
These are sent on START_SPARK_CONNECT_SERVER and installed on the Connect driver and on pushdown UDFs. They are omitted when unset. Packages require an artifact repository.
| Option | Meaning |
|---|---|
.artifact_repository(name) |
One Artifact Repository name, bound as ARTIFACT_REPOSITORY |
.python_packages([...]) |
Pip specs, bound as PYTHON_PACKAGES |
spark_artifact_repository / spark_python_packages inside .remote(...) |
Same values, passed as Snowflake kwargs |
Attaching to an existing sc:// URL rejects them, because attach does not START a server. getOrCreate() also rejects them when a session already exists. Leave both unset to reuse that session.
.remote(**kwargs) Snowflake parameters
These are the same parameters as snowpark-python.
Spark-specific kwargs:
| Parameter | Notes |
|---|---|
spark_idlettl / spark_sessionttl / spark_tokenttl |
Same as .idlettl() / .sessionttl() / .tokenttl() |
spark_artifact_repository / spark_python_packages |
Same as .artifact_repository() / .python_packages() |
3 maintainers
Metadata
3 maintainers
Release files for snowflake-spark-connect 0.1.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| snowflake_spark_connect-0.1.2.tar.gz | 1.9 MB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| snowflake_spark_connect-0.1.2-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 4.4 MB
Release files / snowflake_spark_connect-0.1.2.tar.gz
| Download URL | snowflake_spark_connect-0.1.2.tar.gz |
|---|---|
| Size | 1.9 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
e6548c9044964195b144ce54467f16f35b134544d533dae0dcc9294695fa9f78
|
|
BLAKE2b-256 checksum How to use checksums |
a707783201567d4ee5068459508916032aab2fc5077229ba45ed0dcabe1adb85
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.16
|
Release files / snowflake_spark_connect-0.1.2-py3-none-any.whl
| Download URL | snowflake_spark_connect-0.1.2-py3-none-any.whl |
|---|---|
| Size | 2.5 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b132cf72a9a136dd21a65b2153142889cccc74721393f4c7eb9ca8b8f37dab47
|
|
BLAKE2b-256 checksum How to use checksums |
70cc01e63d8da34017bd8f867926d74f6a0e6065e9392a6d401bb6ffe641d9e4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.16
|
Security for snowflake-spark-connect 0.1.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
