sras-client

Python client for Snowflake Runtime for Apache Spark.

pip install snowflake-spark-connect

Usage

from datetime import timedelta

from snowflake.sparkconnect import SparkSession

spark = (
    SparkSession.builder.idlettl(timedelta(minutes=30))
    .sessionttl(timedelta(hours=4))
    .getOrCreate()
)

spark.sql("SELECT 1").show()
spark.stop()

getOrCreate() reuses the process session when the builder has no credentials. A later call with different credentials, TTLs, or database/schema settings raises instead of sharing another identity, including TTL-only builders such as .idlettl(...). create() starts a new Spark Connect server with a new session in the current process.

In a Snowflake Notebook, get_active_session() binds to the active Snowpark session. Pass optional named database and schema arguments to set that context before the Spark Connect server starts.

Authentication options

Use one of the following authentication options when creating a session. Pick one. .connection("prod") uses that toml profile as-is (not mixed with SNOWFLAKE_*).

Do not mix .connection("prod"), .remote(...), and .snowpark_session(...).

Session lifetime

These apply only when the builder STARTs a server. They cannot be combined with a sc:// URL.

Database and schema

Explicit .database(...) / .schema(...) values always override database/schema from a connection, configuration, or supplied/active Snowpark session. They apply only when the builder STARTs a server and cannot be combined with a sc:// URL.

The snowflake.sparkconnect.SparkSession refreshes its token in the background to automatically stay authenticated.

.remote(**kwargs) Snowflake parameters

These are the same parameters as snowpark-python.

Spark-specific kwargs:

Environment variables

When the builder has no .connection() / .remote(), Snowflake SNOWFLAKE_* login vars are used (including SNOWFLAKE_HOST), then SNOWFLAKE_CONNECTION_NAME, then the default toml profile, then an active Notebook session. Blank env values are ignored.