Spark Connect adapter for Ocean Spark
Project description
ocean-spark-connect
Wrapper to create Spark Connect session for Spark Applications in Ocean
from ocean_spark_connect.ocean_spark_session import OceanSparkSession
spark = OceanSparkSession.Builder().cluster_id("osc-cluster").appid("appid").profile("default").getOrCreate()
spark.sql("select random()").show()
spark.stop()
To use periodic ping to keep the session alive, use the ping_interval option (in seconds). The default value is off (-1).
spark = OceanSparkSession.Builder() \
.ping_interval(5.0) \
.cluster_id("osc-cluster") \
.appid("appid") \
.profile("default") \
.getOrCreate()
To use java Spark plugin for the websocket bridge instead, add the use_java(True) option.
Options for OceanSparkSession.Builder with and without default values
In addition to the existing SparkSession.Builder option, the following options are available:
- token
- profile
- appid
- account_id
- cluster_id
- host = "api.spotinst.io"
- port = "15002"
- bind_address = "0.0.0.0"
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for ocean_spark_connect-0.3.7.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 754e4457e29431bd45f4ffff50f432e283f193f5166ebaf37dde1004924771d8 |
|
MD5 | 904770814d5d159db4b676c605cfefa1 |
|
BLAKE2b-256 | ab9f8234b41d9b0ea3051a3af65181c9b45ec4cd7e8a97b7f9b553d0fad44cb5 |
Close
Hashes for ocean_spark_connect-0.3.7-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 685054da4fd2b9d805b794d16fae0e7e7b0e288fc050d08d7f8476a8d659e6ea |
|
MD5 | 476fb683eec88114cee0bc56a9efe922 |
|
BLAKE2b-256 | 2129c9c0de9cbdf84bed3a1f4fa054eb5c7c6365a15187048a953e4eb6312f66 |