Skip to main content

Spark Connect adapter for Ocean Spark

Project description

ocean-spark-connect

Wrapper to create Spark Connect session for Spark Applications in Ocean

from ocean_spark_connect.ocean_spark_session import OceanSparkSession

spark = OceanSparkSession.Builder().cluster_id("osc-cluster").appid("appid").profile("default").getOrCreate()
spark.sql("select random()").show()
spark.stop()

To use periodic ping to keep the session alive, use the ping_interval option (in seconds). The default value is off (-1).

spark = OceanSparkSession.Builder() \
    .ping_interval(5.0) \
    .cluster_id("osc-cluster") \
    .appid("appid") \
    .profile("default") \
    .getOrCreate()

To use java Spark plugin for the websocket bridge instead, add the use_java(True) option.

Options for OceanSparkSession.Builder with and without default values

In addition to the existing SparkSession.Builder option, the following options are available:

  • token
  • profile
  • appid
  • account_id
  • cluster_id
  • host = "api.spotinst.io"
  • port = "15002"
  • bind_address = "0.0.0.0"

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ocean_spark_connect-0.3.7.tar.gz (6.3 kB view hashes)

Uploaded Source

Built Distribution

ocean_spark_connect-0.3.7-py3-none-any.whl (7.5 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page