Skip to main content

Spark Connect adapter for Ocean Spark

Project description

ocean-spark-connect

Wrapper to create Spark Connect session for Spark Applications in Ocean

from ocean_spark_connect.ocean_spark_session import OceanSparkSession

spark = OceanSparkSession.Builder().cluster_id("osc-cluster").appid("appid").profile("default").getOrCreate()
spark.sql("select random()").show()
spark.stop()

To use periodic ping to keep the session alive, use the ping_interval option (in seconds). The default value is off (-1).

spark = OceanSparkSession.Builder() \
    .ping_interval(5.0) \
    .cluster_id("osc-cluster") \
    .appid("appid") \
    .profile("default") \
    .getOrCreate()

To use java Spark plugin for the websocket bridge instead, add the use_java(True) option.

Options for OceanSparkSession.Builder with and without default values

In addition to the existing SparkSession.Builder option, the following options are available:

  • token
  • profile
  • appid
  • account_id
  • cluster_id
  • host = "api.spotinst.io"
  • port = "15002"
  • bind_address = "0.0.0.0"

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ocean_spark_connect-0.3.7.tar.gz (6.3 kB view details)

Uploaded Source

Built Distribution

ocean_spark_connect-0.3.7-py3-none-any.whl (7.5 kB view details)

Uploaded Python 3

File details

Details for the file ocean_spark_connect-0.3.7.tar.gz.

File metadata

  • Download URL: ocean_spark_connect-0.3.7.tar.gz
  • Upload date:
  • Size: 6.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.0.0 CPython/3.12.3

File hashes

Hashes for ocean_spark_connect-0.3.7.tar.gz
Algorithm Hash digest
SHA256 754e4457e29431bd45f4ffff50f432e283f193f5166ebaf37dde1004924771d8
MD5 904770814d5d159db4b676c605cfefa1
BLAKE2b-256 ab9f8234b41d9b0ea3051a3af65181c9b45ec4cd7e8a97b7f9b553d0fad44cb5

See more details on using hashes here.

File details

Details for the file ocean_spark_connect-0.3.7-py3-none-any.whl.

File metadata

File hashes

Hashes for ocean_spark_connect-0.3.7-py3-none-any.whl
Algorithm Hash digest
SHA256 685054da4fd2b9d805b794d16fae0e7e7b0e288fc050d08d7f8476a8d659e6ea
MD5 476fb683eec88114cee0bc56a9efe922
BLAKE2b-256 2129c9c0de9cbdf84bed3a1f4fa054eb5c7c6365a15187048a953e4eb6312f66

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page