Connect Pyspark to remote clusters
Project description
Pypsark Gateway is a library to seamlessly connect to remote spark clusters.
Quick Start
Install the pysparkgateway package on both the remote Spark cluster you are connecting to and the local machine.
pip install pysparkgateway
Start the Pyspark Gateway server on the cluster.
pyspark-gateway start
Pyspark Gateway communicates over 3 ports, 25000, 25001, 25002. Currently the client only supports connecting to these ports on localhost so you’ll need to tunnel them.
ssh myuser@foo.bar.cluster.com -L 25000:localhost:25000 -L 25001:localhost:25001 -L 25002:localhost:25002
Now you’re ready to connect. The main thing to keep in mind is the Pyspark Gateway import needs to come before any other import. Pypsark Gateway needs to patch your local pyspark in order to function properly.
The way that your local Python connects to the remote cluster is via a custom py4j gateway. Pyspark Gateway will create and configure automatically, you just need to pass it into the SparkContext options.
Also to enable all pyspark functions to work, spark.io.encryption.enabled needs to be set to true.
# This import comes first! from pyspark_gateway import PysparkGateway pg = PysparkGateway() from pyspark import SparkContext, SparkConf conf = conf.set('spark.io.encryption.enabled', 'true') sc = SparkContext(gateway=pg.gateway, conf=conf)
Now you have a working spark context connected to a remote cluster.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
File details
Details for the file PysparkGateway-0.0.14.tar.gz
.
File metadata
- Download URL: PysparkGateway-0.0.14.tar.gz
- Upload date:
- Size: 9.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.22.0 setuptools/41.0.0 requests-toolbelt/0.8.0 tqdm/4.26.0 CPython/2.7.13
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6338d5ba174f95b4a77c47a8721c38c4f3e10abcc2364538fbfc0d9114491958 |
|
MD5 | 72e6f5b423680de2afd41f4c21530e8b |
|
BLAKE2b-256 | f157b7e4b51c22b1cb2d2d3e8185f8410f07452fcd7c9a4e5b9291aaf8e6f972 |