Run spark task in k8s
Project description
###spark-helper
####在k8s集群中调用spark (https://github.com/GoogleCloudPlatform/spark-on-k8s-operator)
Author: yuchao.li@liangdao.ai Created: 2023.3.9
Usage:
s3_config = {'endpoint': 'http://minio.minio-tenant',
'access': 'xxx',
'secret': 'xxx'}
set_config(image='pyspark:driver-3.3.1',
namespace='spark-operator',
service_account='sparkoperator',
s3_config = s3_config )
start_spark('hello-world-spark',
's3a://airflow-dags/spark/pyspark_helloworld.py',
False)
####配置优先级: 从高到低 set_config 设置的配置 auto_load_config 自动加载的环境变量配置
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for k8s-spark-helper-liangdao_data-0.0.3.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | c7ed29a1ba093cba899ef7a03bbd6965551dad22f0c0d6ef569632136ebbf8fd |
|
MD5 | 3f2dd277a3123640ddf2f55a7d57860c |
|
BLAKE2b-256 | 223a2c3ed325a7564f6f0d8e3ac2609e23d3d2ed470fb36b1eee64cf3fc8bc8b |
Close
Hashes for k8s_spark_helper_liangdao_data-0.0.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2354478d9db4d1aa3a4030ed180c752b0a7214cfd8acb12ba48308bc73a1eb62 |
|
MD5 | 9277b2b359757aa7954878512d74bceb |
|
BLAKE2b-256 | 622e4e0cf6cbafa805d5c4fdb0cf0d66733e887a7b029ac67a25f263f40dc491 |