Run spark task in k8s
Project description
###spark-helper
####在k8s集群中调用spark (https://github.com/GoogleCloudPlatform/spark-on-k8s-operator)
Author: yuchao.li@liangdao.ai Created: 2023.3.9
Usage:
s3_config = {'endpoint': 'http://minio.minio-tenant',
'access': 'xxx',
'secret': 'xxx'}
set_config(image='pyspark:driver-3.3.1',
namespace='spark-operator',
service_account='sparkoperator',
s3_config = s3_config )
start_spark('hello-world-spark',
's3a://airflow-dags/spark/pyspark_helloworld.py',
False)
####配置优先级： 从高到低 set_config 设置的配置 auto_load_config 自动加载的环境变量配置
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file k8s-spark-helper-liangdao_data-0.0.3.tar.gz.
File metadata
- Download URL: k8s-spark-helper-liangdao_data-0.0.3.tar.gz
- Upload date:
- Size: 4.4 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.4.2 requests/2.22.0 setuptools/45.2.0 requests-toolbelt/0.8.0 tqdm/4.30.0 CPython/3.8.10
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | c7ed29a1ba093cba899ef7a03bbd6965551dad22f0c0d6ef569632136ebbf8fd |
|
| MD5 | 3f2dd277a3123640ddf2f55a7d57860c |
|
| BLAKE2b-256 | 223a2c3ed325a7564f6f0d8e3ac2609e23d3d2ed470fb36b1eee64cf3fc8bc8b |
File details
Details for the file k8s_spark_helper_liangdao_data-0.0.3-py3-none-any.whl.
File metadata
- Download URL: k8s_spark_helper_liangdao_data-0.0.3-py3-none-any.whl
- Upload date:
- Size: 8.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.4.2 requests/2.22.0 setuptools/45.2.0 requests-toolbelt/0.8.0 tqdm/4.30.0 CPython/3.8.10
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 2354478d9db4d1aa3a4030ed180c752b0a7214cfd8acb12ba48308bc73a1eb62 |
|
| MD5 | 9277b2b359757aa7954878512d74bceb |
|
| BLAKE2b-256 | 622e4e0cf6cbafa805d5c4fdb0cf0d66733e887a7b029ac67a25f263f40dc491 |
