Skip to main content

spark_hdfs_tools

Project description

spark_hdfs_tools

Github License Updates Python 3 Code coverage

spark_hdfs_tools is a Python library that implements hdfs filesystem in sandbox

Installation

The code is packaged for PyPI, so that the installation consists in running:

Usage

wrapper run hdfs filesystem

Sandbox

Installation

!yes| pip uninstall spark-hdfs-tools
pip install spark-hdfs-tools --user --upgrade

IMPORTS

import os
import pyspark
from spark_hdfs_tools import dq_path_workspace
from spark_hdfs_tools import dq_download_jar
from spark_hdfs_tools import dq_spark_session

Variables

project_sda="SDA_37036"
url_conf = "http://artifactory-gdt.central-02.nextgen.igrupobbva/artifactory/gl-datio-spark-libs-maven-local/com/datiobd/cdd-hammurabi/4.0.9/DQ_LOCAL_CONFS/KCOG/KCOG_branch_MRField.conf"

Creating Workspace

dq_path_workspace(project_sda=project_sda)

Download haas jar

dq_download_jar(haas_version="4.8.0", force=True)

Spark Session

spark, sc = dq_spark_session()

Validate Conf

dq_validate_conf(url_conf=url_conf)

Extract Params

dq_extract_parameters(url_conf=url_conf)

Run

dq_run_sandbox(spark=spark,
               sc=sc,
               parameter_conf_list=parameter_conf_list,
               url_conf=url_conf)

License

Apache License 2.0.

New features v1.0

BugFix

  • choco install visualcpp-build-tools

Reference

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

spark_hdfs_tools-0.1.1.tar.gz (7.4 kB view hashes)

Uploaded Source

Built Distribution

spark_hdfs_tools-0.1.1-py3-none-any.whl (6.9 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page