spark_datax_tools
Project description
spark_datax_tools
spark_datax_tools is a Python library that implements for dataX schemas
Installation
The code is packaged for PyPI, so that the installation consists in running:
pip install spark-datax-tools
Usage
wrapper take DataX
Nomenclature Datax
================================
table_name = "t_pmfi_lcl_suppliers_purchases"
origen = "host"
destination = "hdfs"
datax_generated_nomenclature(table_name=table_name,
origen=origen,
destination=destination,
output=True)
List of adaptaders
================================
datax_list_adapters()
Generated Ticket Adapter
============================================================
adapter_id = "ADAPTER_HDFS_OUTSTAGING"
parameter = {"uuaa":"na8z"}
datax_generated_ticket_adapter(adapter_id=adapter_id,
parameter=parameter,
is_dev=True
)
Generated Ticket Transfer
============================================================
folder="CR-PEMFIMEN-T02"
job_name="PMFITP4012"
crq="CRQ100000"
periodicity="mensual"
hour="10AM"
weight="50MB"
origen="host"
destination="hdfs"
datax_generated_ticket_transfer(
folder=folder,
job_name=job_name,
crq=crq,
periodicity=periodicity,
hour=hour,
weight=weight ,
table_name=table_name,
origen=origen,
destination=destination,
is_dev=True
)
Generated Schema JSON Artifactory
============================================================
path_json = "lclsupplierspurchases.output.schema"
is_schema_origen_in = True
schema_type = "host"
convert_string = False
datax_generated_schema_artifactory(
path_json=path_json,
is_schema_origen_in=schema_type,
schema_type=schema_type,
convert_string=convert_string
)
Generated Schema Json Datum
============================================================
spark = SparkSession.builder.master("local[*]").appName("SparkAPP").getOrCreate()
path="fields_pe_datum2.csv"
table_name="t_pmfi_lcl_suppliers_purchases"
origen="host"
destination="hdfs"
storage_zone="master"
datax_generated_schema_datum(
spark=spark,
path=path,
table_name=table_name,
origen=origen,
destination=destination,
storage_zone=storage_zone,
convert_string=False
)
License
New features v1.0
BugFix
- choco install visualcpp-build-tools
Reference
- Jonathan Quiza github.
- Jonathan Quiza RumiMLSpark.
- Jonathan Quiza linkedin.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
spark_datax_tools-0.2.0.tar.gz
(15.5 kB
view hashes)
Built Distribution
Close
Hashes for spark_datax_tools-0.2.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0bb5ba02d5f65af2c6c6ed99f8c3b8a458522220ff75e23228c88df3c63f0a8f |
|
MD5 | 2b4938f0e57835ff64f1959c3b9ae9cf |
|
BLAKE2b-256 | d7ad01c3bb274152705e60ea6aeda2a1d563dab32cb350b182443f48d498d070 |