spark_datax_tools
Project description
spark_datax_tools
spark_datax_tools is a Python library that implements for dataX schemas
Installation
The code is packaged for PyPI, so that the installation consists in running:
pip install spark-datax-tools
Usage
wrapper take DataX
Nomenclature Datax
================================
table_name = "t_pmfi_lcl_suppliers_purchases"
origen = "host"
destination = "hdfs"
datax_generated_nomenclature(table_name=table_name,
origen=origen,
destination=destination,
output=True)
List of adaptaders
================================
datax_list_adapters()
Generated Ticket Adapter
============================================================
adapter_id = "ADAPTER_HDFS_OUTSTAGING"
parameter = {"uuaa":"na8z"}
datax_generated_ticket_adapter(adapter_id=adapter_id,
parameter=parameter,
is_dev=True
)
Generated Ticket Transfer
============================================================
folder="CR-PEMFIMEN-T02"
job_name="PMFITP4012"
crq="CRQ100000"
periodicity="mensual"
hour="10AM"
weight="50MB"
origen="host"
destination="hdfs"
datax_generated_ticket_transfer(
folder=folder,
job_name=job_name,
crq=crq,
periodicity=periodicity,
hour=hour,
weight=weight ,
table_name=table_name,
origen=origen,
destination=destination,
is_dev=True
)
Generated Schema JSON Artifactory
============================================================
path_json = "lclsupplierspurchases.output.schema"
is_schema_origen_in = True
schema_type = "host"
convert_string = False
datax_generated_schema_artifactory(
path_json=path_json,
is_schema_origen_in=schema_type,
schema_type=schema_type,
convert_string=convert_string
)
Generated Schema Json Datum
============================================================
spark = SparkSession.builder.master("local[*]").appName("SparkAPP").getOrCreate()
path="fields_pe_datum2.csv"
table_name="t_pmfi_lcl_suppliers_purchases"
origen="host"
destination="hdfs"
storage_zone="master"
datax_generated_schema_datum(
spark=spark,
path=path,
table_name=table_name,
origen=origen,
destination=destination,
storage_zone=storage_zone,
convert_string=False
)
License
New features v1.0
BugFix
- choco install visualcpp-build-tools
Reference
- Jonathan Quiza github.
- Jonathan Quiza RumiMLSpark.
- Jonathan Quiza linkedin.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
spark_datax_tools-0.6.8.tar.gz
(14.9 kB
view hashes)
Built Distribution
Close
Hashes for spark_datax_tools-0.6.8-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5191815aa18bb2e6a7165a1b86abb470a39a867cdbf458e11533865cd7893d2f |
|
MD5 | 09f9a4599bac5e42165517aaada0b26e |
|
BLAKE2b-256 | bb057fd1b95540e8a43e10a086b4e128a0d4f2ef3b0ccd876e5a50de53c22dec |