Build Python interfaces to the AWS Glue ETL library for use as a local dependency.
Note that this package must be used in conjunction with the AWS Glue service and is not executable independently. Many of the classes and methods use the Py4J library to interface with code that is available on the Glue platform. This repository can be used as a reference and aid for writing Glue scripts.
While scripts using this library can only be run on the AWS Glue service, it is possible to import this library locally. This may be helpful to provide auto-completion in an IDE, for instance. To import the library successfully you will need to install PySpark, which can be done using pip:
pip install pyspark
This package contains Python interfaces to the key data structures and methods used in AWS Glue. The following are some important modules. More information can be found in the public documentation.
The file context.py contains the GlueContext class. GlueContext extends PySpark's SQLContext class to provide Glue-specific operations. Most Glue programs will start by instantiating a GlueContext and using it to construct a DynamicFrame.
The DynamicFrame, defined in dynamicframe.py, is the core data structure used in Glue scripts. DynamicFrames are similar to Spark SQL's DataFrames in that they represent distributed collections of data records, but DynamicFrames provide more flexible handling of data sets with inconsistent schemas. By representing records in a self-describing way, they can be used without specifying a schema up front or requiring a costly schema inference step.
DynamicFrames support many operations, but it is also possible to convert them to DataFrames using the
toDF method to make use of existing Spark SQL operations.
The transforms directory contains a variety of operations that can be performed on DynamicFrames. These include simple operations, such as
DropFields, as well as more complex transformations like
Relationalize, which flattens a nested data set into a collection of tables that can be loaded into a Relational Database. Once imported, transforms can be invoked using the following syntax:
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Hashes for awsglue_local-0.9.1-py2.py3-none-any.whl