delta_table_utils
Delta table utilities.
The basic use case for this library is if you are working in Databricks and want to do upserts using AutoLoader.
Basic usage:
from delta_table.delta_table_utils import DeltaTableColumn, DeltaTable
schema_name = 'my_schema'
table_name = 'my_table'
# Define the delta table schema
column_list = [
DeltaTableColumn('id', data_type='STRING', nulls_allowed=False, is_unique_id=True),
DeltaTableColumn('col1', data_type='STRING', nulls_allowed=False),
DeltaTableColumn('col2', data_type='DOUBLE'),
DeltaTableColumn('col3', data_type='DOUBLE'),
DeltaTableColumn('col4', data_type='DOUBLE'),
DeltaTableColumn('created_at', data_type='TIMESTAMP'),
DeltaTableColumn('updated_at', data_type='TIMESTAMP')
]
# Create the DeltaTable object
delta_table = DeltaTable(schema_name=schema_name, table_name=table_name, upload_path="<location_of_data_in_s3>", column_list=column_list)
# Create the table and start the stream
delta_table.create_if_not_exists(sqlContext)
delta_table.stream(spark, cloudFiles_format='csv')
Additional notes
By default, when you use the stream method in this library, it stops as soon as no new data is detected. This is useful if you don't want a cluster running all the time and rather you just want to update your delta tables on some sort of a schedule.
1 maintainer
Metadata
1 maintainer
Release files for delta-table-utils 0.0.17
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| delta_table_utils-0.0.17.tar.gz | 7.2 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| delta_table_utils-0.0.17-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 14.8 kB
Release files / delta_table_utils-0.0.17.tar.gz
| Download URL | delta_table_utils-0.0.17.tar.gz |
|---|---|
| Size | 7.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7ee9066f72b9c097bce2cea76bcec04a4893cb6d6e971d1adb198ab44a9fd01c
|
|
BLAKE2b-256 checksum How to use checksums |
c1263f5c0aaccdf171bce44c8274ba322490089dabf7c54839ce568b75a91a73
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.8.15
|
Release files / delta_table_utils-0.0.17-py3-none-any.whl
| Download URL | delta_table_utils-0.0.17-py3-none-any.whl |
|---|---|
| Size | 7.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
02265ace80496ed2d6471161d69d853133db2549ddba3cd059bf76aa6c231151
|
|
BLAKE2b-256 checksum How to use checksums |
5b30c91c019bd6013957c14fc8d761a24498075a6805377456bde85f407e777d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.8.15
|
