DF to Azure
Python module for fast upload of pandas DataFrame to azure using automatic created pipelines in Azure Data Factory.
Introduction
The purpose of this project is to upload large datasets using Azure Data Factory combined with a Azure SQL Server. In steps the following process kicks off:
1. The data will be uploaded as a .csv file to Azure Blob storage.
2. A SQL Database table is prepared based on the parameters in the settings/yml/adf_settings.yml
3. A pipeline is created for uploading the .csv from the Blob storage into the SQL table.
4. The pipeline is triggered, so that the .csv file is bulk inserted into the SQL table.
How it works
Based on the following attributes, it is possible to bulk insert your dataframe into the SQL Database:
from df_to_azure import df_to_azure
df_to_azure(df=df, tablename="table_name", schema="schema", method="create", id_field="col_a")
df: dataframe you wish to exporttablename: desired name of the tableschema: desired sql schemamethod: option for "create" "append" or "upsert"id_field: id field of the table. Necessary if 4 is set to True
Upsert / create or append
It is possible to upsert the SQL table with (new) records, if present in the dataframe you want to upload. Based on the id_field, the SQL table is being checked on overlapping values. If there are new records, the "old" records will be updated in the SQL table. The new records will be uploaded and appended to the current SQL table.
Settings
To use this module, you need to add the azure subscriptions settings and azure data factory settings to your environment variables.
We recommend to work with .env files and load them in during runtime. But this is optional and they can be set as system variables as well.
Use the following template when using .env
# --- ADF SETTINGS ---
# general run settings
create="False"
# data factory settings
rg_name : ""
rg_location: "westeurope"
df_name : ""
# blob settings
ls_blob_name : "Python Blob Linked Service"
ls_blob_account_name : ""
ls_blob_container_name : ""
ls_blob_account_key : ""
# SQL settings
ls_sql_name : "Python SQL Linked Service"
ls_sql_server_name: ""
ls_sql_database_name: ""
ls_sql_database_user: ""
ls_sql_database_password: ""
# pipeline settings
trigger : True
# --- AZURE SETTINGS ---
# azure credentials for connecting to azure subscription.
client_id : ""
secret : ""
tenant : ""
subscription_id : ""
Maintained by Zypp:
Testing
To run the test suite, use:
pytest df_to_azure
To run pytest for a single test:
pytest df_to_azure/tests/test_df_to_azure::test_duplicate_keys_upsert
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file df_to_azure-0.1.0.tar.gz.
File metadata
- Download URL: df_to_azure-0.1.0.tar.gz
- Upload date:
- Size: 13.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/3.3.0 pkginfo/1.6.1 requests/2.25.0 setuptools/51.0.0 requests-toolbelt/0.9.1 tqdm/4.55.0 CPython/3.8.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
242ebdca8dd21699737e0094dd356f43f35889522be8326dd7aa55b62627ed01
|
|
| MD5 |
7973256d1891673387b4e090991c7d89
|
|
| BLAKE2b-256 |
65c1ac33f43062cafb160f303c7fe602f05a4ec16d1e29da37ed31b3cdf36c43
|
File details
Details for the file df_to_azure-0.1.0-py3-none-any.whl.
File metadata
- Download URL: df_to_azure-0.1.0-py3-none-any.whl
- Upload date:
- Size: 17.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/3.3.0 pkginfo/1.6.1 requests/2.25.0 setuptools/51.0.0 requests-toolbelt/0.9.1 tqdm/4.55.0 CPython/3.8.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
d23555b629b527b171855724f1613d21adebd08794ba34d61b382d0369f85634
|
|
| MD5 |
03a4d5ea128a37114730e578107cc963
|
|
| BLAKE2b-256 |
472e6b98a7a1d08856981830b1bb40758b3cc56f92e7b0e56ab8ba71df2daa29
|
