A quick filetree uploader for the databricks filestore, local to cloud.
Project description
Databricks FileStore Uploader
A convenience tool for uploading local directories to the DBFS FileStore
Upload Local Files
This tool uploads specified source directories to specified paths in the DBFS FileStore.
Create a file adjacent to main.py
called input.yml
, and copy the following:
host : <workspace-url>
token: <user-access-token>
# THIS TOOL CREATES OR OVERWRITES THE TARGET FOLDER!
# Folder paths don't start with './'
# Folder paths don't end with '/'
payloads:
some-local-folder : some-filestore-folder
some-other-local-folder : some-other/filestore-folder
# ...
Replace <user-access-token>
with your token in databricks, and replace the example text under payloads:
one or more "source : destination" pairs.
These local source directory paths are relative to main.py
.
You can then upload all files contained in the local source folders to their specified destinations with:
./main.py
Though make sure you have PyYAML installed.
Verify Uploads
In a databricks notebook, you can verify uploads with:
display(dbutils.fs.ls("/FileStore/some-filestore-folder"))
Please Note❗
This tool CREATES or OVERWRITES it's target dbfs:/FileStore path. Only target folders that are meant to be overwritten, or don't exist.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for databricks-filestore-uploader-0.1.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 16727f608685930b8779d37ffedb280c63deb2eefaad1ed1e7b1d40b3d9cbe1d |
|
MD5 | 338c365292a97e6ec38badadcbdc718f |
|
BLAKE2b-256 | ef77c1574df8de55abfa0acfc02ca2a5941399cac27f4874e6e3ba78db46763b |
Hashes for databricks_filestore_uploader-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7750872c3e292e917bad107eeff46322d154383554b8821278ef978de3cc275e |
|
MD5 | 0923ed421a9862c25109d5a3d1145250 |
|
BLAKE2b-256 | b6fc5dd60bdd4478c2fe8ed43c9e47114d8b89747412ddb1768984f7305269c4 |