A plugin for pipen to handle file metadata in Google Cloud Storage
Project description
pipen-gcs
A plugin for pipen to handle files in Google Cloud Storage
Installation
pip install -U pipen-gcs
# uninstall to disable
pip uninstall pipen-gcs
Usage
from pipen import Proc, Pipen
class MyProc(Proc):
input = "infile:file"
input_data = ["gs://bucket/path/to/file"]
output = "outfile:file:gs://bucket/path/to/output"
script = "cat {{infile}} > {{outfile}}"
class MyPipen(Pipen):
starts = MyProc
# input files/directories will be downloaded to /tmp
# output files/directories will be generated in /tmp and then uploaded
# to the cloud storage
plugin_opts = {"gcs_localize": "/tmp"}
if __name__ == "__main__":
MyPipen().run()
You can also disable localization, then you will have to handle the cloud storage files yourself.
from pipen import Proc, Pipen
class MyProc(Proc):
input = "infile:file"
input_data = ["gs://bucket/path/to/file"]
output = "outfile:file:gs://bucket/path/to/output"
script = "gsutil cp {{infile}} {{outfile}}"
class MyPipen(Pipen):
starts = MyProc
plugin_opts = {"gcs_localize": False}
if __name__ == "__main__":
MyPipen().run()
Configuration
gcs_localize
: The directory to localize the cloud storage files. If set toFalse
, the files will not be localized. Default isFalse
.gcs_credentials
: The path to the Google Cloud Service Account credentials file.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pipen_gcs-0.0.1.tar.gz
(9.0 kB
view hashes)
Built Distribution
Close
Hashes for pipen_gcs-0.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5be333bd6cd77b9a69c22d25ee6ed8f83a87f8d2146c643c3f1adb85c79d1ed7 |
|
MD5 | b8f5662f314570a2f9b8f129351df340 |
|
BLAKE2b-256 | fcf6d382843189998ffe0de1ae3a134682e6ac3aba153d43fa8a9e3cea1bfefa |