No project description provided
Project description
Welcome to Bucket Brigade
Bucket Brigade contains helper functions for using cloud storage as a simple data processing queue
Cloud providers such as AWS, GCP and Azure have excellent object/blob storage capabilities. Once data is in cloud storage, it is as safe from accidental deletion or corruption as anywhere but the best self-hosted solutions.
Everything gets more complicated and more expensive once you start using the cloud providers to do more sophisticated activities such as hosting your databases and running your server instances.
This library is aimed at making it easy for developers to store their data as objects in cloud storage but use other tools for processing the data - either self-hosted or hosted in the cloud.
The key function is get_unprocessed_docs which lists unprocessed documents in a cloud storage bucket and sends them to a queue for serverless functions to process. The initial target for the bucketbrigade is modal.com but the intent is to accommodate any serverless function provider.
Key function:
- get_unprocessed_docs
Supporting functions:
- list_docs
- save_doc
- read_doc
- mark_completed
- skip_doc
This library has started with storage on AWS but will expand soon to GCP, Azure and others.
Thanks to https://github.com/uhjish for donating the name bucketbrigade!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for bucketbrigade-0.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a293b11e4e15f17629fe5c209d3d3958b7146a686759b81a20831e3969025edf |
|
MD5 | 1048051f1746e7bb7c0f20e88c2428f4 |
|
BLAKE2b-256 | 27cf16435c051731d23592edaa480e8f01784d46a27781f628e6065383c94699 |