Deploy configurable unix pipeline jobs serverlessly.
Project description
handoff.cloud
Deploy configurable unix pipeline jobs serverlessly.
NEW: Monitor metrics with Grafana
What is it?
handoff (repository) is a framework for executing unix-pipeline processes serverlessly. It separates the code and run-time configurations, making the deployment faster, more flexible, and more secure.
handoff helps the teams who want to:
- Automate the ETL/ELT process (earthquake data ELT example)
- Massively web-crawl and collect data (market & education data example)
- Relieve the pain of managing Apache Airflow Worker nodes.
...and more!
handoff is an open-source (APLv2) project sponsored by ANELEN.
How it works
handoff was originally designed to deploy a single-line ETL process like singer.io on AWS Fargate, and it was extended to run any commands and program.
In handoff.cloud framework, the configurations are stored and retrieved from a secure parameter store such as AWS Systems Manager Parameter Store. This avoids storing sensitive information or configurations that changes frequently on the Docker image.
Supporting files for commands can also be fetched from a cloud storage such as AWS S3. The artifacts, the files produced by each command can also be stored in the cloud storage.
This repository also contains AWS Cloudformation templates and deployment scripts to support the workflow from local development to the production deployment.
Get started easily
Check out the super quick example
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for handoff-0.2.0b4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 773654d6cebe4372ca22e71cefec6851ed6aa9e38b4ab29012e3dd6202828e30 |
|
MD5 | 68956f0f9cd05e3dbf8125a45abe01b2 |
|
BLAKE2b-256 | 49f1f2bce038a4df949c7fe36275a4e6e387e7ca05431ad71e29535ee4e2ef4f |