Airless
Airless is a package that aims to build a serverless and lightweight orchestration platform, creating workflows of multiple tasks being executed on Google Cloud Functions
Why not just use Apache Airflow?
Airflow is the industry standard when we talk about job orchestration and worflow management. However, in some cases, we believe it may not be the best solution. I would like to highlight 3 main cases we face that Airflow struggles to handle.
- Serverless
At the beginning of a project we want to avoid dealing with infrastructure since it demands time and it has a fixed cost to reserve an instance to run Airflow. Since we didn't have that many jobs, it didn't make sense to have an instance of Airflow up 24-7.
When the project starts to get bigger and, if we use Airflow's instance to run the tasks, we start facing performance issues on the workflow.
In order to avoid this problems we decided to build a 100% serverless platform.
- Parallel processing
The main use case we designed Airless for is for data scrappers. The problem with data scrappers is that normally you want them to process a lot of tasks in parallel, for instance, first you want to fetch a website and collect all links in that page and send them forward for another task to be executed and then that task does the same and so on and so forth.
Building this workflow that does not know before hand how many tasks are going to be executed is something hard be built on Airflow.
- Data sharing between tasks
In order to built this massive parallel processing workflow that we explained on the previous topic, we need to be able to dynamically create and send data to the next task. So use the data from the first task as a trigger and an input data for the next tasks.
How it works
Airless builts its workflows based on Google Cloud Functions, Google Pub/Sub and Google Cloud Scheduler.
- Everything starts with the Cloud Scheduler, which is a serverless product from Google Cloud that is able to publish a message to a Pub/Sub with a cron scheduler
- When a message is published to a Pub/Sub it can trigger a Cloud Function and get executed with that message as an input
- This Cloud Functions is able to publish as many messages as it wants to as many Pub/Sub topics as it wants
- Repeat from 2
Preparation
Environment variables
ENVGCP_PROJECTPUBSUB_TOPIC_ERRORLOG_LEVELPUBSUB_TOPIC_EMAIL_SENDPUBSUB_TOPIC_SLACK_SENDBIGQUERY_DATASET_ERRORBIGQUERY_TABLE_ERROREMAIL_SENDER_ERROREMAIL_RECIPIENTS_ERRORSLACK_CHANNELS_ERROR
2 maintainers
Release files for airless 0.0.73
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| airless-0.0.73.tar.gz | 22.4 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| airless-0.0.73-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 51.8 kB
Release files / airless-0.0.73.tar.gz
| Download URL | airless-0.0.73.tar.gz |
|---|---|
| Size | 22.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
330046c05b81d359344871178e0a9c50ce5e0227ca55717015833ebc70c66217
|
|
BLAKE2b-256 checksum How to use checksums |
54c7a003b516882a54aab4983c9e5dec757803638ff2311f8f902e41acc71bd8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.0 CPython/3.12.5
|
Release files / airless-0.0.73-py3-none-any.whl
| Download URL | airless-0.0.73-py3-none-any.whl |
|---|---|
| Size | 29.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
5d0827430672a5331bb4e756ec741c2b3986b9915133aeb7e8d880f88cd0c562
|
|
BLAKE2b-256 checksum How to use checksums |
a1fdeffbcddd5f76b88e68a17677d8963c18b3ed5f83d1893ab7aef56c58bb2b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.0 CPython/3.12.5
|
