Create asynchronous data pipelines and deploy to cloud or airflow
Project description
Discord :sunglasses: |
Forum :wave: |
Installation :floppy_disk: |
Documentation :notebook:
️
️
Elegant YAML DAGS for Data Pipelines
Deploy to your existing Airflow.
️
Why Typhoon? |
Key Features |
Example YAML |
Installation
️
Why Typhoon (+ Airflow)?
Airflow is great!
Typhoon lets you write Airflow DAGS faster :rocket::
**Workflow**: Typhoon YAML DAG --> Typhoon build --> Airflow DAG
Simplicity and re-usability; a toolkit designed to be loved by Data Engineers :heart:
Key features
Elegant - YAML; low-code and easy to learn. Code-completion - Fast to compose. (VS Code recommended). Data sharing - data flows between tasks making it super intuitive. Composability - Functions and connections combine like Lego. |
|
Components - reduce complex tasks to 1 re-usable tasks Packaged examples:
UI: Share pre-built components (data pipelines) with your team :raised_hands: |
|
Rich CLI & Shell: Inspired by others; instantly familiar. Testable Tasks - automate DAG task tests. Testable Python - test functions or full DAGs with PyTest. |
Example YAML DAG
name: favorite_authors
schedule_interval: rate(1 day)
tasks:
choose_favorites:
function: typhoon.flow_control.branch
args:
branches:
- J. K. Rowling
- George R. R. Martin
- James Clavell
get_author:
input: choose_favorites
function: functions.open_library_api.get_author
args:
author: !Py $BATCH
write_author_json:
input: get_author
function: typhoon.filesystem.write_data
args:
hook: !Hook data_lake
data: !MultiStep
- !Py $BATCH['docs']
- !Py typhoon.data.json_array_to_json_records($1)
path: !MultiStep
- !Py $BATCH['docs'][0]['key']
- !Py f'/authors/{$1}.json'
create_intermediate_dirs: True
Getting the works of my favorite authors from Open Library API
Installation
See documentation for detailed guidance on installation and walkthroughs.
with pip (typhoon standalone)
Install typhoon:
pip install typhoon-orchestrator[dev]
Optionally, install and activate virtualenv.
Then:
typhoon init hello_world
cd hello_world
typhoon status
This will create a directory named hello_world that serves as an example project. As in git, when we cd into the directory it will detect that it's a Typhoon project and consider that directory the base directory for Typhoon (TYPHOON_HOME).
Adding connnections
You can add a default connections as follows in the cli
typhoon connection add --conn-id data_lake --conn-env local
# Check that it was added
typhoon connection ls -l
With Docker and Airflow
To deploy Typhoon with Airflow you need:
- Docker / Docker Desktop (You must use WSL2 on Windows)
- Download the [docker-compose.yaml][1] (or use curl below)
- Create a directory for your TYPHOON_PROJECTS_HOME
The following sets up your project directory and gets the docker-compose.yml:
TYPHOON_PROJECTS_HOME="/tmp/typhoon_projects" # Or any other path you prefer
mkdir -p $TYPHOON_PROJECTS_HOME/typhoon_airflow_test
cd $TYPHOON_PROJECTS_HOME/typhoon_airflow_test
mkdir src
curl -LfO https://raw.githubusercontent.com/typhoon-data-org/typhoon-orchestrator/master/docker-compose-af.yml
docker compose -f docker-compose-af.yml up -d
docker-compose -f docker-compose-af.yml run --rm typhoon-af airflow initdb
docker-compose -f docker-compose-af.yml run --rm typhoon-af typhoon status
docker-compose -f docker-compose-af.yml run --rm typhoon-af typhoon connection add --conn-id data_lake --conn-env local # Adding our first connection!
docker-compose -f docker-compose-af.yml run --rm typhoon-af typhoon dag build --all
docker restart typhoon-af # Wait while docker restarts
This runs a container with only 1 service, typhoon-af
. This has both Airflow and Typhoon installed on it ready to work with.
You should be able to then check typhoon status
and also the airlfow UI at http://localhost:8088
Typhoon DAGS listed in airflow UI
Development hints are in the docs.
Favorite Authors DAG - as displayed in airflow UI
We can extend the above task to give an example with more complexity. The tutorial for this has some more advanced tips. The airflow compiled DAG handles complex DAG structures very nicely:
Favorite Authors Extended - a complex DAG example
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for typhoon-orchestrator-0.0.51.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3bfe89307ef76895453f3dc10caf4ef471fb684b6e7482c1d56586cc81df69b1 |
|
MD5 | fd7cc318c7bb395bcf492ea19b819ee5 |
|
BLAKE2b-256 | 2b0174478092f26a95e212c5fa9c1551b06e993b695adf719caa4051a09832a5 |
Hashes for typhoon_orchestrator-0.0.51-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 348161508b78a17e0cf3f8d004927fcebf801d410491f28bd21925fff8e64eb4 |
|
MD5 | 03c55352cdcba3ee00f05af218ac03e6 |
|
BLAKE2b-256 | 6f13c9e8e5d0f46147e039bfa8923e8863883d9311230e0675afd3e2242f18ce |