Create asynchronous data pipelines and deploy to cloud or airflow
Project description
Forum :wave: |
Installation :floppy_disk: |
Documentation :notebook:
️
️
Elegant YAML DAGS for Data Pipelines
Deploy to your existing Airflow.
️
@todo Badges here
Why Typhoon? |
Key Features |
Example YAML |
Installation
️
Why Typhoon (+ Airflow)?
Airflow is great!
Typhoon lets you write Airflow DAGS faster :rocket::
**Workflow**: Typhoon YAML DAG --> Typhoon build --> Airflow DAG
Simplicity and re-usability; a toolkit designed to be loved by Data Engineers :heart:
Key features
Elegant - YAML; low-code and easy to learn. Code-completion - Fast to compose. (VS Code recommended). Data sharing - data flows between tasks making it super intuitive. Composability - Functions and connections combine like Lego. |
|
Components - reduce complex tasks to 1 re-usable tasks Packaged examples:
UI: Share pre-built components (data pipelines) with your team :raised_hands: |
|
Rich CLI & Shell: Inspired by others; instantly familiar. Testable Tasks - automate DAG task tests. Testable Python - test functions or full DAGs with PyTest. |
Example YAML DAG
```yaml linenums="1"
name: favorite_authors
schedule_interval: rate(1 day)
tasks:
choose_favorites:
function: typhoon.flow_control.branch
args:
branches:
- J. K. Rowling
- George R. R. Martin
- James Clavell
get_author:
input: choose_favorites
function: functions.open_library_api.get_author
args:
author: !Py $BATCH
write_author_json:
input: get_author
function: typhoon.filesystem.write_data
args:
hook: !Hook data_lake
data: !MultiStep
- !Py $BATCH['docs']
- !Py typhoon.data.json_array_to_json_records($1)
path: !MultiStep
- !Py $BATCH['docs'][0]['key']
- !Py f'/authors/{$1}.json'
create_intermediate_dirs: True
```
Installation
See documentation for detailed guidance on installation and walkthroughs.
with pip (typhoon standalone)
Install typhoon:
pip install typhoon-orchestrator[dev]
Optionally, install and activate virtualenv.
Then:
typhoon init hello_world
cd hello_world
typhoon status
This will create a directory named hello_world that serves as an example project. As in git, when we cd into the directory it will detect that it's a Typhoon project and consider that directory the base directory for Typhoon (TYPHOON_HOME).
With Docker and Airflow
To deploy Typhoon with Airflow you need:
- Docker / Docker Desktop (You must use WSL2 on Windows)
- Download the [docker-compose.yaml][1] (or use curl below)
- Create a directory for your TYPHOON_PROJECTS_HOME
The following sets up your project directory and gets the docker-compose.yml:
TYPHOON_PROJECTS_HOME="/tmp/typhoon_projects" # Or any other path you prefer
mkdir -p $TYPHOON_PROJECTS_HOME/typhoon_airflow_test
cd $TYPHOON_PROJECTS_HOME/typhoon_airflow_test
mkdir src
curl -LfO https://raw.githubusercontent.com/typhoon-data-org/typhoon-orchestrator/master/docker-compose-af.yml
docker compose -f docker-compose-af.yml up -d
docker exec -it typhoon-af bash # Then you're in the typhoon home.
airflow initdb # !! To initiate Airflow DB !!
typhoon status # To see status of dags & connections
typhoon dag build --all # Build the example DAGS
exit # exits docker
docker restart typhoon-af # Wait while docker restarts
This runs a container with only 1 service, typhoon-af
. This has both Airflow and Typhoon installed on it ready to work with.
You should be able to then check typhoon status
and also the airlfow UI at http://localhost:8088
Development hints are in the docs.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for typhoon-orchestrator-0.0.36.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 837aa200a58a0610607687a8ae67b35f527e8b068f44c3be907ebb4210c24ed3 |
|
MD5 | 8f2703edc025e72f564b9d19078b0827 |
|
BLAKE2b-256 | 6983d75e38beb6da253ab54ce4614adb5da265612712728addbe4893e55a870d |
Hashes for typhoon_orchestrator-0.0.36-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7859117d6227714b52a93c33ca65d20b9aa595e6c518250a098078825b73d029 |
|
MD5 | c4d18bf84814781de5b8017747b56656 |
|
BLAKE2b-256 | ad221f7382a175aec966ab848605063995b49b292fca095d2e0af861f94f1c96 |