FlowTask
FlowTask is a plugin-based, component-driven task execution framework for create complex Tasks.
FlowTask runs Tasks defined in JSON, YAML or TOML files, any Task is a combination of Components, and every component in the Task run sequentially or depend of others, like a DAG.
Can create a Task combining Commands, Shell scripts and other specific Components (as TableInput: Open a Table using a datasource, DownloadFromIMAP: Download a File from a IMAP Folder, and so on), any Python Callable can be a Component inside a Task, or can extends UserComponent to build your own componets.
Every designed Task can run from CLI, programmatically, via RESTful API (using our aioHTTP-based Handler), called by WebHooks or even dispatched to a external Worker using our built-in Scheduler.
Quickstart
pip install flowtask
Tasks can organizated into directory structure like this:
tasks / ├── programs / ├── test / ├── tasks /
The main reason of this structure, is maintain organized several tasks by tenant/program, avoiding filling a directory with several task files.
FlowTask support "TaskStorage", a Task Storage is the main repository for tasks, main Task Storage is a directory in any filesystem path (optionally you can syncronize that path using git), but Tasks can be saved onto a Database or a S3 bucket.
Dependencies
- aiohttp (Asyncio Web Framework and Server) (required by navigator)
- AsyncDB
- QuerySource
- Navigator-api
- (Optional) Qworker (for distributing asyncio Tasks on distributed workers).
Features
- Component-based Task execution framework with several components covering several actions (download files, create pandas dataframes from files, mapping dataframe columns to a json-dictionary, etc)
- Built-in API for execution of Tasks.
How I run a Task?
Can run a Task from CLI:
task --program=test --task=example
on CLI, you can pass an ENV (enviroment) to change the environment file on task execution.
ENV=dev task --program=test --task=example
or Programmatically:
from flowtask import Task
import asyncio
task = Task(program='test', task='example')
results = asyncio.run(task.run())
# we can alternatively, using the execution mode of task object:
results = asyncio.run(task())
Requirements
- Python >= 3.9
- asyncio (https://pypi.python.org/pypi/asyncio/)
- aiohttp >= 3.6.2
Contribution guidelines
Please have a look at the Contribution Guide
- Writing tests
- Code review
- Other guidelines
Who do I talk to?
- Repo owner or admin
- Other community or team contact
License
Navigator is licensed under Apache 2.0 License. See the LICENSE file for more details.
2 maintainers
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file flowtask-5.12.13.tar.gz.
File metadata
- Download URL: flowtask-5.12.13.tar.gz
- Upload date:
- Size: 1.5 MB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
ee6db375a0d0c76a40b66f158ba9837616213d14c213fd35748d34fc1fe21a02
|
|
| MD5 |
611b7a2ab2998f45242731d86c0655d9
|
|
| BLAKE2b-256 |
b5e6a23bba47eb2c2c287a2a8c8f40525999045cdef187626b86882191c27d2e
|
Provenance
The following attestation bundles were made for flowtask-5.12.13.tar.gz:
Publisher:
release.yml on phenobarbital/flowtask
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
flowtask-5.12.13.tar.gz -
Subject digest:
ee6db375a0d0c76a40b66f158ba9837616213d14c213fd35748d34fc1fe21a02 - Sigstore transparency entry: 2539306426
- Sigstore integration time:
-
Permalink:
phenobarbital/flowtask@e9ee7133a8222db534c80ad83c55d1be71836267 -
Branch / Tag:
refs/tags/5.12.13 - Owner: https://github.com/phenobarbital
-
Access:
private
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@e9ee7133a8222db534c80ad83c55d1be71836267 -
Trigger Event:
release
-
Statement type:
File details
Details for the file flowtask-5.12.13-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.
File metadata
- Download URL: flowtask-5.12.13-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
- Upload date:
- Size: 3.6 MB
- Tags: CPython 3.13, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
82d3da0e8ddaf7c00d7f5d55f946b6be4ebef7fad402e84cc77fd56b46afbd4a
|
|
| MD5 |
e312e5ef92a8e38310155e93930c2c79
|
|
| BLAKE2b-256 |
21bf73f5e929ef08a571e66081ca25a963efd8f337bf91450b637a5a77380cbd
|
Provenance
The following attestation bundles were made for flowtask-5.12.13-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl:
Publisher:
release.yml on phenobarbital/flowtask
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
flowtask-5.12.13-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl -
Subject digest:
82d3da0e8ddaf7c00d7f5d55f946b6be4ebef7fad402e84cc77fd56b46afbd4a - Sigstore transparency entry: 2539306514
- Sigstore integration time:
-
Permalink:
phenobarbital/flowtask@e9ee7133a8222db534c80ad83c55d1be71836267 -
Branch / Tag:
refs/tags/5.12.13 - Owner: https://github.com/phenobarbital
-
Access:
private
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@e9ee7133a8222db534c80ad83c55d1be71836267 -
Trigger Event:
release
-
Statement type:
File details
Details for the file flowtask-5.12.13-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.
File metadata
- Download URL: flowtask-5.12.13-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
- Upload date:
- Size: 3.6 MB
- Tags: CPython 3.12, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
cd8421020015650657cdc7f4f1c197ada47e49152420b3fdc71a7af98825ea5a
|
|
| MD5 |
7507f2570c32661e9859e5fbea5fe0e1
|
|
| BLAKE2b-256 |
d6ce0501ccc391255a7b2a8f5b6ea92af659296d8107cfdb9f2b8575d72d6efb
|
Provenance
The following attestation bundles were made for flowtask-5.12.13-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl:
Publisher:
release.yml on phenobarbital/flowtask
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
flowtask-5.12.13-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl -
Subject digest:
cd8421020015650657cdc7f4f1c197ada47e49152420b3fdc71a7af98825ea5a - Sigstore transparency entry: 2539306560
- Sigstore integration time:
-
Permalink:
phenobarbital/flowtask@e9ee7133a8222db534c80ad83c55d1be71836267 -
Branch / Tag:
refs/tags/5.12.13 - Owner: https://github.com/phenobarbital
-
Access:
private
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@e9ee7133a8222db534c80ad83c55d1be71836267 -
Trigger Event:
release
-
Statement type:
File details
Details for the file flowtask-5.12.13-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.
File metadata
- Download URL: flowtask-5.12.13-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
- Upload date:
- Size: 3.5 MB
- Tags: CPython 3.11, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
f70a9a4419305c35201155519696916b7f861dee4648b23fd82ee825cc7b54f7
|
|
| MD5 |
b4a5e76f6dcc4bc946e5d1a363ee6312
|
|
| BLAKE2b-256 |
242ca467235dd1e8870c2762cf109b039c3f9d81ef4023b5227ccf3b7a7a41c8
|
Provenance
The following attestation bundles were made for flowtask-5.12.13-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl:
Publisher:
release.yml on phenobarbital/flowtask
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
flowtask-5.12.13-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl -
Subject digest:
f70a9a4419305c35201155519696916b7f861dee4648b23fd82ee825cc7b54f7 - Sigstore transparency entry: 2539306470
- Sigstore integration time:
-
Permalink:
phenobarbital/flowtask@e9ee7133a8222db534c80ad83c55d1be71836267 -
Branch / Tag:
refs/tags/5.12.13 - Owner: https://github.com/phenobarbital
-
Access:
private
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@e9ee7133a8222db534c80ad83c55d1be71836267 -
Trigger Event:
release
-
Statement type:
