io-collection 0.8.0

Collection of tasks for I/O.

Collection of tasks for I/O. Designed to be used both in Prefect workflows and as modular, useful pieces of code.

Installation

The collection can be installed using:

pip install io-collection

We recommend using Poetry to manage and install dependencies. To install into your Poetry project, use:

poetry add io-collection

Usage

Prefect workflows

All tasks in this collection are wrapped in a Prefect @task decorator, and can be used directly in a Prefect @flow. Running tasks within a Prefect flow enables you to take advantage of features such as automatically retrying failed tasks, monitoring workflow states, running tasks concurrently, deploying and scheduling flows, and more.

from prefect import flow
from io_collection.<module_name> import <task_name>

@flow
def run_flow():
    <task_name>()

if __name__ == "__main__":
    run_flow()

See cell-abm-pipeline for examples of using tasks from different collections to build a pipeline for simulating and analyzing agent-based model data.

Individual tasks

Not all use cases require a full workflow. Tasks in this collection can be used without the Prefect @task decorator by simply importing directly from the module:

from io_collection.<module_name>.<task_name> import <task_name>

def main():
    <task_name>()

if __name__ == "__main__":
    main()

or using the .fn() method:

from io_collection.<module_name> import <task_name>

def main():
    <task_name>.fn()

if __name__ == "__main__":
    main()