PyCAPIO

PyCAPIO brings transparent data streaming to file-based Python workflows, minimizing I/O bottlenecks without requiring code modifications. PyCAPIO works by monkey patching built-in IO methods and classes, interfacing them natively with the CAPIO methodology.

Prerequisites

Before installing PyCAPIO, ensure your system meets the following requirements:

At Compile time, CAPIO is also fetched and compiled from source.

Installation

From PyPI (Recommended)

PyCAPIO is available as prebuilt wheels on PyPI. You can install it inside a virtual environment:

python3 -m venv venv
source venv/bin/activate
pip install pycapio

From Source

To build and install PyCAPIO from the source repository:

git clone [https://github.com/High-Performance-IO/PyCAPIO.git](https://github.com/High-Performance-IO/PyCAPIO.git)
cd PyCAPIO
python3 -m venv venv
source venv/bin/activate
pip install .

Usage

1. Global Interception (CLI) PyCAPIO can be invoked via the command line to inject streaming capabilities across your entire workflow step:

   pycapio --capio-dir ./dir --app-name app --workflow-name ./workflow my_step.py
   

2. Fine-Grained Interception (CapioContext) For precise control over which regions handle interception, use the CapioContext decorator. Only I/O operations executed inside this context will be intercepted by CAPIO.

   from pycapio import CapioContext
   
   
   @CapioContext(capio_dir=".", app_name="reader", workflow_name="example_workflow")
   def read(path):
       with open(path, "r") as f:
           data = f.read()
           return data
   
   # Only I/O within this function call is intercepted
   data = read("example.txt")
   

3. PyCAPIO internal It is finally possible to use PyCAPIO's internal methods to intercept I/O operations. To do this, simply import the PyCAPIO internal module and use its methods.

   from pycapio.internals import *
   

   This will import all the necessary methods and classes that can be used to develop custom IO modules which will interface directly with the CAPIO server.

Important Notes

[!IMPORTANT] PyCAPIO natively intercepts I/O operations that use Python's built-in methods. If your script relies on external libraries compiled into shared objects (.so files), please bypass Python-level interception and use the standard LD_PRELOAD approach for CAPIO instead.

Team

Developing team