proxystore 0.0.0

Advanced data flow management for distributed Python applications

ProxyStore

ProxyStore facilitates efficient data flow management in distributed Python applications, such as dynamic task-based workflows or serverless and edge applications.

The transparent object proxy, the core building block within ProxyStore, acts like a wide-area reference that can be cheaply communicated. Unlike traditional references that are only valid within the virtual address space of a single process, the proxy references an object in remote storage and can be implicitly dereferenced in arbitrary processes—even on remote machines. The proxy is transparent in that it implicitly dereferences its target object when used—referred to a just-in-time resolution—and afterwards forwards all operations on itself to the cached target object.

This paradigm results in the best of both pass-by-reference and pass-by-value semantics, improves performance and portability by reducing transfer overheads through intermediaries, and abstracts low-level communication methods which reduces code complexity. A proxy contains within itself all the information and logic necessary to resolve the target object. This self-contained nature means a proxy consumer need not be aware of the low-level communication mechanisms used by the proxy; rather, this is unilaterally determined by the producer of the proxy.

ProxyStore supports a diverse set of programming patterns built on the proxy paradigm:

• Task-based Workflows
• Function-as-a-Service/Serverless Applications
• Distributed Futures
• Bulk Data Streaming
• and more!

ProxyStore can leverage many popular mediated data transfer and storage systems: DAOS, Globus Transfer, Kafka, KeyDB, and Redis. Custom communication methods built on Mochi, UCX, WebRTC, and ZeroMQ are provided for high-performance and peer-to-peer applications.

Read more about ProxyStore's concepts here. Complete documentation for ProxyStore is available at docs.proxystore.dev.

Installation

The base ProxyStore package can be installed with pip.

pip install proxystore

Leveraging third-party libraries may require dependencies not installed by default but can be enabled via extras installation options (e.g., endpoints, kafka, or redis). All additional dependencies can be installed with:

pip install proxystore[all]

This will also install the proxystore-ex package which contains extensions and experimental features. The extensions package can also be installed with pip using proxystore[extensions] or proxystore-ex.

See the Installation guide for more information about the available extras installation options. See the Contributing guide to get started for local development.

Example

Creating proxies of objects only requires a few lines of code.

from proxystore.connectors.local import LocalConnector
from proxystore.proxy import Proxy
from proxystore.store import Store

def process(x: dict[str, str]) -> None:
    # x is transparently resolved from the store when first
    # used by the function. After which the proxy behaves as
    # target object (the dict) for the rest of its existence.
    assert isinstance(x, dict)
    assert x['hello'] == 'world'

with Store('example', connector=LocalConnector(), register=True) as store:
    # Store the object via a connector interface (here, a thread local
    # store). The returned proxy acts like a reference to the object.
    x = {'hello': 'world'}
    proxy = store.proxy(x)
    assert isinstance(proxy, Proxy)

    # Invoking a function with proxy works without function changes.
    process(proxy)

Check out the Get Started guide to learn more!

Citation

If you use ProxyStore or any of this code in your work, please cite our ProxyStore (SC '23) and Proxy Patterns (TPDS) papers.

@inproceedings{pauloski2023proxystore,
    title = {Accelerating {C}ommunications in {F}ederated {A}pplications with {T}ransparent {O}bject {P}roxies},
    author = {Pauloski, J. Gregory and Hayot-Sasson, Valerie and Ward, Logan and Hudson, Nathaniel and Sabino, Charlie and Baughman, Matt and Chard, Kyle and Foster, Ian},
    address = {New York, NY, USA},
    articleno = {59},
    booktitle = {Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis},
    doi = {10.1145/3581784.3607047},
    isbn = {9798400701092},
    location = {Denver, CO, USA},
    numpages = {15},
    publisher = {Association for Computing Machinery},
    series = {SC '23},
    url = {https://doi.org/10.1145/3581784.3607047},
    year = {2023}
}

@article{pauloski2024proxystore,
    title = {Object {P}roxy {P}atterns for {A}ccelerating {D}istributed {A}pplications},
    author = {Pauloski, J. Gregory and Hayot-Sasson, Valerie and Ward, Logan and Brace, Alexander and Bauer, André and Chard, Kyle and Foster, Ian},
    doi = {10.1109/TPDS.2024.3511347},
    journal = {IEEE Transactions on Parallel and Distributed Systems},
    number = {},
    pages = {1-13},
    volume = {},
    year = {2024}
}