Skip to main content

Exca - ⚔

Execute and cache seamlessly in python.

workflow badge

Quick install

pip install exca

Full documentation

Documentation is available at https://facebookresearch.github.io/exca/

Basic overview

exca provides simple decorators to:

  • execute a (hierarchy of) computation(s) either locally or on distant nodes,
  • cache the result.

The problem:

In ML pipelines, the use of a simple python function, such as my_task:

import numpy as np

def my_task(param: int = 12) -> float:
    return param * np.random.rand()

often requires cumbersome overheads to (1) configure the parameters, (2) submit the job on a cluster, (3) cache the results: e.g.

import pickle
from pathlib import Path
import submitit

# Configure
param = 12

# Check task has already been executed
filepath = tmp_path / f'result-{param}.npy'
if not filepath.exists():

    # Submit job on cluster
    executor = submitit.AutoExecutor(cluster=None, folder=tmp_path)
    job = executor.submit(my_task, param)
    result = job.result()

    # Cache result
    with filepath.open("wb") as f:
        pickle.dump(result, f)

These overheads lead to several issues, such as debugging, handling hierarchical execution and properly saving the results (ending in the classic 'result-parm12-v2_final_FIX.npy').

The solution:

exca can be used to decorate the method of a pydantic model so as to seamlessly configure its execution and caching:

import numpy as np
import pydantic
import exca as xk

class MyTask(pydantic.BaseModel):
    param: int = 12
    infra: xk.TaskInfra = xk.TaskInfra()

    @infra.apply
    def process(self) -> float:
        return self.param * np.random.rand()


task = MyTask(param=1, infra={"folder": tmp_path, "cluster": "auto"})
out = task.process()  # runs on slurm if available
# calling process again will load the cache and not a new random number
assert out == task.process()

See the API reference for all the details

Quick comparison

feature \ tool lru_cache hydra submitit exca
RAM cache ✔ ✔
file cache ✔
remote compute ✔ ✔ ✔
pure python (vs command line) ✔ ✔ ✔
hierarchical config ✔ ✔

Contributing

See the CONTRIBUTING file for how to help out.

Citing

@misc{exca,
    author = {J. Rapin and J.-R. King},
    title = {{Exca - Execution and caching}},
    year = {2024},
    publisher = {GitHub},
    journal = {GitHub repository},
    howpublished = {\url{https://github.com/facebookresearch/exca}},
}

License

exca is MIT licensed, as found in the LICENSE file. Also check-out Meta Open Source Terms of Use and Privacy Policy.

Metadata

Release files for exca 0.6.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for exca 0.6.1
File Size Uploaded Provenance Actions
exca-0.6.1.tar.gz 180.3 kB Without provenance Details for exca-0.6.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for exca 0.6.1
File Interpreter ABI Platform Actions
exca-0.6.1-py3-none-any.whl Python 3 none any Details for exca-0.6.1-py3-none-any.whl

Total release size: 394.0 kB

Release files / exca-0.6.1.tar.gz

Download URL exca-0.6.1.tar.gz
Size 180.3 kB
Tags Source
SHA-256 checksum
How to use checksums
29bc262e50d3f17ebf6b9724b3f68eb772a7457521e044d0cb4236ad0a0a7bd9
BLAKE2b-256 checksum
How to use checksums
b65e820c4a3ba1d43e093cffb169538531abb4e9ad895b293c1cffeeebccc63b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.12.8

Release files / exca-0.6.1-py3-none-any.whl

Download URL exca-0.6.1-py3-none-any.whl
Size 213.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
c46b1c24439683ab9a5fd0e8a4b0c31e2c6790eff7a4dffab78f11fb3865cbfd
BLAKE2b-256 checksum
How to use checksums
b49419274f6dbde9ccee2891cd240fb0dcbb4f4f6c816f6e5b0a3efb7308a09d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.12.8

Security for exca 0.6.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.6.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page