Exca - ⚔
Execute and cache seamlessly in python.
Quick install
pip install exca
Full documentation
Documentation is available at https://facebookresearch.github.io/exca/
Basic overview
exca provides simple decorators to:
- execute a (hierarchy of) computation(s) either locally or on distant nodes,
- cache the result.
The problem:
In ML pipelines, the use of a simple python function, such as my_task:
import numpy as np
def my_task(param: int = 12) -> float:
return param * np.random.rand()
often requires cumbersome overheads to (1) configure the parameters, (2) submit the job on a cluster, (3) cache the results: e.g.
import pickle
from pathlib import Path
import submitit
# Configure
param = 12
# Check task has already been executed
filepath = tmp_path / f'result-{param}.npy'
if not filepath.exists():
# Submit job on cluster
executor = submitit.AutoExecutor(cluster=None, folder=tmp_path)
job = executor.submit(my_task, param)
result = job.result()
# Cache result
with filepath.open("wb") as f:
pickle.dump(result, f)
These overheads lead to several issues, such as debugging, handling hierarchical execution and properly saving the results (ending in the classic 'result-parm12-v2_final_FIX.npy').
The solution:
exca can be used to decorate the method of a pydantic model so as to seamlessly configure its execution and caching:
import numpy as np
import pydantic
import exca as xk
class MyTask(pydantic.BaseModel):
param: int = 12
infra: xk.TaskInfra = xk.TaskInfra()
@infra.apply
def process(self) -> float:
return self.param * np.random.rand()
task = MyTask(param=1, infra={"folder": tmp_path, "cluster": "auto"})
out = task.process() # runs on slurm if available
# calling process again will load the cache and not a new random number
assert out == task.process()
See the API reference for all the details
Quick comparison
| feature \ tool | lru_cache | hydra | submitit | exca |
|---|---|---|---|---|
| RAM cache | ✔ | ✔ | ||
| file cache | ✔ | |||
| remote compute | ✔ | ✔ | ✔ | |
| pure python (vs command line) | ✔ | ✔ | ✔ | |
| hierarchical config | ✔ | ✔ |
Contributing
See the CONTRIBUTING file for how to help out.
Citing
@misc{exca,
author = {J. Rapin and J.-R. King},
title = {{Exca - Execution and caching}},
year = {2024},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/facebookresearch/exca}},
}
License
exca is MIT licensed, as found in the LICENSE file.
Also check-out Meta Open Source Terms of Use and Privacy Policy.
Metadata
Release files for exca 0.6.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| exca-0.6.1.tar.gz | 180.3 kB | Without provenance | Details for exca-0.6.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| exca-0.6.1-py3-none-any.whl | Python 3 | none | any | Details for exca-0.6.1-py3-none-any.whl |
Total release size: 394.0 kB
Release files / exca-0.6.1.tar.gz
| Download URL | exca-0.6.1.tar.gz |
|---|---|
| Size | 180.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
29bc262e50d3f17ebf6b9724b3f68eb772a7457521e044d0cb4236ad0a0a7bd9
|
|
BLAKE2b-256 checksum How to use checksums |
b65e820c4a3ba1d43e093cffb169538531abb4e9ad895b293c1cffeeebccc63b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.8
|
Release files / exca-0.6.1-py3-none-any.whl
| Download URL | exca-0.6.1-py3-none-any.whl |
|---|---|
| Size | 213.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
c46b1c24439683ab9a5fd0e8a4b0c31e2c6790eff7a4dffab78f11fb3865cbfd
|
|
BLAKE2b-256 checksum How to use checksums |
b49419274f6dbde9ccee2891cd240fb0dcbb4f4f6c816f6e5b0a3efb7308a09d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.8
|
Security for exca 0.6.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.