Exca - ⚔
Execute and cache seamlessly in python.
Quick install
pip install exca
Full documentation
Documentation is available at https://facebookresearch.github.io/exca/
Basic overview
exca provides simple decorators to:
- execute a (hierarchy of) computation(s) either locally or on distant nodes,
- cache the result.
The problem:
In ML pipelines, the use of a simple python function, such as my_task:
import numpy as np
def my_task(param: int = 12) -> float:
return param * np.random.rand()
often requires cumbersome overheads to (1) configure the parameters, (2) submit the job on a cluster, (3) cache the results: e.g.
import pickle
from pathlib import Path
import submitit
# Configure
param = 12
# Check task has already been executed
filepath = tmp_path / f'result-{param}.npy'
if not filepath.exists():
# Submit job on cluster
executor = submitit.AutoExecutor(cluster=None, folder=tmp_path)
job = executor.submit(my_task, param)
result = job.result()
# Cache result
with filepath.open("wb") as f:
pickle.dump(result, f)
These overheads lead to several issues, such as debugging, handling hierarchical execution and properly saving the results (ending in the classic 'result-parm12-v2_final_FIX.npy').
The solution:
exca can be used to decorate the method of a pydantic model so as to seamlessly configure its execution and caching:
import numpy as np
import pydantic
import exca as xk
class MyTask(pydantic.BaseModel):
param: int = 12
infra: xk.TaskInfra = xk.TaskInfra()
@infra.apply
def process(self) -> float:
return self.param * np.random.rand()
task = MyTask(param=1, infra={"folder": tmp_path, "cluster": "auto"})
out = task.process() # runs on slurm if available
# calling process again will load the cache and not a new random number
assert out == task.process()
See the API reference for all the details
Quick comparison
| feature \ tool | lru_cache | hydra | submitit | exca |
|---|---|---|---|---|
| RAM cache | ✔ | ✔ | ||
| file cache | ✔ | |||
| remote compute | ✔ | ✔ | ✔ | |
| pure python (vs command line) | ✔ | ✔ | ✔ | |
| hierarchical config | ✔ | ✔ |
Contributing
See the CONTRIBUTING file for how to help out.
Citing
@misc{exca,
author = {J. Rapin and J.-R. King},
title = {{Exca - Execution and caching}},
year = {2024},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/facebookresearch/exca}},
}
License
exca is MIT licensed, as found in the LICENSE file.
Also check-out Meta Open Source Terms of Use and Privacy Policy.
1 maintainer
Metadata
1 maintainer
Release files for exca 0.6.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| exca-0.6.0.tar.gz | 176.0 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| exca-0.6.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 385.1 kB
Release files / exca-0.6.0.tar.gz
| Download URL | exca-0.6.0.tar.gz |
|---|---|
| Size | 176.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2aea4da5a9153f76e94360e2e5e6e69c1b6cf7da0e8bd1b2f6de114b59344c20
|
|
BLAKE2b-256 checksum How to use checksums |
9e242b1260f79491b151024e536f660fddff6cd58d05ffe89b76ba04cabdc86a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.8
|
Release files / exca-0.6.0-py3-none-any.whl
| Download URL | exca-0.6.0-py3-none-any.whl |
|---|---|
| Size | 209.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
63c2871534c763d688f12694612b7a256525b2099e778f5bb177d82ec3d883d5
|
|
BLAKE2b-256 checksum How to use checksums |
f58eb32dc8c0b90f75a2f78a9ee0332d4ed40f394bce4978294bf9be2281f351
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.8
|
