Cache ML – layer on top of joblib to cache parsed datasets, dramatically reducing load time of large data files. Also supports encryption at rest. Currently supported backends are local filesystem and S3.
Example Usage
Here is an example from a Jupyter notebook:
import pandas as pd
from cacheml.cache import LocalFile, Cache
cache = Cache()
@cache.cache # this function's result will be cached
def read_and_filter_commits(commits_file_obj):
return pd.read_csv(commits_file_obj.path)
ts_all = read_and_filter_commits(LocalFile(commits.csv.gz))
Performance Test Results
There are from running the unit tests which simulate loading the time series data from datahut.ai, which is in a 216MB compressed csv file. The first case just loads into a dataframe, while the second case does some additional processing (sorting, removing entries outside a time range).
File location |
Time for raw df read |
Time for initial read and caching of file |
Time for cached read |
|---|---|---|---|
Local File |
134.0 |
130.9 |
0.41 |
S3 |
153.6 |
144.6 |
0.38 |
File location |
Time for original function |
Time for initial read and caching of file |
Time for cached read |
|---|---|---|---|
Local File |
139.6 |
142.49 |
1.04 |
S3 |
153.4 |
155.8 |
0.99 |
Copyright
Copyright 2021 by Benedat LLC. Available under the Apache 2.0 license.
1 maintainer
Metadata
1 maintainer
Release files for CacheML 1.0.4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| CacheML-1.0.4.tar.gz | 12.5 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| CacheML-1.0.4-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 25.7 kB
Release files / CacheML-1.0.4.tar.gz
| Download URL | CacheML-1.0.4.tar.gz |
|---|---|
| Size | 12.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
fcde1402bd2cb546d85ff5fcf5957dcab53de443c942ca7f1eeb2c3fd5ee74ef
|
|
BLAKE2b-256 checksum How to use checksums |
f91bdb28818d9f3963b57399001c03d237715556911074758e611f847b3182bb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.6.0 importlib_metadata/4.8.2 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.7
|
Release files / CacheML-1.0.4-py3-none-any.whl
| Download URL | CacheML-1.0.4-py3-none-any.whl |
|---|---|
| Size | 13.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
d42c16f0dbe88d758c08b24b11abc0164e6070ebb96dca4bd518aea1def765a7
|
|
BLAKE2b-256 checksum How to use checksums |
abcddd083bca88b8f0fe1b9db628313ce00eeffe0f02d2ae85b2ef9a5022214b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.6.0 importlib_metadata/4.8.2 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.7
|
