Skip to main content

python-file-cache

一个简单易用的Python的文件缓存工具

A simple and easy-to-use file caching tool

安装 Install

pip install filecaching

如何使用 How to use

# 引入 Import
from filecaching import FileCache
# 实例化 Instantiate
fc = FileCache()
#(可以使用FileCache(cache_path)指定缓存目录路径)
# (You can use FileCache(cache_path) specify the cache directory path)
fc = FileCache(os.path.join(os.path.dirname(os.path.abspath(__file__)), 'cache'))
# 设置缓存 Set cache
fc.cache(key, value)
# 设置具有有效期的缓存 Set cache with an expiration date(单位是秒)
fc.cache(key, value, expiration_seconds)
# 读取缓存 Get cache
fc.cache(key)
# 删除 Remove cache
fc.rm(key)

一些实例 Some examples

from filecaching import FileCache
fc = FileCache()
# 设置缓存 Set cache
fc.cache('a', {'v': 'abc'})
# 设置具有有效期的缓存 Set cache with an expiration date (30s)
fc.cache('b', [1,2,3], 30)
# 读取缓存 Get cache
print(fc.cache('a'))
print(fc.cache('b'))
# 删除 Remove cache
fc.rm('a')

清理过期缓存 Clean up expired cache

exp 只保证过期之后读不到值:get 命中一条过期缓存时会顺手把它删掉。 如果一条缓存写入后再也没被读到,它的文件会一直留在磁盘上。 要让文件真正消失,需要运行清理脚本 —— 定期手动跑、加进 crontab、或在常驻进程里循环调用,任选其一。

命令行 / 定时任务 Command line & cron

# 推荐用 python -m:cron 环境里的 python 往往不是应用 venv 里的那个
python -m filecaching.purger --path /var/app/cache
filecaching-purge --path /var/app/cache          # 等价的命令行入口

# crontab:每小时清一次,顺手删掉清空的分片目录
0 * * * * cd /var/app && /var/app/venv/bin/python -m filecaching.purger --path cache --vacuum -q

在代码里调用 In code

from filecaching.purger import purge
purge('/var/app/cache')                       # 返回一份统计字典
FileCache('/var/app/cache').purge()           # 等价

# 交给 subprocess / supervisor 的命令行数组
FileCache('/var/app/cache').purge_command('--vacuum')
# [sys.executable, '-m', 'filecaching.purger', '--path', '/var/app/cache', '--vacuum']

参数 Options

参数 说明
--path 缓存目录,即 FileCache(path) 的 path(必填)
--rebuild 忽略已有索引,完整扫一遍 payload
--dry-run 只统计会删多少,不删除、不写索引
--shard XX 只处理某个分片(两位十六进制,可重复),便于拆成多份并行清理
--max-files N 最多删 N 个文件后退出,下次接着做
--deadline SEC 超过 SEC 秒后退出,下次接着做
--vacuum 分片被清空后连目录一起删除
--json / -q JSON 输出 / 静默

退出码:0 正常,1 出错,2 已有清理进程在跑。脚本自带目录锁,可以安全地重复调度。

性能 Performance

20000 条 512 字节缓存、其中一半已过期,Windows NTFS 本地盘实测:

场景 耗时
应用写入这 20000 条 7.5 s
首轮清理(没有索引,逐条读 payload 建立索引) 6.1 s
之后每轮(无任何变动) 0.66 s
之后每轮(2000 条被改写) 0.79 s

也就是说首轮大致等于"把缓存重新写一遍"的代价,之后每轮只与变动的文件数成正比,且基本不受缓存值大小影响。 索引文件 cache/xx/.index 每万条约占 0.17 MB。Linux / ext4 上以上各项都要再快一个数量级。

关于 .index

  • 每个分片目录下多出一个 .index,只记录「文件名、过期时间、文件修改时间」三件事。
  • 它只是用来免去重复打开文件的加速结构,过期与否最终以数据文件内容为准;可以随时删掉,下一轮自动重建。
  • 库的读写路径(cache / set / get / rm)完全不碰它,只有清理脚本读写它。
  • 脚本只处理形如 cache/两位十六进制/十六位十六进制 的文件,目录里的其他内容不会被碰。

清理脚本会反序列化缓存目录下的数据文件来读取过期时间,这点和 get() 相同。 因此缓存目录不应放在其他用户可写的位置。

LICENSE

Apache-2.0 License

Source code

https://github.com/jeeaay/python-file-cache

Metadata

Release files for filecaching 0.0.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for filecaching 0.0.5
File Size Uploaded Provenance Actions
filecaching-0.0.5.tar.gz 18.0 kB Without provenance Details for filecaching-0.0.5.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for filecaching 0.0.5
File Interpreter ABI Platform Actions
filecaching-0.0.5-py3-none-any.whl Python 3 none any Details for filecaching-0.0.5-py3-none-any.whl

Total release size: 33.1 kB

Release files / filecaching-0.0.5.tar.gz

Download URL filecaching-0.0.5.tar.gz
Size 18.0 kB
Tags Source
SHA-256 checksum
How to use checksums
09609f73cd9cf3aac880531aa4cc31fe41748b819d40d629e9d407e10be166e1
BLAKE2b-256 checksum
How to use checksums
239aaae5f0080de7d0f2a09f3f039595468ea1ba1c3b25dbb5222b5523196fbe
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.13.14

Release files / filecaching-0.0.5-py3-none-any.whl

Download URL filecaching-0.0.5-py3-none-any.whl
Size 15.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
d602da80dd00a9b98f4b8b249a80e946f62ff37e5f3d1591e407842a520518af
BLAKE2b-256 checksum
How to use checksums
7be519c96864f4348588f80d7ac942ff98581df97837045100dbe2005247000a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.13.14

Security for filecaching 0.0.5

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page