Files3 - Python Object File System

1. English Version
   1. Overview
   2. Installation
   3. Quick Start
   4. Use Cases
   5. Advanced Usage
   6. In-Memory Backend
   7. Embedded Packaging
   8. CLI Commands
   9. Notice
2. Chinese Version
   1. 概述
   2. 安装
   3. 快速开始
   4. 应用场景
   5. 高级用法
   6. 内存后端
   7. 嵌入式打包
   8. 命令行工具
   9. 注意事项

Overview

A Windows-native Python object persistence library. Save any Python object to the file system with a dict-like interface. Built on cloudpickle + lz4 compression.

When to Use

Core Advantages

• Any object storage: cloudpickle handles lambdas, closures, local classes, module references
• Fast compression: lz4 enabled by default, balancing space and speed
• Source code relinking: objects defined in __main__ are auto-relinked on load even after script rename/move
• Dict-like API: f['key'], f.key, f.set() all work
• Sub-key support: one primary key can expand into multiple sub-keys, auto-converting to a folder
• Dual backend: file system (F3Shell) or shared memory (F3Mem) with identical APIs

Not Recommended For

• Cross-platform data exchange (Windows only)
• High-concurrency write scenarios (no locking, relies on filesystem atomicity)
• Massive key-value stores (hundreds of thousands+ keys, filesystem inode bottleneck)
• Complex querying requiring SQL-like search

Installation

pip install files3

After installation, associate a file extension with the f3open viewer:

f3assoc .ist

Quick Start

from files3 import files
<p>f = files('./data')  # workspace directory, default suffix '.ist'</p>
<h1 id="user-content-save">Save<a href="#user-content-save" aria-label="Link to heading 'Save'" data-heading-content="Save" class="anchor"></a></h1>
<p>f.set('model', {'weights': [0.1, 0.2], 'epoch': 10})</p>
<h1 id="user-content-load">Load<a href="#user-content-load" aria-label="Link to heading 'Load'" data-heading-content="Load" class="anchor"></a></h1>
<p>print(f.get('model'))  # {'weights': [0.1, 0.2], 'epoch': 10}</p>
<h1 id="user-content-check">Check<a href="#user-content-check" aria-label="Link to heading 'Check'" data-heading-content="Check" class="anchor"></a></h1>
<p>print(f.has('model'))  # True</p>
<h1 id="user-content-delete">Delete<a href="#user-content-delete" aria-label="Link to heading 'Delete'" data-heading-content="Delete" class="anchor"></a></h1>
<p>f.delete('model')

Use Cases

1. Config Persistence

from files3 import files
<p>f = files('./config')</p>
<h1 id="user-content-save-a-complex-config-with-custom-classes">Save a complex config with custom classes<a href="#user-content-save-a-complex-config-with-custom-classes" aria-label="Link to heading 'Save a complex config with custom classes'" data-heading-content="Save a complex config with custom classes" class="anchor"></a></h1>
<p>f['app_cfg'] = {
'lr_scheduler': lambda epoch: 0.1 ** (epoch // 10),  # lambda is fine
'model_cls': MyModel,  # class reference is fine
'layers': [64, 128, 256],
}</p>
<h1 id="user-content-load-it-later-even-in-another-script">Load it later (even in another script)<a href="#user-content-load-it-later-even-in-another-script" aria-label="Link to heading 'Load it later (even in another script)'" data-heading-content="Load it later (even in another script)" class="anchor"></a></h1>
<p>cfg = f['app_cfg']

2. Function Result Cache

from files3 import files
<p>f = files('./cache')</p>
<p>def expensive_compute(x):
key = f'compute_{x}'
if f.has(key):
return f[key]
result = sum(i ** 2 for i in range(x))
f[key] = result
return result

3. Data Exchange Between Processes

# script_a.py
from files3 import files
f = files('./shared')
f['model'] = trained_model
<h1 id="user-content-script_bpy">script_b.py<a href="#user-content-script_bpy" aria-label="Link to heading 'script_b.py'" data-heading-content="script_b.py" class="anchor"></a></h1>
<p>from files3 import files
f = files('./shared')
model = f['model']

4. Batch Operations with Filter Syntax

import re
<p>f = files('./data')</p>
<h1 id="user-content-delete-all-keys-starting-with-temp_">Delete all keys starting with 'temp_'<a href="#user-content-delete-all-keys-starting-with-temp_" aria-label="Link to heading 'Delete all keys starting with 'temp_''" data-heading-content="Delete all keys starting with 'temp_'" class="anchor"></a></h1>
<p>del f[re.compile(r'^temp_')]</p>
<h1 id="user-content-set-multiple-keys-at-once">Set multiple keys at once<a href="#user-content-set-multiple-keys-at-once" aria-label="Link to heading 'Set multiple keys at once'" data-heading-content="Set multiple keys at once" class="anchor"></a></h1>
<p>f['a', 'b', 'c'] = 100</p>
<h1 id="user-content-delete-by-custom-filter">Delete by custom filter<a href="#user-content-delete-by-custom-filter" aria-label="Link to heading 'Delete by custom filter'" data-heading-content="Delete by custom filter" class="anchor"></a></h1>
<p>del f[lambda name, ftype: name.startswith('old_')]</p>
<h1 id="user-content-clear-everything">Clear everything<a href="#user-content-clear-everything" aria-label="Link to heading 'Clear everything'" data-heading-content="Clear everything" class="anchor"></a></h1>
<p>del f[...]

Advanced Usage

Dict-Style Access

f = files('./data')
<h1 id="user-content-write">Write<a href="#user-content-write" aria-label="Link to heading 'Write'" data-heading-content="Write" class="anchor"></a></h1>
<p>f.a = 1          # same as f.set('a', 1, error=True)
f['b'] = 2       # same as above
f['c', 'data'] = [1, 2, 3]  # sub-key</p>
<h1 id="user-content-read">Read<a href="#user-content-read" aria-label="Link to heading 'Read'" data-heading-content="Read" class="anchor"></a></h1>
<p>print(f.a)       # same as f.get('a', error=False)
print(f['b'])    # same as f.get('b', error=True)</p>
<h1 id="user-content-delete-1">Delete<a href="#user-content-delete-1" aria-label="Link to heading 'Delete'" data-heading-content="Delete" class="anchor"></a></h1>
<p>del f.a
del f['b']</p>
<h1 id="user-content-check-1">Check<a href="#user-content-check-1" aria-label="Link to heading 'Check'" data-heading-content="Check" class="anchor"></a></h1>
<p>'a' in f         # same as f.has('a', error=True)
len(f)           # count of primary keys

Sub-Keys

One primary key can hold multiple sub-keys. The primary key automatically becomes a folder.

f = files('./data')
f.set('user', {'name': 'alice'})           # user.ist (file)
f.set('user', {'age': 30}, skey='age')      # user.ist/ (folder)
                                            #   _.ist   original content
                                            #   age.ist new content
<p>f['user', '<em>']     # read default sub-key
f['user', 'age']   # read age sub-key
f.list('user')     # ['</em>', 'age']

Serialization Tools

from files3 import files
<h1 id="user-content-serialize-to-bytes">Serialize to bytes<a href="#user-content-serialize-to-bytes" aria-label="Link to heading 'Serialize to bytes'" data-heading-content="Serialize to bytes" class="anchor"></a></h1>
<p>b = files.dumps({'data': [1, 2, 3]})
obj = files.loads(b)</p>
<h1 id="user-content-pack-a-filedirectory-into-bytes">Pack a file/directory into bytes<a href="#user-content-pack-a-filedirectory-into-bytes" aria-label="Link to heading 'Pack a file/directory into bytes'" data-heading-content="Pack a file/directory into bytes" class="anchor"></a></h1>
<p>b = files.pack(r'C:\my_resource')
files.unpack(b, r'C:\extract_to')</p>
<h1 id="user-content-pack-into-a-python-module-no-files3-dependency-to-unpack">Pack into a Python module (no files3 dependency to unpack)<a href="#user-content-pack-into-a-python-module-no-files3-dependency-to-unpack" aria-label="Link to heading 'Pack into a Python module (no files3 dependency to unpack)'" data-heading-content="Pack into a Python module (no files3 dependency to unpack)" class="anchor"></a></h1>
<p>code = files.packpy(r'C:\my_resource')
with open('resource.py', 'w') as fh:
fh.write(code)</p>
<h1 id="user-content-unpack-from-the-module">Unpack from the module<a href="#user-content-unpack-from-the-module" aria-label="Link to heading 'Unpack from the module'" data-heading-content="Unpack from the module" class="anchor"></a></h1>
<p>from resource import F3DATA
files.unpackpy(F3DATA, r'C:\extract_to')

In-Memory Backend (F3Mem)

Zero disk IO, cross-process sharing, lost on reboot. Identical API to files.

from files3 import memfiles
<p>m = memfiles('my_ns')
m['key'] = {'speed': 'fast'}
m['key', 'sub'] = 'zero_disk_io'</p>
<h1 id="user-content-persist-to-disk">Persist to disk<a href="#user-content-persist-to-disk" aria-label="Link to heading 'Persist to disk'" data-heading-content="Persist to disk" class="anchor"></a></h1>
<p>m.save('./backup')</p>
<h1 id="user-content-load-from-disk">Load from disk<a href="#user-content-load-from-disk" aria-label="Link to heading 'Load from disk'" data-heading-content="Load from disk" class="anchor"></a></h1>
<p>m2 = memfiles('another_ns')
m2.load('./backup')</p>
<h1 id="user-content-cleanup">Cleanup<a href="#user-content-cleanup" aria-label="Link to heading 'Cleanup'" data-heading-content="Cleanup" class="anchor"></a></h1>
<p>m.clear()

Embedded Packaging

Pack any file or directory into a .py file so it can travel with your source code. No database or extra dependency needed on the receiving end.

Pack to a .py file

from files3 import prefab
<h1 id="user-content-pack-cmy_data-file-or-folder-into-cmy_datapy">Pack C:\my_data (file or folder) into C:\my_data.py<a href="#user-content-pack-cmy_data-file-or-folder-into-cmy_datapy" aria-label="Link to heading 'Pack C:\my_data (file or folder) into C:\my_data.py'" data-heading-content="Pack C:\my_data (file or folder) into C:\my_data.py" class="anchor"></a></h1>
<p>prefab.aspy(r'C:\my_data')

What happens:

1. Zip-compresses the target.
2. Converts the zip bytes into a Python bytes literal.
3. Writes the literal into C:\my_data.py as variable F3DATA.

Auto-extract on first run

Use astarget in your script. If the target is missing, it extracts from the adjacent .py file automatically.

from files3 import prefab
<h1 id="user-content-if-cmy_data-does-not-exist-extract-from-cmy_datapy">If C:\my_data does not exist, extract from C:\my_data.py<a href="#user-content-if-cmy_data-does-not-exist-extract-from-cmy_datapy" aria-label="Link to heading 'If C:\my_data does not exist, extract from C:\my_data.py'" data-heading-content="If C:\my_data does not exist, extract from C:\my_data.py" class="anchor"></a></h1>
<p>prefab.astarget(r'C:\my_data')

Low-level API

If you need the code string in memory instead of a file:

from files3 import files
<h1 id="user-content-returns-a-python-code-string-contains-f3data-variable">Returns a Python code string (contains F3DATA variable)<a href="#user-content-returns-a-python-code-string-contains-f3data-variable" aria-label="Link to heading 'Returns a Python code string (contains F3DATA variable)'" data-heading-content="Returns a Python code string (contains F3DATA variable)" class="anchor"></a></h1>
<p>code = files.packpy(r'C:\my_data')</p>
<h1 id="user-content-later-feed-the-string-back-to-unpack">Later, feed the string back to unpack<a href="#user-content-later-feed-the-string-back-to-unpack" aria-label="Link to heading 'Later, feed the string back to unpack'" data-heading-content="Later, feed the string back to unpack" class="anchor"></a></h1>
<p>files.unpackpy(code, r'C:\extract_to')

CLI Commands

f3 [name] [type] -d [dir]   # open a files3 object interactively
f3open [filepath]             # open a single .ist file
f3assoc [type]                # associate file extension with f3open
f3unassoc [type]              # remove file association

Notice

• Security: pickle is not safe. Do not loads() data from untrusted sources.
• Cannot save: F3Bool instances, F3Shell instances, active exception objects, generators, open file handles, and some C-extension objects.
• Windows only: Relies on Win32 APIs for file associations and folder icons.
• ModuleNotFoundError on load: If the source script was moved/renamed, use f.relink(new_path, 'key') to fix.

概述

Windows原生Python对象持久化库。以文件系统为后端，将任意Python对象序列化存储，提供类字典的交互接口。基于 cloudpickle + lz4 压缩。

适用场景

核心优势

• 任意对象存储：基于cloudpickle，支持lambda、闭包、局部类、模块引用等标准pickle无法处理的类型
• lz4压缩：默认启用高速压缩，平衡存储空间和读写性能
• 源代码重链接：保存__main__中定义的类/函数时，自动记录源文件路径，加载时自动修正模块名
• 类字典接口：支持f['key']、f.key、f.set()等多种交互方式
• 子键支持：单主键可扩展为多子键，主键自动转为文件夹管理
• 双后端：文件系统（F3Shell）或共享内存（F3Mem），API完全一致

不推荐的场景

• 跨平台数据交换（仅支持Windows）
• 高并发写入场景（无锁机制，依赖文件系统原子性）
• 超大规模键值存储（数十万级以上，文件系统inode成为瓶颈）
• 需要SQL查询的复杂检索场景

安装

pip install files3

安装后，可在cmd中将文件后缀关联到f3open查看器：

f3assoc .ist

快速开始

from files3 import files
<p>f = files('./data')  # 工作目录，默认后缀 '.ist'</p>
<h1 id="user-content-保存">保存<a href="#user-content-保存" aria-label="Link to heading '保存'" data-heading-content="保存" class="anchor"></a></h1>
<p>f.set('model', {'weights': [0.1, 0.2], 'epoch': 10})</p>
<h1 id="user-content-读取">读取<a href="#user-content-读取" aria-label="Link to heading '读取'" data-heading-content="读取" class="anchor"></a></h1>
<p>print(f.get('model'))  # {'weights': [0.1, 0.2], 'epoch': 10}</p>
<h1 id="user-content-检查">检查<a href="#user-content-检查" aria-label="Link to heading '检查'" data-heading-content="检查" class="anchor"></a></h1>
<p>print(f.has('model'))  # True</p>
<h1 id="user-content-删除">删除<a href="#user-content-删除" aria-label="Link to heading '删除'" data-heading-content="删除" class="anchor"></a></h1>
<p>f.delete('model')

应用场景

1. 配置持久化

from files3 import files
<p>f = files('./config')</p>
<h1 id="user-content-保存含自定义类的复杂配置">保存含自定义类的复杂配置<a href="#user-content-保存含自定义类的复杂配置" aria-label="Link to heading '保存含自定义类的复杂配置'" data-heading-content="保存含自定义类的复杂配置" class="anchor"></a></h1>
<p>f['app_cfg'] = {
'lr_scheduler': lambda epoch: 0.1 ** (epoch // 10),  # lambda没问题
'model_cls': MyModel,  # 类引用没问题
'layers': [64, 128, 256],
}</p>
<h1 id="user-content-之后读取甚至另一个脚本">之后读取（甚至另一个脚本）<a href="#user-content-之后读取甚至另一个脚本" aria-label="Link to heading '之后读取（甚至另一个脚本）'" data-heading-content="之后读取（甚至另一个脚本）" class="anchor"></a></h1>
<p>cfg = f['app_cfg']

2. 函数结果缓存

from files3 import files
<p>f = files('./cache')</p>
<p>def expensive_compute(x):
key = f'compute_{x}'
if f.has(key):
return f[key]
result = sum(i ** 2 for i in range(x))
f[key] = result
return result

3. 跨进程数据交换

# script_a.py
from files3 import files
f = files('./shared')
f['model'] = trained_model
<h1 id="user-content-script_bpy-1">script_b.py<a href="#user-content-script_bpy-1" aria-label="Link to heading 'script_b.py'" data-heading-content="script_b.py" class="anchor"></a></h1>
<p>from files3 import files
f = files('./shared')
model = f['model']

4. 批量筛选操作

import re
<p>f = files('./data')</p>
<h1 id="user-content-删除所有以temp_开头的键">删除所有以'temp_'开头的键<a href="#user-content-删除所有以temp_开头的键" aria-label="Link to heading '删除所有以'temp_'开头的键'" data-heading-content="删除所有以'temp_'开头的键" class="anchor"></a></h1>
<p>del f[re.compile(r'^temp_')]</p>
<h1 id="user-content-同时设置多个键">同时设置多个键<a href="#user-content-同时设置多个键" aria-label="Link to heading '同时设置多个键'" data-heading-content="同时设置多个键" class="anchor"></a></h1>
<p>f['a', 'b', 'c'] = 100</p>
<h1 id="user-content-自定义条件删除">自定义条件删除<a href="#user-content-自定义条件删除" aria-label="Link to heading '自定义条件删除'" data-heading-content="自定义条件删除" class="anchor"></a></h1>
<p>del f[lambda name, ftype: name.startswith('old_')]</p>
<h1 id="user-content-清空全部">清空全部<a href="#user-content-清空全部" aria-label="Link to heading '清空全部'" data-heading-content="清空全部" class="anchor"></a></h1>
<p>del f[...]

高级用法

字典式访问

f = files('./data')
<h1 id="user-content-写入">写入<a href="#user-content-写入" aria-label="Link to heading '写入'" data-heading-content="写入" class="anchor"></a></h1>
<p>f.a = 1          # 等价 f.set('a', 1, error=True)
f['b'] = 2       # 等价同上
f['c', 'data'] = [1, 2, 3]  # 子键</p>
<h1 id="user-content-读取-1">读取<a href="#user-content-读取-1" aria-label="Link to heading '读取'" data-heading-content="读取" class="anchor"></a></h1>
<p>print(f.a)       # 等价 f.get('a', error=False)
print(f['b'])    # 等价 f.get('b', error=True)</p>
<h1 id="user-content-删除-1">删除<a href="#user-content-删除-1" aria-label="Link to heading '删除'" data-heading-content="删除" class="anchor"></a></h1>
<p>del f.a
del f['b']</p>
<h1 id="user-content-检查-1">检查<a href="#user-content-检查-1" aria-label="Link to heading '检查'" data-heading-content="检查" class="anchor"></a></h1>
<p>'a' in f         # 等价 f.has('a', error=True)
len(f)           # 主键数量

子键

一个主键下可存储多个子键，主键自动变为文件夹。

f = files('./data')
f.set('user', {'name': 'alice'})           # user.ist（文件）
f.set('user', {'age': 30}, skey='age')      # user.ist/（文件夹）
                                            #   _.ist   原内容
                                            #   age.ist 新内容
<p>f['user', '<em>']     # 读取默认子键
f['user', 'age']   # 读取age子键
f.list('user')     # ['</em>', 'age']

序列化工具

from files3 import files
<h1 id="user-content-序列化为bytes">序列化为bytes<a href="#user-content-序列化为bytes" aria-label="Link to heading '序列化为bytes'" data-heading-content="序列化为bytes" class="anchor"></a></h1>
<p>b = files.dumps({'data': [1, 2, 3]})
obj = files.loads(b)</p>
<h1 id="user-content-将文件目录打包为bytes">将文件/目录打包为bytes<a href="#user-content-将文件目录打包为bytes" aria-label="Link to heading '将文件/目录打包为bytes'" data-heading-content="将文件/目录打包为bytes" class="anchor"></a></h1>
<p>b = files.pack(r'C:\my_resource')
files.unpack(b, r'C:\extract_to')</p>
<h1 id="user-content-打包为python模块解压时无需安装files3">打包为Python模块（解压时无需安装files3）<a href="#user-content-打包为python模块解压时无需安装files3" aria-label="Link to heading '打包为Python模块（解压时无需安装files3）'" data-heading-content="打包为Python模块（解压时无需安装files3）" class="anchor"></a></h1>
<p>code = files.packpy(r'C:\my_resource')
with open('resource.py', 'w') as fh:
fh.write(code)</p>
<h1 id="user-content-从模块解压">从模块解压<a href="#user-content-从模块解压" aria-label="Link to heading '从模块解压'" data-heading-content="从模块解压" class="anchor"></a></h1>
<p>from resource import F3DATA
files.unpackpy(F3DATA, r'C:\extract_to')

内存后端（F3Mem）

零磁盘IO、跨进程共享、重启丢失。接口与files完全一致。

from files3 import memfiles
<p>m = memfiles('my_ns')
m['key'] = {'speed': 'fast'}
m['key', 'sub'] = 'zero_disk_io'</p>
<h1 id="user-content-持久化到磁盘">持久化到磁盘<a href="#user-content-持久化到磁盘" aria-label="Link to heading '持久化到磁盘'" data-heading-content="持久化到磁盘" class="anchor"></a></h1>
<p>m.save('./backup')</p>
<h1 id="user-content-从磁盘加载">从磁盘加载<a href="#user-content-从磁盘加载" aria-label="Link to heading '从磁盘加载'" data-heading-content="从磁盘加载" class="anchor"></a></h1>
<p>m2 = memfiles('another_ns')
m2.load('./backup')</p>
<h1 id="user-content-清理">清理<a href="#user-content-清理" aria-label="Link to heading '清理'" data-heading-content="清理" class="anchor"></a></h1>
<p>m.clear()

嵌入式打包

将任意文件或目录打包成 .py 文件，随源代码一起分发。接收端无需数据库或其他依赖。

打包为 .py 文件

from files3 import prefab
<h1 id="user-content-将-cmy_data文件或目录打包为-cmy_datapy">将 C:\my_data（文件或目录）打包为 C:\my_data.py<a href="#user-content-将-cmy_data文件或目录打包为-cmy_datapy" aria-label="Link to heading '将 C:\my_data（文件或目录）打包为 C:\my_data.py'" data-heading-content="将 C:\my_data（文件或目录）打包为 C:\my_data.py" class="anchor"></a></h1>
<p>prefab.aspy(r'C:\my_data')

内部流程：

1. 将目标 zip 压缩。
2. 将 zip 字节流转换为 Python bytes 字面量。
3. 写入 C:\my_data.py，变量名为 F3DATA。

首次运行时自动解压

在脚本中使用 astarget。如果目标不存在，自动从相邻的 .py 文件解压。

from files3 import prefab
<h1 id="user-content-若-cmy_data-不存在则从-cmy_datapy-解压">若 C:\my_data 不存在，则从 C:\my_data.py 解压<a href="#user-content-若-cmy_data-不存在则从-cmy_datapy-解压" aria-label="Link to heading '若 C:\my_data 不存在，则从 C:\my_data.py 解压'" data-heading-content="若 C:\my_data 不存在，则从 C:\my_data.py 解压" class="anchor"></a></h1>
<p>prefab.astarget(r'C:\my_data')

底层 API

如果需要字符串而非直接生成文件：

from files3 import files
<h1 id="user-content-返回-python-代码字符串内含-f3data-变量">返回 Python 代码字符串（内含 F3DATA 变量）<a href="#user-content-返回-python-代码字符串内含-f3data-变量" aria-label="Link to heading '返回 Python 代码字符串（内含 F3DATA 变量）'" data-heading-content="返回 Python 代码字符串（内含 F3DATA 变量）" class="anchor"></a></h1>
<p>code = files.packpy(r'C:\my_data')</p>
<h1 id="user-content-后续传入字符串解压">后续传入字符串解压<a href="#user-content-后续传入字符串解压" aria-label="Link to heading '后续传入字符串解压'" data-heading-content="后续传入字符串解压" class="anchor"></a></h1>
<p>files.unpackpy(code, r'C:\extract_to')

命令行工具

f3 [name] [type] -d [dir]   # 交互式打开files3对象
f3open [filepath]             # 打开单个.ist文件
f3assoc [type]                # 关联文件后缀到f3open
f3unassoc [type]              # 移除文件关联

注意事项

• 安全性：pickle不安全，不要对不信任的数据调用loads()。
• 无法保存：F3Bool实例、F3Shell实例、活动异常对象、生成器、打开的文件句柄、某些C扩展对象。
• 仅限Windows：依赖Win32 API实现文件关联和文件夹图标设置。
• 加载时ModuleNotFoundError：如果源脚本被移动/重命名，使用f.relink(new_path, 'key')修复。