cattensors 0.1.0

Fast safetensors loading for distributed filesystems

cattensors

Faster safetensors loading for JuiceFS.

Note: This library was developed and benchmarked against JuiceFS, but in theory should work with any network backed distributed filesystem.

cattensors monkey-patches safetensors to pre-warm the OS page cache before tensor reads. On network backed filesystems, the first read of a large model file is slow. cattensors reads the file into page cache using parallel threads so subsequent mmap based access hits RAM instead of the network.

Compatibility

• POSIX platforms (Linux/macOS)
• Not supported on Windows (fcntl.flock is used for cross-process coordination)

Installation

pip install git+https://github.com/badayvedat/cattensors@16ebb9ab66a59747c810530c2d4ca1b53bb41539

After installation, cattensors is auto-activated at interpreter startup via a .pth file, so no code changes are required.

To disable this behavior, set CATTENSORS_AUTO_PATCH environment variable to false

[!WARNING]

Installing cattensors ships a .pth file into your site-packages. Python evaluates .pth files at every interpreter startup, which means import cattensors.startup runs in every Python process in that environment; before your code, before python -c, before pytest, etc. It then monkey-patches safetensors globally, and also flashpack if it is installed.

This is intentional and is what makes the library zero-config, but it has two implications you should know about:

1. Global side effects. Any process in the env that imports safetensors gets the patched version, even if it never imports cattensors. To opt out per-process, set CATTENSORS_AUTO_PATCH=false before Python starts. To opt out entirely, uninstall the package or remove cattensors.pth from site-packages.
2. Install-time trust. Because .pth runs unconditionally, installing this package means trusting the source you installed it from to run code in every future Python invocation in that environment. Prefer pinning to a specific commit/tag, and avoid installing into shared/system Python environments.

Configuration

Environment variables:

Variable	Default	Description
CATTENSORS_AUTO_PATCH	true	Auto-patch safetensors (and flashpack, if installed) at interpreter startup
CATTENSORS_WARM_THREADS	8	Parallel reader threads per file
CATTENSORS_PARALLEL_SHARDS	false	Pre-warm sibling shards in background when loading sharded models
HF_ENABLE_PARALLEL_LOADING	false	If true, cattensors skips sibling-shard parallel warming to avoid overlapping I/O
CATTENSORS_DEBUG	false	Enable debug logs to stderr

Behavior notes:

• CATTENSORS_AUTO_PATCH is evaluated once at interpreter startup (set it before Python starts).
• Other flags are read when used, so changes affect subsequent load calls.