Pure-Python safetensors
Project description
pure_safetensors
Safetensors library but in pure clean Python. Run it on PyPy or IronPython or wherever.
Dependencies
We try to keep dependencies light:
- attrs dataclass library (2881 LoC)
- marshmallow serialization and validation library (2647 LoC)
- sortedcollections tiny sorted collections library (339 LoC) built on top of sortedcontainers (1493 LoC)
- (optional) sparsefile sparse file library (191 LoC)
Optionally, this library integrates with NumPy (if available). PyTorch integration is planned, someday.
To run the tests, you'll need pytest
, numpy
, and optionally hypothesis
.
Examples
from pure_safetensors import SafeTensors
with SafeTensors("/path/to/example.safetensors", "r+") as sf, sf.as_numpy() as arrays:
arrays["hello"][3, :] += 420.69
arrays["world"] = arrays["hello"][0:2] * 10
# assign multiple arrays! much faster!
arrays.update(
{
"q": my_array_1,
"k": my_array_2,
"v": my_array_3,
}
)
# delete arrays! such wonders!
del arrays["v"]
Bugs
The space allocator is a greedy algorithm based on first-fit-decreasing bin packing. So if you add/remove tensors to an existing file, it may leave too much empty space behind.
PyTorch support isn't implemented yet.
Alternatives
pure_safetensors | safetensors | pure_torch.py | safetensors.cpp | |
---|---|---|---|---|
Written in pure Python? | ✅ | ❌ | ✅ | — |
Supports NumPy (without PyTorch)? | ✅ | ✅ | ❌ | — |
Can work without numpy or pytorch? | ✅ | ✅ | ❌ | — |
Can write safetensors files? | ✅ | ✅ | ❌ | ❌ |
Can modify file in-place to add/remove tensors? | ✅ | ❌ | ❌ | ❌ |
Has test suite? | ✅ | ✅ | ❌ | ❌ |
Stable API? | 🤷 | ✅ | ❔ | ❔ |
Automatically makes files sparse to save space? | ✅ | ❌ | ❌ | ❌ |
Works on platforms without mmap? | ✅ | ❌ | ❌ | ❌ |
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pure_safetensors-0.2.0.tar.gz
(13.2 kB
view hashes)
Built Distribution
Close
Hashes for pure_safetensors-0.2.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d211161c8f030c7c390023531dc478f8013cdfbbd9f2730866dffad2082f6c66 |
|
MD5 | 9c24e5d566b082e61911e165f8f38ee2 |
|
BLAKE2b-256 | 49632b20aa0c2ffe9fbfd58d150a4a6b4f07a4ac2c5b814f3ecf41c845d91d32 |