pure-safetensors 0.1.0

Pure-Python safetensors

pure_safetensors

Safetensors library but in pure clean Python. Run it on PyPy or IronPython or wherever.

Dependencies

We try to keep dependencies light:

• attrs dataclass library (2881 LoC)
• marshmallow serialization and validation library (2647 LoC)
• sortedcollections tiny sorted collections library (339 LoC) built on top of sortedcontainers (1493 LoC)
• (optional) sparsefile sparse file library (191 LoC)

Optionally, this library integrates with NumPy (if available). PyTorch integration is planned.

To run the tests, you'll need pytest, numpy, and optionally hypothesis.

Examples

TODO

Bugs

The space allocator is a greedy algorithm based on first-fit-decreasing bin packing. So if you add/remove tensors to an existing file, it may leave too much empty space behind.

Alternatives

• safetensors
• Narsil/pure_torch.py
• Narsil/safetensors.cpp

	pure_safetensors	safetensors	pure_torch.py	safetensors.cpp
Written in pure Python?	✅	❌	✅	—
Supports NumPy directly (no PyTorch)?	✅	✅	❌	—
Can write safetensors files?	✅	✅	❌	❌
Can add/delete tensors in-place (no copying)?	✅	❌	❌	❌
Has unit tests?	✅	✅	❌	❌
Stable API?	🤷	✅	❔	❔
Automatically make files sparse to save space?	✅	❌	❌	❌