Like pickle. But different
Project description
image source.
antipickle keeps your heterogeneous data fresh
antipickle
when you want to use pickle, but you shouldn't
Why? Because pickle isn't the right way to persist or share data, and we all know that.
When it comes to practice, it takes time and effort to substitute pickle.
'Hmm, I can use json here' — I thought on many occasions, and usually was wrong.
Something small but annoying was in the way:
datetime
that can't be stored or np.array
that serializers don't know how to deal with.
Or even bytes
! And many smaller things.
At this point I either had to give up and pickle it
OR allocate time on figuring out 'how do I make this right'.
antipickle
solves this for me.
antipickle
is a restricted format for safe, persistent, and platform-independent storage.
Also, it is very convenient:
import antipickle
antipickle.dump(data, 'data.antipickle')
antipickle.dump(data, 's3://mybucket/data.antipickle') # stores in s3
antipickle.dump(data, 's3://mybucket/data.antipickle.gz') # will additionally gzip
loaded_date = antipickle.load('s3://mybucket/data.antipickle.gz') # or local file
To download/upload you need an additional dependency:
- s3:
pip install s3fs
. - gcs:
pip install gcsfs
. - ssh:
pip install sshfs
.
Batteries included:
Here is a simple example of what antipickle can save/load:
data = {
'constants': [3.1415, 2.718, True, False, 42],
'with nones': [1, None, 0],
b'bytes': b"can be stored too!",
'nested lists and tuples': [[1, [2]], (1, 2, None), {'nested': 'dict'}],
('tuple', 'as', 'key'): {'is_ok': True},
'numpy nd': np.zeros([3, 4], dtype='uint32'),
}
antipickle.dump(data, 'data.antipickle')
More formally, antipickle
supports python pieces commonly used for computations:
bytes
,str
,int
,float
,complex
,bool
, andNone
list
,tuple
,set
(all of them are stored as different entities)dict
(including integer keys andtuple
keys)numpy
arrays (native.npy
format used;dtype=O
not supported)pandas
series and dataframe (using parquet serialization viapyarrow
)polars
series and dataframe (using parquet)- Any tree-formed structure of the above (no loops allowed)
Configurable support: dataclasses
and pydantic
classes.
For reference, other non-pythonic formats (json and its binary relatives) have problems with native types
(not making difference between list and tuple) or encodings (not storing bytes)
or collections (not allowing integers, bytes and tuples in dict keys).
Antipickle is python-centric and has it solved.
Installation
pip install antipickle
What is it for
Let's set the expectation bar. antipickle is
- not fast, but isn't slow either
- not super-compact, but quite ok
- restricted: it wasn't designed to serialize just anything, it focuses on common python types and cases for data folks
At the same time, antipickle is
- safe
- persistent
- very convenient
- modular and easy to extend
and thus suitable for data sharing and data preservation.
When to (not) use pickle
pickle
is designed for interprocess communication or as a temporary storage.
pickle
has a good tradeoff of space- and time- efficiency and can serialize almost anything, including graphs with cycles.
Name pickle
suggests you could use it for long-term preservation of data, but that's not true:
pickle
's serialization is tied to an internal object representation, which is not guaranteed
to be preserved in the next release (or even on a different OS).
Developers of some packages (notably scikit-learn
) provide some guarantees about being able to parse models
that were saved with previous 1-2 minor package releases, but that's an exception not a rule.
Second, pickle
is insecure. And unreadable. And pickles can be large.
During unpickling they can do anything python can, i.e. anything at all.
So python docs say it clear: Only unpickle data you trust!
.
That said, pickle
is extremely convenient and simple to use, and works as a short-term solution for many cases,
so we all (python data folks) kinda doing that wrong pickling thing from time to time, because of convenience.
And because very few of us are ready to spend time on figuring out proper serialization.
All comments above apply to pickle
-like libs like joblib
, dill
, cloudpickle
.
License
antipickle
is distributed under the terms of the MIT license.
Other
antipickle
builds upon msgpack-python
(the only dependency).
antipickle
supports all maintained python versions (Python 3.7+)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for antipickle-0.2.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 16e437f59f24c6608629aeeca2eaf831951d718bf55fdfe3bed6be84912c91da |
|
MD5 | bf1079398757efdd36b2e9d13f1e9faa |
|
BLAKE2b-256 | 4172e86064a1c2b820eedf07bfd8d66cb5ed5063ad6b0325de3b94edb7cbd4f4 |