Pickle pandas DataFrames without AttributeError: Can't pickle local object ...
Project description
Pickle pandas DataFrames without "AttributeError: Can't pickle local object ... "
pip install a-pandas-ex-dillpickle
import pandas as pd
from a_pandas_ex_dillpickle import pd_add_dillpickle
# Good example, because dirdf includes lambda functions: https://github.com/hansalemaos/DirDF
from dirdf import pd_add_dfdir
pd_add_dfdir()
pd_add_dillpickle()
df2 = pd.Q_folder_to_df_with_functions(
folder=r"F:\30000",
ls_path="ls",
last_access_time=True,
exit_keys="ctrl+x",
timeout=None,
strings_path="strings",
fzf_path="fzf",
rip_grep_path="rg.exe",
add_flatcopy_sorted=True,
add_flatcopy=True,
add_extract_strings=True,
add_fuzzy_extract=True,
add_ripgrep=True,
add_open_file=True,
add_move_file=True,
)
print(df2)
aa_date ... ff_flatcopy
0 2021-03-08 03:41:41.239919600-03:00 ... dest_folder:str, foldersep:str='ǀ', symlink:bo...
1 2021-03-08 03:41:44.653175900-03:00 ... dest_folder:str, foldersep:str='ǀ', symlink:bo...
2 2021-03-08 03:41:47.120014100-03:00 ... dest_folder:str, foldersep:str='ǀ', symlink:bo...
3 2021-03-08 04:01:53.086047300-03:00 ... dest_folder:str, foldersep:str='ǀ', symlink:bo...
4 2021-03-08 05:11:38.713301-03:00 ... dest_folder:str, foldersep:str='ǀ', symlink:bo...
5 2021-03-08 05:12:22.607746500-03:00 ... dest_folder:str, foldersep:str='ǀ', symlink:bo...
6 2021-03-08 05:32:22.251188300-03:00 ... dest_folder:str, foldersep:str='ǀ', symlink:bo...
[7 rows x 21 columns]
# regular pandas version
df2.to_pickle('f:\\testpi.pkl')
Traceback (most recent call last):
File "C:\Users\Gamer\anaconda3\envs\dfdir\lib\site-packages\IPython\core\interactiveshell.py", line 3398, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-3-b78c152ff8d4>", line 27, in <cell line: 27>
df2.to_pickle('f:\\testpi.pkl')
File "C:\Users\Gamer\anaconda3\envs\dfdir\lib\site-packages\pandas\core\generic.py", line 3064, in to_pickle
to_pickle(
File "C:\Users\Gamer\anaconda3\envs\dfdir\lib\site-packages\pandas\io\pickle.py", line 112, in to_pickle
pickle.dump(obj, handles.handle, protocol=protocol)
AttributeError: Can't pickle local object 'copy_func.<locals>.<lambda>'
# dillpickle - uses https://github.com/uqfoundation/dill
df2.to_dillpickle('f:\\dillpi.pkl')
# The file can be loaded using dill or pickle
df3=pd.read_dillpickle('f:\\dillpi.pkl')
df3=pd.read_pickle('f:\\dillpi.pkl')
print(df3)
aa_date ... ff_flatcopy
0 2021-03-08 03:41:41.239919600-03:00 ... dest_folder:str, foldersep:str='ǀ', symlink:bo...
1 2021-03-08 03:41:44.653175900-03:00 ... dest_folder:str, foldersep:str='ǀ', symlink:bo...
2 2021-03-08 03:41:47.120014100-03:00 ... dest_folder:str, foldersep:str='ǀ', symlink:bo...
3 2021-03-08 04:01:53.086047300-03:00 ... dest_folder:str, foldersep:str='ǀ', symlink:bo...
4 2021-03-08 05:11:38.713301-03:00 ... dest_folder:str, foldersep:str='ǀ', symlink:bo...
5 2021-03-08 05:12:22.607746500-03:00 ... dest_folder:str, foldersep:str='ǀ', symlink:bo...
6 2021-03-08 05:32:22.251188300-03:00 ... dest_folder:str, foldersep:str='ǀ', symlink:bo...
[7 rows x 21 columns]
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file a_pandas_ex_dillpickle-0.10.tar.gz
.
File metadata
- Download URL: a_pandas_ex_dillpickle-0.10.tar.gz
- Upload date:
- Size: 4.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.9.13
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 36deb155ba65f16b03c78348811359b83d5036601690cb0e30fff944f50b8454 |
|
MD5 | 13575a426a85995858685ca8ea175f55 |
|
BLAKE2b-256 | f69838a3a327c55f406e40e18564d2dc9c7d5ba82b66ddb079b7f7651933f25d |
Provenance
File details
Details for the file a_pandas_ex_dillpickle-0.10-py3-none-any.whl
.
File metadata
- Download URL: a_pandas_ex_dillpickle-0.10-py3-none-any.whl
- Upload date:
- Size: 5.4 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.9.13
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | ab5d934bd401e49c59714f27fc72b305f80a34150e357853340ec81cfc9f374a |
|
MD5 | 96753d79546a68d3a4d8ea0e9fbacfac |
|
BLAKE2b-256 | a07b45f466df237c837e0d2ffb80f27f092b4b61b47b2a56565a1f4114d3ed20 |