Skip to main content

Pickle pandas DataFrames without AttributeError: Can't pickle local object ...

Project description

Pickle pandas DataFrames without "AttributeError: Can't pickle local object ... "

pip install a-pandas-ex-dillpickle

import pandas as pd

from a_pandas_ex_dillpickle import pd_add_dillpickle



# Good example, because dirdf includes lambda functions: https://github.com/hansalemaos/DirDF

from dirdf import pd_add_dfdir

pd_add_dfdir()

pd_add_dillpickle()

df2 = pd.Q_folder_to_df_with_functions(

    folder=r"F:\30000",

    ls_path="ls",

    last_access_time=True,

    exit_keys="ctrl+x",

    timeout=None,

    strings_path="strings",

    fzf_path="fzf",

    rip_grep_path="rg.exe",

    add_flatcopy_sorted=True,

    add_flatcopy=True,

    add_extract_strings=True,

    add_fuzzy_extract=True,

    add_ripgrep=True,

    add_open_file=True,

    add_move_file=True,

)

print(df2)



                              aa_date  ...                                        ff_flatcopy

0 2021-03-08 03:41:41.239919600-03:00  ...  dest_folder:str, foldersep:str='ǀ', symlink:bo...

1 2021-03-08 03:41:44.653175900-03:00  ...  dest_folder:str, foldersep:str='ǀ', symlink:bo...

2 2021-03-08 03:41:47.120014100-03:00  ...  dest_folder:str, foldersep:str='ǀ', symlink:bo...

3 2021-03-08 04:01:53.086047300-03:00  ...  dest_folder:str, foldersep:str='ǀ', symlink:bo...

4    2021-03-08 05:11:38.713301-03:00  ...  dest_folder:str, foldersep:str='ǀ', symlink:bo...

5 2021-03-08 05:12:22.607746500-03:00  ...  dest_folder:str, foldersep:str='ǀ', symlink:bo...

6 2021-03-08 05:32:22.251188300-03:00  ...  dest_folder:str, foldersep:str='ǀ', symlink:bo...

[7 rows x 21 columns]



# regular pandas version

df2.to_pickle('f:\\testpi.pkl')



Traceback (most recent call last):

  File "C:\Users\Gamer\anaconda3\envs\dfdir\lib\site-packages\IPython\core\interactiveshell.py", line 3398, in run_code

    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-3-b78c152ff8d4>", line 27, in <cell line: 27>

    df2.to_pickle('f:\\testpi.pkl')

  File "C:\Users\Gamer\anaconda3\envs\dfdir\lib\site-packages\pandas\core\generic.py", line 3064, in to_pickle

    to_pickle(

  File "C:\Users\Gamer\anaconda3\envs\dfdir\lib\site-packages\pandas\io\pickle.py", line 112, in to_pickle

    pickle.dump(obj, handles.handle, protocol=protocol)

AttributeError: Can't pickle local object 'copy_func.<locals>.<lambda>'



# dillpickle - uses https://github.com/uqfoundation/dill



df2.to_dillpickle('f:\\dillpi.pkl')



# The file can be loaded using dill or pickle 

df3=pd.read_dillpickle('f:\\dillpi.pkl')

df3=pd.read_pickle('f:\\dillpi.pkl')

print(df3)

                              aa_date  ...                                        ff_flatcopy

0 2021-03-08 03:41:41.239919600-03:00  ...  dest_folder:str, foldersep:str='ǀ', symlink:bo...

1 2021-03-08 03:41:44.653175900-03:00  ...  dest_folder:str, foldersep:str='ǀ', symlink:bo...

2 2021-03-08 03:41:47.120014100-03:00  ...  dest_folder:str, foldersep:str='ǀ', symlink:bo...

3 2021-03-08 04:01:53.086047300-03:00  ...  dest_folder:str, foldersep:str='ǀ', symlink:bo...

4    2021-03-08 05:11:38.713301-03:00  ...  dest_folder:str, foldersep:str='ǀ', symlink:bo...

5 2021-03-08 05:12:22.607746500-03:00  ...  dest_folder:str, foldersep:str='ǀ', symlink:bo...

6 2021-03-08 05:32:22.251188300-03:00  ...  dest_folder:str, foldersep:str='ǀ', symlink:bo...

[7 rows x 21 columns]

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

a_pandas_ex_dillpickle-0.10.tar.gz (4.1 kB view details)

Uploaded Source

Built Distribution

a_pandas_ex_dillpickle-0.10-py3-none-any.whl (5.4 kB view details)

Uploaded Python 3

File details

Details for the file a_pandas_ex_dillpickle-0.10.tar.gz.

File metadata

  • Download URL: a_pandas_ex_dillpickle-0.10.tar.gz
  • Upload date:
  • Size: 4.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.9.13

File hashes

Hashes for a_pandas_ex_dillpickle-0.10.tar.gz
Algorithm Hash digest
SHA256 36deb155ba65f16b03c78348811359b83d5036601690cb0e30fff944f50b8454
MD5 13575a426a85995858685ca8ea175f55
BLAKE2b-256 f69838a3a327c55f406e40e18564d2dc9c7d5ba82b66ddb079b7f7651933f25d

See more details on using hashes here.

Provenance

File details

Details for the file a_pandas_ex_dillpickle-0.10-py3-none-any.whl.

File metadata

File hashes

Hashes for a_pandas_ex_dillpickle-0.10-py3-none-any.whl
Algorithm Hash digest
SHA256 ab5d934bd401e49c59714f27fc72b305f80a34150e357853340ec81cfc9f374a
MD5 96753d79546a68d3a4d8ea0e9fbacfac
BLAKE2b-256 a07b45f466df237c837e0d2ffb80f27f092b4b61b47b2a56565a1f4114d3ed20

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page