Skip to main content

A helper class that makes appending to a Pandas DataFrame efficient

Project description

pandas-appender

Build Status Coverage Apache License 2.0

Have you ever wanted to append a bunch of rows to a Pandas DataFrame? Turns out that it's extremely inefficient to do! For a large dataframe, you're supposed to make multiple dataframes and pd.concat() them instead.

Also, Pandas deprecated dataframe.append() in version 1.4 and intends to remove it in 2.0.

So... helper function? Pandas doesn't have one. Roll your own? Ugh. OK then: here's that helper function. It can append around 1 million very small rows per cpu-second. It has a modest additional memory usage of around 5 megabytes, dynamically growing with the number of rows appended.

Install

pip install pandas-appender

Usage

from pandas_appender import DF_Appender

dfa = DF_Appender(ignore_index=True)  # note that ignore_index moves to the init
for i in range(1_000_000):
    dfa = dfa.append({'i': i})

df = dfa.finalize()  # must call .finalize() before you can use the results

Type hints and category detection

Using narrower types and categories can often dramatically reduce the size of a DataFrame. There are two ways to do this in pandas-appender. One is to append to an existing dataframe:

dfa = DF_Appender(df, ignore_index=True)

and the second is to pass in a dtypes= argument:

dfa = DF_Appender(ignore_index=True, dtypes=another_dataframe.dtypes)

pandas-appender also offers a way to infer which columns would be smaller if they were categories. This code will either analyze an existing dataframe that you're appending to:

dfa = DF_Appender(df, ignore_index=True, infer_categories=True)

or it will analyze the first chunk of appended lines:

dfa = DF_Appender(ignore_index=True, infer_categories=True)

These inferred categories will override existing types or a dtypes= argument.

Incompatibilities with pandas.DataFrame.append()

DF_Appender must be finalized before use

  • Pandas: df_new = df.append() # df_new is a dataframe
  • DF_Appender: dfa_new = dfa.append() # must do df = dfa.finalize() to get a DataFrame

pandas.DataFame.append is idempotent, DF_Appender is not

  • Pandas: df_new = df.append() # df is not changed
  • DF_Appender: dfa_new = dfa.append() # modifies dfa, and dfa_new == dfa

pandas.DataFrame.append will promote types, while DF_Appender is strict

  • Pandas: append 0.1 to an integer column, and the column will be promoted to float
  • DF_Appender: when initialized with dtypes= or an existing DataFrame, appending 0.1 to an integer column causes 0.1 to be cast to an integer, i.e. 0.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pandas_appender-0.9.8.4.tar.gz (12.5 kB view details)

Uploaded Source

File details

Details for the file pandas_appender-0.9.8.4.tar.gz.

File metadata

  • Download URL: pandas_appender-0.9.8.4.tar.gz
  • Upload date:
  • Size: 12.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.3

File hashes

Hashes for pandas_appender-0.9.8.4.tar.gz
Algorithm Hash digest
SHA256 18fe83760ea2f2d109c5f2db648fde8eb7b09888e2586dbb682efacb5d82d6cd
MD5 d9a72ae889f9fb99e24334057b5cc938
BLAKE2b-256 7137ef6fc8ebc2b9a82adb9d6f9f14d642ddff27e80947a28ab5a29744ef2564

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page